pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
To: Michael Paquier <michael@paquier.xyz>
Cc: gkokolatos@pm.me, Andrew Dunstan <andrew@dunslane.net>
Cc: Alexander Lakhin <exclusion@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: shiy.fnst@fujitsu.com, pgsql-hackers@lists.postgresql.org, Rachel Heaton <rachelmheaton@gmail.com>
Subject: Re: Add LZ4 compression in pg_dump
Date: Mon, 8 May 2023 20:00:39 +0200
Message-ID: <f735df01-0bb4-2fbc-1297-73a520cfc534@enterprisedb.com> (raw)
In-Reply-To: <220484.1683508611@sss.pgh.pa.us>
References: <7zu6GYFi72jByVk2SEe_nd8r1DHhwA35vSntAE9sAfiMX9Lfk6bwimkwmu7GDgYaEvL27SE4qF1uK4_LHWQM4H-SXOwkMAI-GV8PKGmED_E=@pm.me>
	<JduQzRIUMhF5NDvC_klfyHTDhzb5Py60xH9ICRUtluHQ4dD02nw-BH7WfX7DQG6gaFWl1Q919RvX3Xvm_5mau_0YKRQeUs6ezUSFDlFVPsE=@pm.me>
	<32ba9612-1982-4b2d-135a-5d1222987082@enterprisedb.com>
	<XPMIIorJ5munrlLRZGquartiN5tuapdjiBhaPZ6iEmJccRiKWCIHSowSLPhaGj4gLpqvaS1pQn7PGkA_WNwvuUTVdl-lrU_TzHA81LqujJ4=@pm.me>
	<ZBjg4D/X05X+JbcJ@telsasoft.com>
	<3beb0dd7-c927-3eca-689f-716e255d5b27@enterprisedb.com>
	<94ae9bca-5ebb-1e68-bb7b-4f32e89fefbe@gmail.com>
	<MmuCeC4k017VCWzr-f8MEQ9hx8DRak2ZGbb5SVwKvBBHqCt8tLKzVVdMcPMsi3Zj314OJbs1UPEzwc5RwOv3IMp1lAkC7rg-yDOaDBJGww8=@pm.me>
	<71868915-0345-1b7b-f3d6-ec66547c662a@dunslane.net>
	<P-Dl7-QRVPmjv35H2n-Wnj9amDG_MUEb0-jtlzs05n5z03lq0IZNTsI5h4DjTmLekfmKexgFBQhmVXtUO402vAIFx0e002lIzsb8O8Hi1wA=@pm.me>
	<ZFhCyn4Gm2eu60rB@paquier.xyz>
	<219662.1683508165@sss.pgh.pa.us>
	<220484.1683508611@sss.pgh.pa.us>

On 5/8/23 03:16, Tom Lane wrote:
> I wrote:
>> Michael Paquier <michael@paquier.xyz> writes:
>>> While testing this patch, I have triggered an error pointing out that
>>> the decompression path of LZ4 is broken for table data.  I can
>>> reproduce that with a dump of the regression database, as of:
>>> make installcheck
>>> pg_dump --format=d --file=dump_lz4 --compress=lz4 regression
> 
>> Ugh.  Reproduced here ... so we need an open item for this.
> 
> BTW, it seems to work with --format=c.
> 

The LZ4Stream_write() forgot to move the pointer to the next chunk, so
it was happily decompressing the initial chunk over and over. A bit
embarrassing oversight :-(

The custom format calls WriteDataToArchiveLZ4(), which was correct.

The attached patch fixes this for me.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Attachments:

  [text/x-patch] pg-dump-lz4-fix.patch (397B, ../f735df01-0bb4-2fbc-1297-73a520cfc534@enterprisedb.com/2-pg-dump-lz4-fix.patch)
  download | inline diff:
diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c
index 423e1b7976f..43c4b9187ef 100644
--- a/src/bin/pg_dump/compress_lz4.c
+++ b/src/bin/pg_dump/compress_lz4.c
@@ -584,6 +584,8 @@ LZ4Stream_write(const void *ptr, size_t size, CompressFileHandle *CFH)
 			errno = (errno) ? errno : ENOSPC;
 			return false;
 		}
+
+		ptr = ((char *) ptr) + chunk;
 	}
 
 	return true;


view thread (205+ messages)  latest in thread

Message-ID: <f735df01-0bb4-2fbc-1297-73a520cfc534@enterprisedb.com>
Permalink:  ../f735df01-0bb4-2fbc-1297-73a520cfc534@enterprisedb.com/
Also on:    postgresql.org/message-id/f735df01-0bb4-2fbc-1297-73a520cfc534@enterprisedb.com

 ·  · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: tomas.vondra@enterprisedb.com, tgl@sss.pgh.pa.us, michael@paquier.xyz, andrew@dunslane.net, exclusion@gmail.com, pryzby@telsasoft.com, rachelmheaton@gmail.com
  Subject: Re: Add LZ4 compression in pg_dump
  In-Reply-To: <f735df01-0bb4-2fbc-1297-73a520cfc534@enterprisedb.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox