Content-Type: multipart/mixed; boundary="------------KlEPLecfgPwt0ebUCTEo0V0f"
Message-ID: <f735df01-0bb4-2fbc-1297-73a520cfc534@enterprisedb.com>
Date: Mon, 8 May 2023 20:00:39 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.10.0
Subject: Re: Add LZ4 compression in pg_dump
To: Tom Lane <tgl@sss.pgh.pa.us>, Michael Paquier <michael@paquier.xyz>
Cc: gkokolatos@pm.me, Andrew Dunstan <andrew@dunslane.net>,
 Alexander Lakhin <exclusion@gmail.com>, Justin Pryzby
 <pryzby@telsasoft.com>, shiy.fnst@fujitsu.com,
 pgsql-hackers@lists.postgresql.org, Rachel Heaton <rachelmheaton@gmail.com>
References: 
 <7zu6GYFi72jByVk2SEe_nd8r1DHhwA35vSntAE9sAfiMX9Lfk6bwimkwmu7GDgYaEvL27SE4qF1uK4_LHWQM4H-SXOwkMAI-GV8PKGmED_E=@pm.me>
 <JduQzRIUMhF5NDvC_klfyHTDhzb5Py60xH9ICRUtluHQ4dD02nw-BH7WfX7DQG6gaFWl1Q919RvX3Xvm_5mau_0YKRQeUs6ezUSFDlFVPsE=@pm.me>
 <32ba9612-1982-4b2d-135a-5d1222987082@enterprisedb.com>
 <XPMIIorJ5munrlLRZGquartiN5tuapdjiBhaPZ6iEmJccRiKWCIHSowSLPhaGj4gLpqvaS1pQn7PGkA_WNwvuUTVdl-lrU_TzHA81LqujJ4=@pm.me>
 <ZBjg4D/X05X+JbcJ@telsasoft.com>
 <3beb0dd7-c927-3eca-689f-716e255d5b27@enterprisedb.com>
 <94ae9bca-5ebb-1e68-bb7b-4f32e89fefbe@gmail.com>
 <MmuCeC4k017VCWzr-f8MEQ9hx8DRak2ZGbb5SVwKvBBHqCt8tLKzVVdMcPMsi3Zj314OJbs1UPEzwc5RwOv3IMp1lAkC7rg-yDOaDBJGww8=@pm.me>
 <71868915-0345-1b7b-f3d6-ec66547c662a@dunslane.net>
 <P-Dl7-QRVPmjv35H2n-Wnj9amDG_MUEb0-jtlzs05n5z03lq0IZNTsI5h4DjTmLekfmKexgFBQhmVXtUO402vAIFx0e002lIzsb8O8Hi1wA=@pm.me>
 <ZFhCyn4Gm2eu60rB@paquier.xyz> <219662.1683508165@sss.pgh.pa.us>
 <220484.1683508611@sss.pgh.pa.us>
Content-Language: en-US
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
In-Reply-To: <220484.1683508611@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/f735df01-0bb4-2fbc-1297-73a520cfc534%40enterprisedb.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------KlEPLecfgPwt0ebUCTEo0V0f
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

On 5/8/23 03:16, Tom Lane wrote:
> I wrote:
>> Michael Paquier <michael@paquier.xyz> writes:
>>> While testing this patch, I have triggered an error pointing out that
>>> the decompression path of LZ4 is broken for table data.  I can
>>> reproduce that with a dump of the regression database, as of:
>>> make installcheck
>>> pg_dump --format=d --file=dump_lz4 --compress=lz4 regression
> 
>> Ugh.  Reproduced here ... so we need an open item for this.
> 
> BTW, it seems to work with --format=c.
> 

The LZ4Stream_write() forgot to move the pointer to the next chunk, so
it was happily decompressing the initial chunk over and over. A bit
embarrassing oversight :-(

The custom format calls WriteDataToArchiveLZ4(), which was correct.

The attached patch fixes this for me.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--------------KlEPLecfgPwt0ebUCTEo0V0f
Content-Type: text/x-patch; charset=UTF-8; name="pg-dump-lz4-fix.patch"
Content-Disposition: attachment; filename="pg-dump-lz4-fix.patch"
Content-Transfer-Encoding: base64

ZGlmZiAtLWdpdCBhL3NyYy9iaW4vcGdfZHVtcC9jb21wcmVzc19sejQuYyBiL3NyYy9iaW4v
cGdfZHVtcC9jb21wcmVzc19sejQuYwppbmRleCA0MjNlMWI3OTc2Zi4uNDNjNGI5MTg3ZWYg
MTAwNjQ0Ci0tLSBhL3NyYy9iaW4vcGdfZHVtcC9jb21wcmVzc19sejQuYworKysgYi9zcmMv
YmluL3BnX2R1bXAvY29tcHJlc3NfbHo0LmMKQEAgLTU4NCw2ICs1ODQsOCBAQCBMWjRTdHJl
YW1fd3JpdGUoY29uc3Qgdm9pZCAqcHRyLCBzaXplX3Qgc2l6ZSwgQ29tcHJlc3NGaWxlSGFu
ZGxlICpDRkgpCiAJCQllcnJubyA9IChlcnJubykgPyBlcnJubyA6IEVOT1NQQzsKIAkJCXJl
dHVybiBmYWxzZTsKIAkJfQorCisJCXB0ciA9ICgoY2hhciAqKSBwdHIpICsgY2h1bms7CiAJ
fQogCiAJcmV0dXJuIHRydWU7Cg==

--------------KlEPLecfgPwt0ebUCTEo0V0f--