Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pw94i-0005t6-Ep for pgsql-hackers@arkaria.postgresql.org; Mon, 08 May 2023 22:11:20 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1pw94h-0000Qo-5O for pgsql-hackers@arkaria.postgresql.org; Mon, 08 May 2023 22:11:19 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pw94g-0000Qf-TG for pgsql-hackers@lists.postgresql.org; Mon, 08 May 2023 22:11:18 +0000 Received: from wout2-smtp.messagingengine.com ([64.147.123.25]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pw94Z-001SOg-D1 for pgsql-hackers@lists.postgresql.org; Mon, 08 May 2023 22:11:18 +0000 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 5D88B320091B; Mon, 8 May 2023 18:11:05 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 08 May 2023 18:11:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=paquier.xyz; h= cc:cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1683583864; x=1683670264; bh=qf OZ4BGGh3vVNNXM2o7YsPIKyVQSdTtU0oltHwxzzO8=; b=zXKTjhrXueOIpyTZM7 H8vxHchb2JBkB6iVJa+pqMo5cecJtoPnnas9D4uGyunwvRm4W/yD61jn+KFVdLO8 0oJiDYvbRBcWiKsHlXv3G2cfV3fAqCm18Y+zmkZMZg0+mwwbSH2P+HEn4p5hAypm CJlWuM+rEt1hhs3vyrGR+6tmgp9KfoKOSrt7cwoKf/3rGgiYrDKyo8IqgmevPv/b gzblpps/n5zPtySNt8xQp4WlEED+rvoXHoNDDd2qJ+pNebKtvmlEy6WcomQ5uiow rHIlw5WdEp8lpzY4Ryimg0HqD2ivQhX4QX3vRZeiGGQQ/LyJFC0B/H2xYyHh4RD0 bO4g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1683583864; x=1683670264; bh=qfOZ4BGGh3vVN NXM2o7YsPIKyVQSdTtU0oltHwxzzO8=; b=gTj0wK5IhBdAGdvga7qJ6EckS76rD +gs9qKP4zfprZ2ozzl2f0EY2bwYtF+qxSs7TipfNSyxW51qDDEEwu0J/WCMf5Ar4 Uy3oVlHxUXbUP1OYGve03UDZrwARo7NqEr0huLNUF6u8JtrxNuVODywi1jMjZwvt QhtrRWjqmsxUosIfPa+8VMY8LGG55MJPJv3GgpDZliFir1N+Q8T46KUE2T64k0s6 bVhCdPKgOIvJ3bHSP7r1QIwSIHon5c8u+lxhxDwySVNiWz+Uw9KDw9YI1PYtOCLr qqzfLuRl20gXSgQ8B5S+rCcXtrYsZW1BQw3C55+NJFTpWyLUO/OdY/ocQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrfeefledgtdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne gfrhhlucfvnfffucdljedtmdenucfjughrpeffhffvvefukfhfgggtuggjsehgtderredt tddvnecuhfhrohhmpefoihgthhgrvghlucfrrghquhhivghruceomhhitghhrggvlhesph grqhhuihgvrhdrgiihiieqnecuggftrfgrthhtvghrnhepteelieefudffhffhtdetleeg geegfffhkeeuveetiefgudduvedutefggeeivdejnecuvehluhhsthgvrhfuihiivgeptd enucfrrghrrghmpehmrghilhhfrhhomhepmhhitghhrggvlhesphgrqhhuihgvrhdrgiih ii X-ME-Proxy: Feedback-ID: i0fe9450f:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 8 May 2023 18:10:59 -0400 (EDT) Date: Tue, 9 May 2023 07:10:55 +0900 From: Michael Paquier To: Tomas Vondra Cc: Tom Lane , gkokolatos@pm.me, Andrew Dunstan , Alexander Lakhin , Justin Pryzby , shiy.fnst@fujitsu.com, pgsql-hackers@lists.postgresql.org, Rachel Heaton Subject: Re: Add LZ4 compression in pg_dump Message-ID: References: <3beb0dd7-c927-3eca-689f-716e255d5b27@enterprisedb.com> <94ae9bca-5ebb-1e68-bb7b-4f32e89fefbe@gmail.com> <71868915-0345-1b7b-f3d6-ec66547c662a@dunslane.net> <219662.1683508165@sss.pgh.pa.us> <220484.1683508611@sss.pgh.pa.us> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="wEs8lh606E7nW9cG" Content-Disposition: inline In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --wEs8lh606E7nW9cG Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 08, 2023 at 08:00:39PM +0200, Tomas Vondra wrote: > The LZ4Stream_write() forgot to move the pointer to the next chunk, so > it was happily decompressing the initial chunk over and over. A bit > embarrassing oversight :-( >=20 > The custom format calls WriteDataToArchiveLZ4(), which was correct. >=20 > The attached patch fixes this for me. Ouch. So this was corrupting the dumps and the compression when trying to write more than two chunks at once, not the decompression steps. That addresses the issue here as well, thanks! -- Michael --wEs8lh606E7nW9cG Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmRZc28ACgkQnvQgOdby QH1ByQ//W0lby0IKeU/mcMFGCL79XVlTSCRSIKlUOM79G1KHH6u/vwPRWsEfed7+ KpTa5KqmfoCJCHa5HMxWcg9ZeqJmsrvS0JitypukhFrdUG+gOwVznOCdoeBHFlMx Kchcn6hAi5CneYUYCRtrVtu9RetRJ4RLdOK4ZYZ63VPcPL4erqKZp+9lwbpwm7X6 Tkh3NqNoJSYrVymDtAcyVvN8mSiETlnKkt4ABgq0xIetp+2mPCjC/5wMnegNm9PQ PIknOqdo5dsdbVzECPguhgggPGk2FY4hA/HkDxI1tUq67mRDYucSOCzthN7bWNjy RFf4cJFH4c7OG+hChMAqQh9rToh+qM5jSSwwCrTOSOpnKE00H9dRrDId5bRRRW8S VWIAURAAwwsAHkRNIP4ulppwfj+CxSh4+647g8wL+CdvCgEHROM+fJdxj5UaPvCI DZtzqbxE2nJPRFdO6CU5rM5g5f15vQRPMB2/4z5B+vBkbKh1E7bokuaQ+Kz/Dmf2 iJx5Eugj+s2zBr33LJ93KBYaEnK6oGsu00yzxswhhlm+ZG4ypRTfI4joneiy9WWQ JYQcv9FvUBOvuU0xZs7zNvUDMa8tAdWNego27lNzz9TPCTCfbuNy3er09r0b2mie oEFUI1PMoi4vCOukZ0TpaiuzQ+RJplFp1DFk9+jrCcNBw+tNjnM= =+yId -----END PGP SIGNATURE----- --wEs8lh606E7nW9cG--