Date: Fri, 10 Mar 2023 07:05:49 -0600
From: Justin Pryzby <pryzby@telsasoft.com>
To: Tomas Vondra <tomas.vondra@enterprisedb.com>
Cc: gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>,
	shiy.fnst@fujitsu.com, pgsql-hackers@lists.postgresql.org,
	Rachel Heaton <rachelmheaton@gmail.com>
Subject: Re: Add LZ4 compression in pg_dump
Message-ID: <ZAsrLTWyz5w0KCRl@telsasoft.com>
References: <8421f53d-c9f7-c02e-3278-9f9522a8e8f3@enterprisedb.com>
 <20230224015115.GS1653@telsasoft.com>
 <Y/g+gi/zQozPfmly@paquier.xyz>
 <ikuHRdKZjpeTCe1_5_z3wajRHSjmS0A7A9yrsQQKcqJdEjYYzXsCK-ZBNtC-x-0P5CYEMG6hLaggIxcLA1scy05qgTquWYOYNcqYzcOz13g=@pm.me>
 <20230225050214.GH1653@telsasoft.com>
 <20230225140553.GI1653@telsasoft.com>
 <20230227044910.GO1653@telsasoft.com>
 <09b37949-cfe5-29bb-cbb1-498ee5700b61@enterprisedb.com>
 <ZAoGO+/Besr40GLp@telsasoft.com>
 <2f680c4f-0944-5fd0-2e39-8cd95ccb89f5@enterprisedb.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <2f680c4f-0944-5fd0-2e39-8cd95ccb89f5@enterprisedb.com>
Archived-At: 
 <https://www.postgresql.org/message-id/ZAsrLTWyz5w0KCRl%40telsasoft.com>
Precedence: bulk

On Thu, Mar 09, 2023 at 06:58:20PM +0100, Tomas Vondra wrote:
> I'm a bit confused about the lz4 vs. lz4f stuff, TBH. If we switch to
> lz4f, doesn't that mean it (e.g. restore) won't work on systems that
> only have older lz4 version? What would/should happen if we take backup
> compressed with lz4f, an then try restoring it on an older system where
> lz4 does not support lz4f?

You seem to be thinking about LZ4F as a weird, new innovation I'm
experimenting with, but compress_lz4.c already uses LZ4F for its "file"
API.  LZ4F is also what's written by the lz4 CLI tool, and I found that
LZ4F has been included in the library for ~8 years:

https://github.com/lz4/lz4/releases?page=2
r126 Dec 24, 2014
New : lz4frame API is now integrated into liblz4

> Maybe if lz4f format is incompatible with regular lz4, we should treat
> it as a separate compression method 'lz4f'?
> 
> I'm mostly afk until the end of the week, but I tried searching for lz4f
> info - the results are not particularly enlightening, unfortunately.
> 
> AFAICS this only applies to lz4f stuff. Or would the streaming mode be a
> breaking change too?

Streaming mode outputs the same format as the existing code, but gives
better compression.  We could (theoretically) change it in a bugfix
release, and old output would still be restorable (I think new output
would even be restorable with the old versions of pg_restore).

But that's not true for LZ4F.  The benefit there is that it avoids
outputing a separate block for each row.  That's essential for narrow
tables, for which the block header currently being written has an
overhead several times larger than the data.

-- 
Justin