Feedback-ID: i0fe9450f:Fastmail
Date: Thu, 25 Dec 2025 09:24:57 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Robert Treat <rob@xzilla.net>
Cc: Dharin Shah <dharinshah95@gmail.com>,
	Peter Eisentraut <peter@eisentraut.org>,
	pgsql-hackers@lists.postgresql.org
Subject: Re: Fwd: [PATCH] Add zstd compression for TOAST using extended
 header format
Message-ID: <aUyEWfBXYBLKGkKQ@paquier.xyz>
References: 
 <CAOj6k6dy2CRVA6Lsb5N59zE-7KNVKt=oYwWyg8ULK8zOOY8e7A@mail.gmail.com>
 <CAOj6k6dkseVvZzmEAWvBd6twZsCU0DbN+qeM7CoDuMM3r9doiw@mail.gmail.com>
 <CAOj6k6eSsQ_PTUTxW0upn6wp7tzbvQwqkQco-=+majwX8p6JJg@mail.gmail.com>
 <4c70b0f6-5aba-464c-b145-464a620c1222@eisentraut.org>
 <aUOgmypuun0rc2xO@paquier.xyz>
 <CAOj6k6fxs0Bwjr34W4aURzFPta0DTGLN-1ic-U+-M_EqJ+Wd8A@mail.gmail.com>
 <aUSDs7SaEcOa65gD@paquier.xyz>
 <CABV9wwOdeFHHNnw+rUsVnFek-oqwsJnJX9KFG6AT5Yqz5RA_vg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="8r7zBwB7NlGaH++d"
Content-Disposition: inline
In-Reply-To: 
 <CABV9wwOdeFHHNnw+rUsVnFek-oqwsJnJX9KFG6AT5Yqz5RA_vg@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/aUyEWfBXYBLKGkKQ%40paquier.xyz>
Precedence: bulk


--8r7zBwB7NlGaH++d
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Dec 24, 2025 at 11:50:48AM -0500, Robert Treat wrote:
> Agreed that I can't see pglz being removed any time soon, if ever.
> Thinking through what a conversion process would look like seems
> unwieldy at best, so I think we definitely need it for backwards
> compatibility, plus I think it is useful to have a self-contained
> option. I'd almost suggest we should look at replacing lz4, but I
> don't think that is significantly easier, it just has a smaller, more
> invested, blast radius.

Backward-compatibility requirements make a replacement of LZ4
basically impossible to me, for the same reasons as pglz.  We could
not replace the bit used in the va_extinfo to track if LZ4 compression
is used, either.  One thing that I do wonder is if it would make
things simpler in the long-run if we introduced a new separated vartag
for LZ4-compressed external TOAST pointers as well.  At least we'd
have a leaner design: it means that we have to keep the
varatt_external available on read, but we could update to the new
format when writing entries.  Or perhaps that's not worth the
complication based on the last sentence you are writing.. =20

> That said, I do suspect ztsd could quickly
> become a popular recommendation and/or default among users /
> consultants / service providers.

=2E.  Because I strongly suspect that this is going to be true, and that
zstd would just be a better replacement over lz4.  That's a trend that
I see is already going on for wal_compression.

Note that I am not on board with simply reusing varatt_external for
zstd-compressed entries, neither do I think that this is the best move
ever.  It makes the core patch simpler, but it makes things like
ToastCompressionId more complicated to think about.  If anything, I'd
consider a rename of varatt_external as the best way to go with an
intermediate "translation" structure only used in memory as I am
proposing on the other thread (something that others seem meh enough
about but I am not seeing alternate proposals floating around,
either).  This would make things like detoast_external_attr() less
confusing, I think, as the latest patch posted on this thread is
actually proving with its shortcut for toast_fetch_datum as one
example of something I'd rather not do..
--
Michael

--8r7zBwB7NlGaH++d
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmlMhFkACgkQnvQgOdby
QH0I4w//cCUPeuGOaThmSkPd9+Xxhw3LDucEPzcZyLV37QZZbbP5QlyJPcpa24pM
DAMtGFUknPwuM8lhHje1+9JbKs6oe9udwcYz0fGNkX+krTgiKisqOB/VYNipnT75
NuiHCt9g9J4H8JkcrVx6UPXmwzu0sbjoCgq4JNHX6COrc2Walcp/voiWuBqfxyVw
uP7P6mZMUkKDpTyikS2McJz15W5oO1sdVbR1tJuUI/wGbYgumbA768UsSpkDUnvV
0vPBJYkQhmQXkgk0Nrum28KovgqC7oTi86xfex8W8IOW4rmY77DUSyZaZfeqHV6i
wlbzz3Ydg6LIu6WfFwEXSKkZx31ipR40a59o4mD2Sk6ky0VidKfxe64AN4g3uVu/
KREB72PWqhOjFQnT/Tewzg1C3kIJvKcOfVkNQqFQDgfS5V0OVLQMugYE+O+uN2JO
M+gbU+4FaFWkX+ws+XMiMATlnuaR7KyMF1VWDRsruWFKYS5gTLOIJn/hdarxdGH2
J8T9qnappjxU6vXxcdnwjAQUjSw6tZIZAeJk1t2CU84osgJYDB70f0m85ARzqyrE
DSCoBwPaUzQO6tXaajPq4ueojRMU8ArIOMVq2F9D1298vxXnE/gGEdwmdPIGbsqK
cqbcoVmfpjQ3vRItzupoVtOYmSSwLPg6lmZ8aF5j3U83XxZORlU=
=apCB
-----END PGP SIGNATURE-----

--8r7zBwB7NlGaH++d--