Feedback-ID: i0fe9450f:Fastmail
Date: Tue, 30 Dec 2025 08:45:47 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Dharin Shah <dharinshah95@gmail.com>
Cc: Robert Treat <rob@xzilla.net>, Peter Eisentraut <peter@eisentraut.org>,
	pgsql-hackers@lists.postgresql.org
Subject: Re: Fwd: [PATCH] Add zstd compression for TOAST using extended
 header format
Message-ID: <aVMSqzx7dGSDu4uG@paquier.xyz>
References: 
 <CAOj6k6dkseVvZzmEAWvBd6twZsCU0DbN+qeM7CoDuMM3r9doiw@mail.gmail.com>
 <CAOj6k6eSsQ_PTUTxW0upn6wp7tzbvQwqkQco-=+majwX8p6JJg@mail.gmail.com>
 <4c70b0f6-5aba-464c-b145-464a620c1222@eisentraut.org>
 <aUOgmypuun0rc2xO@paquier.xyz>
 <CAOj6k6fxs0Bwjr34W4aURzFPta0DTGLN-1ic-U+-M_EqJ+Wd8A@mail.gmail.com>
 <aUSDs7SaEcOa65gD@paquier.xyz>
 <CABV9wwOdeFHHNnw+rUsVnFek-oqwsJnJX9KFG6AT5Yqz5RA_vg@mail.gmail.com>
 <aUyEWfBXYBLKGkKQ@paquier.xyz>
 <CAOj6k6dG0M-G0gs_Htray5y_pfvacXSjKYw0uJaaYJvOeqk7xw@mail.gmail.com>
 <CAOj6k6dEVi0NvLjMLDhyrJS_n_NZO5D_OU89AO1u53u6NCDDwQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="QOmSrVUgh/TzW+xL"
Content-Disposition: inline
In-Reply-To: 
 <CAOj6k6dEVi0NvLjMLDhyrJS_n_NZO5D_OU89AO1u53u6NCDDwQ@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/aVMSqzx7dGSDu4uG%40paquier.xyz>
Precedence: bulk


--QOmSrVUgh/TzW+xL
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mon, Dec 29, 2025 at 02:45:27PM +0100, Dharin Shah wrote:
> The goal is to centralize =E2=80=9Chow do we interpret an external datum?=
=E2=80=9D so that
> detoast code paths don=E2=80=99t have to reason directly about va_extinfo=
 encoding
> vs payload layout details. This is intended as groundwork for a follow-up
> patch adding a new vartag-based method (e.g., zstd) without scattering
> special cases in detoast paths.

+static bool
+decode_external_toast_pointer(const struct varlena *attr,
+						   DecodedExternalToast *decoded)
[...]
+typedef enum ToastDecompressMethod
+{
+	TOAST_DECOMP_NONE =3D 0,
+	TOAST_DECOMP_PGLZ =3D 1,
+	TOAST_DECOMP_LZ4 =3D 2
+} ToastDecompressMethod;
+
+typedef struct DecodedExternalToast
+{
+	Oid			toastrelid;
+	Oid			valueid;
+	uint32		rawsize;		/* Decompressed size; for future methods without tcinfo=
 */
+	uint32		extsize;		/* On-disk payload size */
+	ToastDecompressMethod method;
+	uint8		flags;
+} DecodedExternalToast;

Yeah, honestly this is a layer I have been thinking about as well as
one option, but contrary to you I have been focusing on putting that
into varatt.h, with the exception of the value being an Oid8.  I think
that you have an interesting point in focusing your implementation to
be stored in the detoast part, though.  I'd need to spend a bit more
time to see the result this would lead at with the larger 8-byte issue
in mind, but this is something that would come at no real cost as it
has no function pointer redirection compared to what I was first
envisioning on the other thread.  That's especially true if it makes
the CompressionId business easier to mold around when adding a new
vartag.

> Why HAS_TCINFO?
> - Previously, =E2=80=9Cis compressed?=E2=80=9D was used as a proxy for wh=
ether the external
> payload begins with tcinfo. This patch makes that explicit: HAS_TCINFO
> captures payload layout, which is distinct from whether the value is
> compressed. This separation is needed for future methods that may store
> external compressed payloads without tcinfo.

It is possible to model the on-memory data as we want.  This
suggestion would be OK with some flags.
--
Michael

--QOmSrVUgh/TzW+xL
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmlTEqsACgkQnvQgOdby
QH1GHQ//eNxwep1/7ED1nHcUQV2bo9+0kRxZiwZ3uOST5+5U3jlZjs44SEefYo/w
tO6GXe95nBbVp+lCwFnFvB8XMpdaS62pA9Oe8/wYhLhGSbvl/wPe1+Ry4Sp59iH1
mO2sh8AdVcdCHmOPkGcECt7s7GafS3p1nzn5qj+OrG9r9XKusxEygcwejr/K62rn
ncrkrs1ViXqklZDDy7b/c10XQnsq7kPN2iJ0mH1XttDJ3Jb3xnYoqCAV8gLkgSpt
xvSFRTawxV3BqIlAJeemcL3oezyKYW8RxBL0G/PYX8uL6tfhzLc37Tkq4LtYK6pe
fTTNGJZdd5suNBBwhs8Kj9b7l5FUkdaf/OiSTA1PA5l+USevjf0aCOCd6Y/9EZza
ukzTqhpTXcIgRZ5is7SrxAW1mr6jU1YhYBKgpldQfrgoooJV/Ruhctv6mjzDQ+pJ
DlB24L8yo9uJSwxTg/5Z+go+EpCOR44Qag3TKGAYlMqnQDV6L2gw248/ShIIbl4z
327iM8dTH/H3/FV1lO5d6zhQt3P+grxQcSWsa3rwN8Bz4HYqNDXG/Nbar5NtiZph
irqXk9So8I7g49J/HwFszPvin93mXDjILZLkoHpnTwaVn66xkUJ+/eoHyt1jXosV
nFUnTqe0Kfw3zIjFDF+TNK3jYdHyOWVzU2DGiwMe0VkiOyVjaAM=
=UIeH
-----END PGP SIGNATURE-----

--QOmSrVUgh/TzW+xL--