public inbox for [email protected]  

help / color / mirror / Atom feed

From: Murtuza Zabuawala <[email protected]>
To: Dharin Shah <[email protected]>
Cc: [email protected]
Subject: Re: [PATCH] Add zstd compression for TOAST using extended header format
Date: Tue, 16 Dec 2025 11:26:30 +0530
Message-ID: <[email protected]> (raw)
In-Reply-To: <CAOj6k6dkseVvZzmEAWvBd6twZsCU0DbN+qeM7CoDuMM3r9doiw@mail.gmail.com>
References: <CAOj6k6dy2CRVA6Lsb5N59zE-7KNVKt=oYwWyg8ULK8zOOY8e7A@mail.gmail.com>
	<CAOj6k6dkseVvZzmEAWvBd6twZsCU0DbN+qeM7CoDuMM3r9doiw@mail.gmail.com>

Hello,

You may want to consider sending the patch to the pgsql-hackers mailing list.




Murtuza Zabuawala
enterprisedb.com <http://enterprisedb.com/;


> On 16 Dec 2025, at 12:46 AM, Dharin Shah <[email protected]> wrote:
> 
> Hello PG Hackers,
> 
> Want to submit a patch that implements zstd compression for TOAST data using a 20-byte TOAST pointer format, directly addressing the concerns raised in prior discussions [1 <https://www.postgresql.org/message-id/flat/CAFAfj_F4qeRCNCYPk1vgH42fDZpjQWKO%2Bufq3FyoVyUa5AviFA%40m...;][2 <https://www.postgresql.org/message-id/flat/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22=5xVBg7S4vr5rQ@mail....;][3 <https://www.postgresql.org/message-id/flat/[email protected];].
> 
> A bit of a background in the 2022 thread [3 <https://www.postgresql.org/message-id/flat/[email protected];], The overall suggestion was to have something extensible for the TOAST header
> 
> i.e. something like:
> 00 = PGLZ
> 01 = LZ4
> 10 = reserved for future emergencies
> 11 = extended header with additional type byte
> 
> This patch implements that idea.
> The new header format:
> 
>   struct varatt_external_extended {
>       int32   va_rawsize;     /* same as legacy */
>       uint32  va_extinfo;     /* cmid=3 signals extended format */
>       uint8   va_flags;       /* feature flags */
>       uint8   va_data[3];     /* va_data[0] = compression method */
>       Oid     va_valueid;     /* same as legacy */
>       Oid     va_toastrelid;  /* same as legacy */
>   };
> 
> A few notes:
> 
> - Zstd only applies to external TOAST, not inline compression. The 2-bit limit in va_tcinfo stays as-is for inline data, where pglz/lz4 work fine anyway. Zstd's wins show up on larger values.
> - A GUC use_extended_toast_header controls whether pglz/lz4 also use the 20-byte format (defaults to off for compatibility, can enable it if you want consistency).
> - Legacy 16-byte pointers continue to work - we check the vartag to determine which format to read.
> 
> The 4 extra bytes per pointer is negligible for typical TOAST data sizes, and it gives us room to grow.
> 
> Regards, 
> Dharin
> <zstd-toast-compression-external.patch>

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: [PATCH] Add zstd compression for TOAST using extended header format
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox