Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1vVSdH-0045Zp-1x for pgadmin-hackers@arkaria.postgresql.org; Tue, 16 Dec 2025 10:50:20 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.96) (envelope-from ) id 1vVSdF-005zWs-1j for pgadmin-hackers@arkaria.postgresql.org; Tue, 16 Dec 2025 10:50:18 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1vVSdF-005zWk-0t for pgadmin-hackers@lists.postgresql.org; Tue, 16 Dec 2025 10:50:18 +0000 Received: from mail-ej1-x633.google.com ([2a00:1450:4864:20::633]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1vVSdC-0011zq-2y for pgadmin-hackers@lists.postgresql.org; Tue, 16 Dec 2025 10:50:17 +0000 Received: by mail-ej1-x633.google.com with SMTP id a640c23a62f3a-b7277324204so718188066b.0 for ; Tue, 16 Dec 2025 02:50:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1765882209; x=1766487009; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=+HEKPP/+BJr5wLqYXawogu80p7SI/U08ZPdMh4CROeE=; b=lbcZ0OEDVONvodrVhbTWVz92LEgtpEQ1U4c13/aEoxxSjAXZN0rhyn/VJJdNEZfiSF DqBHul09lxzgIobnbS6ZmZH9l0h2I7UDFrDCq5RxgWNCYDpj8C0nMQK1U4AvIPbqXDbh zdlLv7eXA+aaXwQNtHInNYrGDbsV7R5QX6s14Z2P/iY+z76fDEzQ/iagqo4LmfLmetLK gy/Fny4r/V+1ggW68CP7VQANu/3KDY6NNjW2RuEGwkuKRYebDE+Lwbs+LmCWPiO9HJrs iuudUs6paqOuwwfHJQxZe920E5/JJ2g5Wzrik7L/Sj54LDOJkdrKmptVe4w+ISVbg9bU /jug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765882209; x=1766487009; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=+HEKPP/+BJr5wLqYXawogu80p7SI/U08ZPdMh4CROeE=; b=t6hH9AMQfoUPUHkxMU1R9mmjPaclZRYZb9LBZ0ZSEKz0dstdur6RvToX2frjMmgmJK ZUPbf2HOeNq+4pgmMsYPtOo47H2bAky+eVj7rwOBkbr5Osj5mwi2lGUJBNo1pLsLpHNT zpzYnUcdywff9F14jtX61PrC+M9+n9Fx4BlHJva8LKiXMzY55Sh1uJo2FZMY6CFE6NZW Qdu/QP3Maza6vNVNzojteTO+2xPRZwHOT6e0ZzJMH+PDSzTZTrCSqw+ixTAUbpXHkqTi 4xL3nRitbNj4qJZEGl1p/On/9yXa8zIY2YIymdb75h1Rw1DhzMceDWTjk1fZ9JGRCgA4 tHYw== X-Gm-Message-State: AOJu0YzuL3GA/Vih/iQ+kdkd6uUbxax/JyxuA+zTXoK9cVcXD/K1j6Yg LX+4KoW6Gaz1rX8t9IuJopfeb+Loa4FltVqgw07/alqFaM8FeE8XjwRQ6YvEX+7tI7A0Cm2sRSe EnhKaAjN5Tk1Vr0nQB55/CDcjq+Sg1uo= X-Gm-Gg: AY/fxX7lrSBRG/aHkPXNihPqzPQMU4wJufoijWmkPG5CThMcINIpb+Y5pOdusOp1w39 HbtngxOFeSH8+3OpHUokjDKFhmn6REihyv/y1QtOH2xGiykqa3MorFIe7cmKw1oHrLGibGhgY/e VDsArKtZMhBAWnmHjGYZdw3hyv1I4Lqe5OJQrdhJRcL/hpUKm5AFXfSc5zooQSyQuhh38wu/hXG qUhFE8wueX45RomUSdfXlUsVl95cB865tCkkn2BWB45qUieBCEoNAUZxTxMDDjV6P1u4cwd102O AqL1+Kh98K5jBZGp/nau+vin1VHZh3Of5PwS4l67 X-Google-Smtp-Source: AGHT+IFwQThmrk8fNZ75vQPuL/d6l49gW4pF6mXG2ruXw7YS9rnvjoZYEtYMTl81IyuM6FdsJH12SJGtVCgtAfl4tm4= X-Received: by 2002:a17:907:980f:b0:b76:3476:a83c with SMTP id a640c23a62f3a-b7d23ad6794mr1310533866b.40.1765882208638; Tue, 16 Dec 2025 02:50:08 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Dharin Shah Date: Tue, 16 Dec 2025 11:49:57 +0100 X-Gm-Features: AQt7F2rgTC38yf7e8GAkxSuYA64PIF3q5VKsL0dJ28nJ5YcX8rpsSM4K4G8yCYI Message-ID: Subject: Re: [PATCH] Add zstd compression for TOAST using extended header format To: Murtuza Zabuawala Cc: pgadmin-hackers@lists.postgresql.org Content-Type: multipart/alternative; boundary="000000000000c2e26206460f7da5" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000c2e26206460f7da5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable THanks Murtuza, My bad, wrong email :( Regards, Dharin On Tue, Dec 16, 2025 at 6:56=E2=80=AFAM Murtuza Zabuawala < murtuza.zabuawala@enterprisedb.com> wrote: > Hello, > > You may want to consider sending the patch to the pgsql-hackers mailing > list. > > > > *Murtuza Zabuawala* > enterprisedb.com > > > On 16 Dec 2025, at 12:46=E2=80=AFAM, Dharin Shah = wrote: > > Hello PG Hackers, > > Want to submit a patch that implements zstd compression for TOAST data > using a 20-byte TOAST pointer format, directly addressing the concerns > raised in prior discussions [1 > > ][2 > > ][3 > > ]. > > A bit of a background in the 2022 thread [3 > = ], > The overall suggestion was to have something extensible for the TOAST hea= der > > i.e. something like: > 00 =3D PGLZ > 01 =3D LZ4 > 10 =3D reserved for future emergencies > 11 =3D extended header with additional type byte > > This patch implements that idea. > The new header format: > > struct varatt_external_extended { > int32 va_rawsize; /* same as legacy */ > uint32 va_extinfo; /* cmid=3D3 signals extended format */ > uint8 va_flags; /* feature flags */ > uint8 va_data[3]; /* va_data[0] =3D compression method */ > Oid va_valueid; /* same as legacy */ > Oid va_toastrelid; /* same as legacy */ > }; > > *A few notes:* > > - Zstd only applies to external TOAST, not inline compression. The 2-bit > limit in va_tcinfo stays as-is for inline data, where pglz/lz4 work fine > anyway. Zstd's wins show up on larger values. > - A GUC use_extended_toast_header controls whether pglz/lz4 also use the > 20-byte format (defaults to off for compatibility, can enable it if you > want consistency). > - Legacy 16-byte pointers continue to work - we check the vartag to > determine which format to read. > > The 4 extra bytes per pointer is negligible for typical TOAST data sizes, > and it gives us room to grow. > > Regards, > Dharin > > > > --000000000000c2e26206460f7da5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
THanks Murtuza,

My bad, wrong email :(

Regar= ds,
Dharin

On Tue, Dec 16, 2025 at 6:56=E2= =80=AFAM Murtuza Zabuawala <murtuza.zabuawala@enterprisedb.com> wrote:
Hello,

=
You may want to consider sending the patch to the pgsql-hackers = mailing list.



Murtu= za Zabuawala


On 16 Dec 2025, at 12:46=E2=80=AFAM= , Dharin Shah <dharinshah95@gmail.com> wrote:

Hello PG Hack= ers,

Want to su= bmit a patch that implements zstd compression for TOAST data using a 20-byt= e TOAST pointer format, directly addressing the concerns raised in prior di= scussions [1][2][3].

A bit of a background in the 2022 thread [3], The overall suggestion was to have something exten= sible for the TOAST header

i.e. something like:
00 =3D= PGLZ
= 01 =3D LZ4
10 =3D reserved for future emergencies
11 =3D extended header with add= itional type byte

This pa= tch implements that idea.
The= new header format:

=C2= =A0 struct varatt_external_extended {
=C2=A0 =C2=A0 =C2=A0 int32 =C2=A0 va_rawsize; =C2=A0 =C2=A0 /* sam= e as legacy */
=C2=A0 =C2=A0 = =C2=A0 uint32 =C2=A0va_extinfo; =C2=A0 =C2=A0 /* cmid=3D3 signals extended = format */
=C2=A0 =C2=A0 =C2= =A0 uint8 =C2=A0 va_flags; =C2=A0 =C2=A0 =C2=A0 /* feature flags */<= /font>
=C2=A0 =C2=A0 =C2=A0 uint8 =C2=A0 va= _data[3]; =C2=A0 =C2=A0 /* va_data[0] =3D compression method */
=C2=A0 =C2=A0 =C2=A0 Oid =C2=A0 =C2=A0 v= a_valueid; =C2=A0 =C2=A0 /* same as legacy */
=C2=A0 =C2=A0 =C2=A0 Oid =C2=A0 =C2=A0 va_toastrelid; =C2= =A0/* same as legacy */
=C2= =A0 };

A fe= w notes:

- Zstd only = applies to external TOAST, not inline compression. The 2-bit limit in va_tc= info stays as-is for inline data, where pglz/lz4 work fine anyway. Zstd'= ;s wins show up on larger values.
- A GUC use_extended_toast_header controls whether pglz/lz4 also use t= he 20-byte format (defaults to off for compatibility, can enable it if you = want consistency).
- Legacy 1= 6-byte pointers continue to work - we check the vartag to determine which f= ormat to read.

The 4 extr= a bytes per pointer is negligible for typical TOAST data sizes, and it give= s us room to grow.

Regards,
Dharin<= /font>
<zstd-toast-compressi= on-external.patch>

--000000000000c2e26206460f7da5--