MIME-Version: 1.0
From: Nikita Malakhov <hukutoc@gmail.com>
Date: Wed, 10 May 2023 23:04:57 +0300
Message-ID: <CAN-LCVMq2X=fhx7KLxfeDyb3P+BXuCkHC0g=9GF+JD4izfVa0Q@mail.gmail.com>
Subject: RFI: Extending the TOAST Pointer
To: pgsql-hackers <pgsql-hackers@postgresql.org>, 
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000ca33dd05fb5c6369"
Archived-At: <https://www.postgresql.org/message-id/CAN-LCVMq2X%3Dfhx7KLxfeDyb3P%2BBXuCkHC0g%3D9GF%2BJD4izfVa0Q%40mail.gmail.com>
Precedence: bulk

--000000000000ca33dd05fb5c6369
Content-Type: text/plain; charset="UTF-8"

Hi hackers!

There were several discussions where the limitations of the existing TOAST
pointers
were mentioned [1], [2] and [3] and from time to time this topic appears in
other places.

We proposed a fresh approach to the TOAST mechanics in [2], but
unfortunately the
patch was met quite unfriendly, and after several iterations was rejected,
although we
still have hopes for it and have several very promising features based on
it.

Anyway, the old TOAST pointer is also the cause of problems like [4], and
this part of
the PostgreSQL screams to be revised and improved.

The TOAST begins with the pointer to the externalized value - the TOAST
Pointer, which
is very limited in means of storing data, and all TOAST improvements
require revision
of this Pointer structure. So we decided to ask the community for thoughts
and ideas on
how to rework this pointer.
The TOAST Pointer (varatt_external structure) stores 4 fields:
[varlena header][<4b - original data size><4b - size in TOAST table><4b -
TOAST table OID><4b - ID of chunk>]
In [2] we proposed the new Custom TOAST pointer structure where main
feature is
extensibility:
[varlena header][<2b - total size of the TOAST pointer><4b size of original
data><4b - OID of algorithm used for TOASTing><variable length field used
for storing any custom data>]
where Custom TOAST Pointer is distinguished from Regular one by va_flag
field which
is a part of varlena header, so new pointer format does not interfere with
the old (regular) one.
The first field is necessary because the Custom TOAST pointer has variable
length due to the
tail used for inline storage, and original value size is used by the
Executor. The third field could
be a subject for discussion.

Thoughts? Objections?

[1] [PATCH] Infinite loop while acquiring new TOAST Oid
<https://www.postgresql.org/message-id/CAJ7c6TPSvR2rKpoVX5TSXo_kMxXF%2B-SxLtrpPaMf907tX%3DnVCw%40mail.gmail.com>
[2] Pluggable Toaster
<https://www.postgresql.org/message-id/flat/224711f9-83b7-a307-b17f-4457ab73aa0a%40sigaev.ru>
[3] [PATCH] Compression dictionaries for JSONB
<https://www.postgresql.org/message-id/flat/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com>
[4] BUG #16722: PG hanging on COPY when table has close to 2^32 toasts in
the table.
<https://www.postgresql.org/message-id/flat/16722-93043fb459a41073%40postgresql.org>

-- 
Regards,
Nikita Malakhov
Postgres Professional
The Russian Postgres Company
https://postgrespro.ru/

--000000000000ca33dd05fb5c6369
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi hackers!<div><br></div><div>There were several discussi=
ons where the limitations of the existing TOAST pointers</div><div>were men=
tioned [1], [2] and [3] and from time to time this topic appears in other p=
laces.</div><div><br></div><div>We proposed=C2=A0a fresh approach to the TO=
AST mechanics in [2], but unfortunately the</div><div>patch=C2=A0was met qu=
ite unfriendly, and after several iterations was rejected, although=C2=A0we=
</div><div>still have hopes for it and have several very promising features=
 based on it.</div><div><br></div><div>Anyway, the old TOAST pointer is als=
o the cause of problems like [4], and this part of</div><div>the PostgreSQL=
 screams to be revised and improved.</div><div><br></div><div>The TOAST beg=
ins with the pointer to the externalized value - the TOAST Pointer, which</=
div><div>is very limited in means of storing data, and all TOAST improvemen=
ts require revision</div><div>of this Pointer structure. So we decided to a=
sk the community for thoughts and ideas on</div><div>how to rework this poi=
nter.</div><div>The TOAST Pointer (varatt_external structure) stores 4 fiel=
ds:</div><div>[varlena header][&lt;4b - original data size&gt;&lt;4b - size=
 in TOAST table&gt;&lt;4b - TOAST table OID&gt;&lt;4b - ID of chunk&gt;]</d=
iv><div>In [2] we proposed the new Custom TOAST pointer structure where mai=
n feature is</div><div>extensibility:</div><div>[varlena header][&lt;2b - t=
otal size of the TOAST pointer&gt;&lt;4b size of original data&gt;&lt;4b - =
OID of algorithm used for TOASTing&gt;&lt;variable length field used for st=
oring any custom data&gt;]</div><div>where Custom TOAST Pointer is distingu=
ished from Regular one by va_flag field which</div><div>is a part of varlen=
a header, so new pointer format does not interfere with the old (regular) o=
ne.</div><div>The first field is necessary because the Custom TOAST pointer=
 has variable length due to the</div><div>tail used for inline storage, and=
 original=C2=A0value size is used by the Executor. The third field could</d=
iv><div>be a subject for discussion.</div><div><br></div><div>Thoughts? Obj=
ections?</div><div><br></div><div>[1]=C2=A0<a href=3D"https://www.postgresq=
l.org/message-id/CAJ7c6TPSvR2rKpoVX5TSXo_kMxXF%2B-SxLtrpPaMf907tX%3DnVCw%40=
mail.gmail.com">[PATCH] Infinite loop while acquiring new TOAST Oid</a></di=
v><div>[2]=C2=A0<a href=3D"https://www.postgresql.org/message-id/flat/22471=
1f9-83b7-a307-b17f-4457ab73aa0a%40sigaev.ru">Pluggable Toaster</a></div><di=
v>[3]=C2=A0<a href=3D"https://www.postgresql.org/message-id/flat/CAJ7c6TOtA=
B0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com">[PATCH] Compr=
ession dictionaries for JSONB</a></div><div>[4]=C2=A0<a href=3D"https://www=
.postgresql.org/message-id/flat/16722-93043fb459a41073%40postgresql.org">BU=
G #16722: PG hanging on COPY when table has close to 2^32 toasts in the tab=
le.</a><br clear=3D"all"><div><br></div><span class=3D"gmail_signature_pref=
ix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature" data-smartmail=
=3D"gmail_signature"><div dir=3D"ltr"><div style=3D"color:rgb(34,34,34)">Re=
gards,</div><div style=3D"color:rgb(34,34,34)">Nikita Malakhov<br></div><di=
v style=3D"color:rgb(34,34,34)">Postgres Professional</div><div style=3D"co=
lor:rgb(34,34,34)">The Russian Postgres Company</div><div style=3D"color:rg=
b(34,34,34)"><a href=3D"https://postgrespro.ru/" target=3D"_blank">https://=
postgrespro.ru/</a></div></div></div></div></div>

--000000000000ca33dd05fb5c6369--