Subject: Re: RFI: Extending the TOAST Pointer

From: Aleksander Alekseev <[email-masked]>
Date: Thu, 18 May 2023 13:51:58 +0300

Hi Nikita,

> this part of the PostgreSQL screams to be revised and improved

I completely agree. The problem with TOAST pointers is that they are
not extendable at the moment which prevents adding new compression
algorithms (e.g. ZSTD), new features like compression dictionaries
[1], etc. I suggest we add extensibility in order to solve this
problem for the foreseeable future for everyone.

> where Custom TOAST Pointer is distinguished from Regular one by va_flag field
> which is a part of varlena header

I don't think that varlena header is the best place to distinguish a
classical TOAST pointer from an extended one. On top of that I don't
see any free bits that would allow adding such a flag to the on-disk
varlena representation [2].

The current on-disk TOAST pointer representation is following:

```
typedef struct varatt_external
{
int32 va_rawsize; /* Original data size (includes header) */
uint32 va_extinfo; /* External saved size (without header) and
                              * compression method */
Oid va_valueid; /* Unique ID of value within TOAST table */
Oid va_toastrelid; /* RelID of TOAST table containing it */
} varatt_external;
```

Note that currently only 2 compression methods are supported:

```
typedef enum ToastCompressionId
{
TOAST_PGLZ_COMPRESSION_ID = 0,
TOAST_LZ4_COMPRESSION_ID = 1,
TOAST_INVALID_COMPRESSION_ID = 2
} ToastCompressionId;
```

I suggest adding a new flag that will mark an extended TOAST format:

```
typedef enum ToastCompressionId
{
TOAST_PGLZ_COMPRESSION_ID = 0,
TOAST_LZ4_COMPRESSION_ID = 1,
TOAST_RESERVED_COMPRESSION_ID = 2,
TOAST_HAS_EXTENDED_FORMAT = 3,
} ToastCompressionId;
```

For an extended format we add a varint (utf8-like) bitmask right after
varatt_external that marks the features supported in this particular
instance of the pointer. The rest of the data is interpreted depending
on the bits set. This will allow us to extend the pointers
indefinitely.

Note that the proposed approach doesn't require running any
migrations. Note also that I described only the on-disk
representation. We can tweak the in-memory representation as we want
without affecting the end user.

Thoughts?

[1]: https://commitfest.postgresql.org/43/3626/
[2]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/include/postgres.h;h=0446daa0e61722067bb75aa693a92b38736e12df;hb=164d174bbf9a3aba719c845497863cd3c49a3ad0#l178

-- 
Best regards,
Aleksander Alekseev

---

From: Matthias van de Meent <[email-masked]>
Date: Thu, 18 May 2023 14:05:54 +0200

> Oid va_toastrelid; /* RelID of TOAST table containing it */

No, that's inaccurate. The complete on-disk representation of a varatt is

{
    uint8       va_header;      /* Always 0x80 or 0x01 */
    uint8       va_tag;         /* Type of datum */
    char        va_data[FLEXIBLE_ARRAY_MEMBER]; /* Type-dependent
data, for toasted values that's currently only a varatt_external */
} varattrib_1b_e;

With va_tag being filled with one of the vartag_external values:

typedef enum vartag_external
{
    VARTAG_INDIRECT = 1,
    VARTAG_EXPANDED_RO = 2,
    VARTAG_EXPANDED_RW = 3,
    VARTAG_ONDISK = 18
} vartag_external;

This enum still has many options to go before it exceeds the maximum
of the uint8 va_tag field. Therefore, I don't think we have no disk
representations left, nor do I think we'll need to add another option
to the ToastCompressionId enum.
As an example, we can add another VARTAG option for dictionary-enabled
external toast; like what the pluggable toast patch worked on. I think
we can salvage some ideas from that patch, even if the main idea got
stuck.

Kind regards,

Matthias van de Meent
Neon Inc.

---

From: Aleksander Alekseev <[email-masked]>
Date: Thu, 18 May 2023 15:57:14 +0300

Hi,

> we can salvage some ideas from that patch, even if the main idea got

The problem here is that the comments are ambiguous regarding what to
call "TOAST pointer" exactly. I proposed a patch for this but it was
not accepted [1].

So the exact on-disk representation of a TOAST'ed value (for
little-endian machines) is:

0b00000001, 18 (va_tag), (varatt_external here)

Where 18 is sizeof(varatt_external) + 2, because the length includes
the length of the header.

I agree that va_tag can have another use. But since we are going to
make varatt_external variable in size (otherwise I don't see how it
could be really **extendable**) I don't think this is the right
approach.

Also I agree that this particular statement is incorrect:

> This will allow us to extend the pointers indefinitely.

varatt_external is going to be limited to 255. But it seems to be a
reasonable limitation for the nearest 10-20 years or so.

[1]: https://commitfest.postgresql.org/39/3820/

-- 
Best regards,
Aleksander Alekseev

---

From: Robert Haas <[email-masked]>
Date: Tue, 23 May 2023 12:33:50 -0400

> we can salvage some ideas from that patch, even if the main idea got

Adding another VARTAG option is somewhat different from adding another
ToastCompressionId. I think that right now we have embedded in various
places the idea that VARTAG_EXTERNAL is the only thing that shows up
on disk, and we'd need to track down all such places and adjust them
if we add other VARTAG types in the future. Depending on how it is to
be used, adding a new ToastCompressionId might be less work. However,
I don't think we can use the possibility of adding a new VARTAG value
as a reason why it's OK to use up the last possible ToastCompressionId
value for something non-extensible.

For projects like this, the details matter a lot. If the goal is to
add a new compression type that behaves like the existing compression
types, more or less, then I think we should allocate the last
ToastCompressionId bit to mean "some other compression ID" and add a
1-byte header that says which one is in use. But if the new feature
being added is enough different from the other compression methods,
then it might be better to do it in some other way e.g. a new VARTAG.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

---

From: Michael Paquier <[email-masked]>
Date: Wed, 24 May 2023 09:17:16 +0900

> then it might be better to do it in some other way e.g. a new VARTAG.

Agreed.  While the compression argument and the possibility to add
more options to toast pointers are very appealing, FWIW, I'd like to
think that the primary target is the 4-byte OID assignment limit of
where backends loop infinitely until a OID can be found, which can be
a real pain for users with a large number of blobs or just enough
toast data to trigger it.

Saying that even if I sent the patch for zstd on toast..
--
Michael

---

From: Nikita Malakhov <[email-masked]>
Date: Wed, 24 May 2023 12:16:50 +0300

Hi!

I've made a WIP patch that uses 64-bit TOAST value ID instead of 32-bit,
and sent it as a part of discussion, but there was no feedback on such a
solution. There was a link to that discussion at the top of this thread.

Also, I have to note that, based on our work on Pluggable TOAST - extending
TOAST pointer with additional structures would require review of the logical
replication engine, currently it is not suitable for any custom TOAST
pointers.
Currently we have no final solution for problems with logical replication
for
custom TOAST pointers.

-- 
Regards,
Nikita Malakhov
Postgres Professional
The Russian Postgres Company
https://postgrespro.ru/