MIME-Version: 1.0
References: <CAN-LCVMq2X=fhx7KLxfeDyb3P+BXuCkHC0g=9GF+JD4izfVa0Q@mail.gmail.com>
 <CAJ7c6TNAYyeMYKVkiwOZChy7UpE_CkjpYOk73gcWTXMkLkEyzw@mail.gmail.com> <CAEze2WibtT+EdGscamqDm3+7qauWi_S6nVUhbK_qg1rkqN_48w@mail.gmail.com>
In-Reply-To: <CAEze2WibtT+EdGscamqDm3+7qauWi_S6nVUhbK_qg1rkqN_48w@mail.gmail.com>
From: Aleksander Alekseev <aleksander@timescale.com>
Date: Thu, 18 May 2023 15:57:14 +0300
Message-ID: <CAJ7c6TOpLG3RxVbkvGoO4DB_-Y3YLGkynH-nfWAoXFcHFpkbmg@mail.gmail.com>
Subject: Re: RFI: Extending the TOAST Pointer
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>, Nikita Malakhov <hukutoc@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Archived-At: <https://www.postgresql.org/message-id/CAJ7c6TOpLG3RxVbkvGoO4DB_-Y3YLGkynH-nfWAoXFcHFpkbmg%40mail.gmail.com>
Precedence: bulk

Hi,

> No, that's inaccurate. The complete on-disk representation of a varatt is
>
> {
>     uint8       va_header;      /* Always 0x80 or 0x01 */
>     uint8       va_tag;         /* Type of datum */
>     char        va_data[FLEXIBLE_ARRAY_MEMBER]; /* Type-dependent
> data, for toasted values that's currently only a varatt_external */
> } varattrib_1b_e;
>
> With va_tag being filled with one of the vartag_external values:
>
> typedef enum vartag_external
> {
>     VARTAG_INDIRECT = 1,
>     VARTAG_EXPANDED_RO = 2,
>     VARTAG_EXPANDED_RW = 3,
>     VARTAG_ONDISK = 18
> } vartag_external;
>
> This enum still has many options to go before it exceeds the maximum
> of the uint8 va_tag field. Therefore, I don't think we have no disk
> representations left, nor do I think we'll need to add another option
> to the ToastCompressionId enum.
> As an example, we can add another VARTAG option for dictionary-enabled
> external toast; like what the pluggable toast patch worked on. I think
> we can salvage some ideas from that patch, even if the main idea got
> stuck.

The problem here is that the comments are ambiguous regarding what to
call "TOAST pointer" exactly. I proposed a patch for this but it was
not accepted [1].

So the exact on-disk representation of a TOAST'ed value (for
little-endian machines) is:

0b00000001, 18 (va_tag), (varatt_external here)

Where 18 is sizeof(varatt_external) + 2, because the length includes
the length of the header.

I agree that va_tag can have another use. But since we are going to
make varatt_external variable in size (otherwise I don't see how it
could be really **extendable**) I don't think this is the right
approach.

Also I agree that this particular statement is incorrect:

> This will allow us to extend the pointers indefinitely.

varatt_external is going to be limited to 255. But it seems to be a
reasonable limitation for the nearest 10-20 years or so.

[1]: https://commitfest.postgresql.org/39/3820/

-- 
Best regards,
Aleksander Alekseev