Feedback-ID: i0fe9450f:Fastmail
Date: Sat, 5 Jul 2025 10:11:32 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Nikita Malakhov <hukutoc@gmail.com>
Cc: Hannu Krosing <hannuk@google.com>,
	Postgres hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Support for 8-byte TOAST values (aka the TOAST infinite loop
 problem)
Message-ID: <aGh7xCegTRJCdQ9q@paquier.xyz>
References: <aFOnKHG7Wn-Srnpv@paquier.xyz>
 <CAMT0RQQvki5v5FPZMXSR3cn4Y2gMUUtEYb+Hf_OQmqUbeBrUeQ@mail.gmail.com>
 <CAN-LCVNsE4x0k11ZRWvU4ySTbe98fwA16qzV7p8dxogWnD5Jng@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="goDe/9YiYu8rYpdA"
Content-Disposition: inline
In-Reply-To: 
 <CAN-LCVNsE4x0k11ZRWvU4ySTbe98fwA16qzV7p8dxogWnD5Jng@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/aGh7xCegTRJCdQ9q%40paquier.xyz>
Precedence: bulk


--goDe/9YiYu8rYpdA
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Fri, Jul 04, 2025 at 02:38:34PM +0300, Nikita Malakhov wrote:
> Hannu, we'd already made an attempt to extract the TOAST functionality as
> API and make it extensible and usable by other AMs in [1], the patch
> set was met calmly but we still have some hopes on it.

Yeah, it's one of these I have studied, and just found that
overcomplicated, preventing us from moving on with a simpler proposal,
because I care about two things first:
- More compression methods, with more meta-data, but let's just add
more vartag_external for that once/if they're really required.
- Enlarge optionally to 8-byte values.
So I really want to stress about these two points, nothing else for
now, echoing from the feedback from 2022 and the fact that all
proposals done after that lacked a simple approach.

IMO, we would live fine enough, *if* being able to plug in a pluggable
TOAST engine makes sense, if we just limit ourselves with an external
interface.  We could allow backends to load their own vartag_external
with their own set of callbacks like the ones I am proposing here, so
as we can translate from/to a Datum in heap (or a different table AM)
to an external source, with the backend able to understand what this
external source should be.  The key is to define a structure good
enough for the backend (toast_external_data in the patch).  So to
answer your and Hannu's question: I had the case of different table
AMs in mind with an interface able to plug into it, yes.  And yes, I
have designed the patch set with this in scope.  Now there's also a
data type component to that, so that's assuming that a table AM would
want to rely on a varlena to store this data externally, somewhere
else that may not be exactly TOAST, still we want an OID and a value
to be able to retrieve this external value, and we want to store this
external OID and this value (+extra like a compression method and
sizes) in a Datum of the main relation file.

FYI, the patch set posted on this thread is not the latest one.  I
have a v2, posted on this branch, where I have reordered things:
https://github.com/michaelpq/postgres/tree/toast_64bit_v2

The refactoring to the new toast_external_data with its callbacks is
done first, and the new vartag_external with 8-byte value support is
added on top of that.  There were still two things I wanted to do, and
could not get down to it because I've spent my last week or so
working on other's stuff so I lacked room:
- Evaluate the cost of the transfer layer to toast_external_data.  The
worst case I was planning to work with is a non-compressed data stored
in TOAST, then check profiles with the the detoasting path by grabbing
slices of the data with pgbench and a read-only query.  The
write/insert path is not going to matter, the detoast is.  The
reordering is actually for this reason: I want to see the effect of
the new interface first, and this needs to happen before we even
consider the option of adding 8-byte values.
- Add a callback for the value ID assignment.  I was hesitating to add
that when I first backed on the patch but I think that's the correct
design moving forward, with an extra logic to be able to check if an
8-byte value is already in use in a relation, as we do for OID
assignment, but applied to the Toast generator added to the patch.
The backend should decide if a new value is required, we should not
decide the rewrite cases in the callback.

There is a second branch that I use for development, force-pushing to
it periodically, as well:
https://github.com/michaelpq/postgres/tree/toast_64bit
That's much dirtier, always WIP, just one of my playgrounds.
--
Michael

--goDe/9YiYu8rYpdA
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmhoe8QACgkQnvQgOdby
QH35KRAAonb3ygwc8HVwP7GoCitW+znrRdHKr9/jU1jr1w9LRnhJPGYbnxuc2vM1
7lyLKuJ4s1ee/Y7dz0cyYcO350G6Exs/QEKX8g1sbQmW8L4t3/JiVlYNEn07PwLN
9EUFFwWX0Il49QG84qwnwbFpVSwPjVVypq8QywR9WgRO8mzJflrwihgk+CqNBp8c
UKxID2pB1NL+8OLbK4zkgRYLSOiQMHye/2EC24w4f5cKX2rUc0BVoJQxbVbe2r29
eep7P/RJX5tLBD5U2H9JQ7KeqtcVEyKQReXT8Ibwmzna3SM4sgDhPREwGqCEn/02
WuAcNhNVLoA01yvMB8TNBjG02GB9NgXBRUYJ1ioohUm6Fr5CCAKYjnE+Uk9btEjB
/ZArPaXQbvGGIJH1toReqrbGspcYQOdFV9Jh8Hl2NUgOzMv6sNEonQeewqQXdoU+
y/42RiBMg7TRGBrxRHOlZJmBNG+pUc9fnDEuG4R6h/UEVrGtYfTpImTvenZFqUu7
cWmuY/NQznyvthPncejAiqNuu1r5AAn7hJAuDUyRy2hoZ7S//IWE7MUvO8IMBwl6
9IMY1HmI8LmLeYS/IHhRyhAI4cY6BxbefmA42fE792xqKcDQjdvjMEIk1Ntsq5y/
l7uY0sp+2jSi64GP1yLJEQwa55Ys4LmpqfqJY4Rl4Pe+0nQfxzk=
=xhRu
-----END PGP SIGNATURE-----

--goDe/9YiYu8rYpdA--