Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1omvRW-00063g-6h for pgsql-hackers@arkaria.postgresql.org; Mon, 24 Oct 2022 11:16:30 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1omvRV-0008D6-2K for pgsql-hackers@arkaria.postgresql.org; Mon, 24 Oct 2022 11:16:29 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1omvRU-0008Cx-Kr for pgsql-hackers@lists.postgresql.org; Mon, 24 Oct 2022 11:16:28 +0000 Received: from mail-ed1-x52e.google.com ([2a00:1450:4864:20::52e]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1omvRR-0001F2-RX for pgsql-hackers@postgresql.org; Mon, 24 Oct 2022 11:16:27 +0000 Received: by mail-ed1-x52e.google.com with SMTP id m15so29828417edb.13 for ; Mon, 24 Oct 2022 04:16:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=voz6jobmIO944IdL1PfXw5zdG3Zz80qY1zfdHZto2VU=; b=O2IFIWaeu6zt2tL3jhjrdtMJCW4+2WfGSInczTp+f/Q5QLfiE7vZXdTMn94oEH/w1x X6ZnksV5qvRb1PehX0w9sCNs0cuzo+vTVFp80kxkDh7FIfPDK0OH40jNmp/2b1XxRk0s PVlZhBnMbrnV42Ks0Uy8E1dbZdINO+MpZ+vi3cj7NedlT4ARxe1XtMquNWyDAD27bo76 jjBt7yTNBSrnZ0M1Rcn3bMDWNtM0zSFEZhkbsf7thtfZvZr63p1vzACUK7LpKzxqBcb2 RKoYe2p7p29ntMJXQJv+G20BKE3uuBq4rwBmK0WubOyuJASr0LdrQM4Ki2vEjrLoNffi KEoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=voz6jobmIO944IdL1PfXw5zdG3Zz80qY1zfdHZto2VU=; b=ArSFyLrsHzRV+dMLRgRUTdFfAAM6NJTjKbYMfVDFNgU3WUNkElcKHc3tA+u4yGgGH4 nev04wLIonnATmE4vGrR6M8WWMn6Q3Y+m5uI0ilmkMMalGbGcekbNKhJ9z7fqOxl7Z70 WAFh+bz0VOqx1qHXofWDt2THX1w/wfmSbbM29qnnTJtDibrnVWBrRyyHekahi/jqU74G cxdNuv7f5/ywozNXC80XASnz2SflEfwo533NTSFUuHC9eDihR0AsS7kt7fhOA74hF0sQ uhwtXGnOmIMxMsG+Zzxe9/xtisZUSWZXMEjatXForvm2x93Zs5TM9KPUh9bKEMt6l/yY TARg== X-Gm-Message-State: ACrzQf2tn2ks+WUw8MnNLqH+yQ4dgbA40eb6HqZu7fB2xvbjtAkPU5SE y62febahaapM+Fswwf/8R4hT0NufF0oqO/vsQPc= X-Google-Smtp-Source: AMsMyM4gHxwkamIaJGSrHE2FyOTolgJZhuGpNn+flN3XSaX7L9nbm6Kkdx4q9XVSTMPG1AF1UG1Hqwls+P3FQic10xc= X-Received: by 2002:a05:6402:1ac4:b0:460:23b2:f680 with SMTP id ba4-20020a0564021ac400b0046023b2f680mr21621775edb.219.1666610183993; Mon, 24 Oct 2022 04:16:23 -0700 (PDT) MIME-Version: 1.0 References: <20220802153704.lhrqpqdrrlpandh7@awork3.anarazel.de> In-Reply-To: From: Nikita Malakhov Date: Mon, 24 Oct 2022 14:16:07 +0300 Message-ID: Subject: Re: Pluggable toaster To: Aleksander Alekseev Cc: pgsql-hackers , Jacob Champion , Andres Freund , Matthias van de Meent , Robert Haas , Greg Stark , Teodor Sigaev Content-Type: multipart/alternative; boundary="000000000000ff005a05ebc5ebb4" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000ff005a05ebc5ebb4 Content-Type: text/plain; charset="UTF-8" Hi! >I don't argue with most of what you say. I am just pointing out the >reason why the chosen approach "N TOASTers x M TableAMs" will not >work: We assume that TAM used in custom Toaster works as it is should work, and leave TAM internals to this TAM developer - say, we do not want to change internals of Heap AM. We don't want to create some kind of silver bullet. There are already existing and widely-known (from production environments) problems with TOAST mechanics, and we suggest not too complex way to solve them. As I mentioned before, Pluggable TOAST does not change Heap AM, it is not minded to change TAMs. >This is what I meant above when talking about the framework for >simplifying this task: That's a kind of generalizing custom TOAST implementation. It is very good intention, but keep in mind that different kinds of data require very different approach to external storage - say, JSON TOAST works with maps of keys and values, super binary object (experimental name) does not work with internals of TOASTed data except searching. But, we thought about that too and reusable code resides in toast_internals.c source - any custom Toaster working with Heap could use it's insert, update and fetch methods, but deal with data on it's own. Even with the general framework there must be a common interface which would be the entry point for those custom methods developed with the framework. That's what the TOAST API is - just an interface that all custom TOAST implementations use to have a common entry point from any TAM, with syntax support to plug in custom TOAST implementations from the SQL. No less, but no more. Moreover, our patches show that even Generic (default) TOAST implementation could still be left as-is, without necessity to route it via our API, though it is logically wrong because common API is meant to be common for all TOAST implementations without exceptions. Have I answered your question? Please don't hesitate to point to any unclear parts, I'd be glad to explain that. The main idea in TOAST API is very elegant and light, and it is designed alike to Pluggable Storage (Table AM API). On Mon, Oct 24, 2022 at 12:10 PM Aleksander Alekseev < aleksander@timescale.com> wrote: > Hi Nikita, > > I don't argue with most of what you say. I am just pointing out the > reason why the chosen approach "N TOASTers x M TableAMs" will not > work: > > > Don't you think that this is an arguable design decision? Basically > > all we know about the underlying TableAM is that it stores tuples > > _somehow_ and that tuples have TIDs [1]. That's it. We don't know if > > it even has any sort of pages, whether they are fixed in size or not, > > whether it uses shared buffers, etc. It may not even require TOAST. > > [...] > > Also I completely agree with: > > > Implementing another Table AM just to implement another TOAST strategy > seems too > > much, the TAM API is very heavy and complex, and you would have to add > it as a contrib. > > This is what I meant above when talking about the framework for > simplifying this task: > > > It looks like the idea should be actually turned inside out. I.e. what > > would be nice to have is some sort of _framework_ that helps TableAM > > authors to implement TOAST (alternatively, the rest of the TableAM > > except for TOAST) if the TableAM is similar to the default one. > > From the user perspective it's much easier to think about one entity - > TableAM, and choosing from heapam_with_default_toast and > heapam_with_different_toast. > > From the extension implementer point of view creating TableAMs is a > difficult task. This is what the framework should address. Ideally the > interface should be as simple as: > > CreateParametrizedDefaultHeapAM(SomeTOASTSubstitutionObject, ...other > arguments, in the future...) > > Where the extension author should be worried only about an alternative > TOAST implementation. > > I think at some point such a framework may address at least one more > issue we have - an inability to change the page size on the table > level. As it was shown by Tomas Vondra [1] the default 8 KB page size > can be suboptimal depending on the load. So it would be nice if the > user could change it without rebuilding PostgreSQL. Naturally this is > out of scope of this particular patchset. I just wanted to point out > opportunities we have here. > > [1]: > https://www.postgresql.org/message-id/flat/b4861449-6c54-ccf8-e67c-c039228cdc6d%40enterprisedb.com > > -- > Best regards, > Aleksander Alekseev > -- Regards, Nikita Malakhov Postgres Professional https://postgrespro.ru/ --000000000000ff005a05ebc5ebb4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi!

>I don't ar= gue with most of what you say. I am just pointing out the
>reason why= the chosen approach "N TOASTers x M TableAMs" will not
>wo= rk:

We assume that TAM used in custom Toaster = works as it is should work,
and leave TAM internals to this TAM d= eveloper - say, we do not want to
change internals of Heap AM.

We don't want to create some kind of silver bull= et. There are already existing
and widely-known (from production = environments) problems with TOAST
mechanics, and we suggest not t= oo complex way to solve them.

As I mentioned befor= e, Pluggable TOAST does not change Heap AM, it is
not minded to c= hange TAMs.=C2=A0

>This is what I meant above w= hen talking about the framework for
>simplifying this task:
=

That's a kind of generalizing custom TOAST implemen= tation. It is very
good intention, but keep in mind that differen= t kinds of data require very
different approach to external stora= ge - say, JSON TOAST works with
maps of keys and values, super bi= nary object (experimental name) does
not work with internals of T= OASTed data except searching. But, we thought
=C2=A0about that to= o and reusable code resides in toast_internals.c source - any
cus= tom Toaster working with Heap could use it's insert, update and fetch
methods, but deal with data on it's own.

<= div>Even with the general framework there must be a common interface which<= /div>
would be the entry point for those custom methods developed with = the
framework. That's what the TOAST API is - just an interfa= ce that all custom
TOAST implementations use to have a common ent= ry point from any TAM,
with syntax support to plug in custom TOAS= T implementations from the SQL.
No less, but no more.

Moreover, our patches show that even Generic (default= ) TOAST implementation
could still be left as-is, without necessi= ty to route it via our API, though it is logically
wrong because = common API is meant to be common for all TOAST implementations
wi= thout exceptions.

Have I answered your question? P= lease don't hesitate to point to any unclear
parts, I'd b= e glad to explain that.

The main idea in TOAST API= is very elegant and light, and it is designed alike
to Pluggable= Storage (Table AM API).

On Mon, Oct 24, 2022 at 12:10 PM Aleksander A= lekseev <aleksander@timescal= e.com> wrote:
Hi Nikita,

I don't argue with most of what you say. I am just pointing out the
reason why the chosen approach "N TOASTers x M TableAMs" will not=
work:

> Don't you think that this is an arguable design decision? Basicall= y
> all we know about the underlying TableAM is that it stores tuples
> _somehow_ and that tuples have TIDs [1]. That's it. We don't k= now if
> it even has any sort of pages, whether they are fixed in size or not,<= br> > whether it uses shared buffers, etc. It may not even require TOAST. > [...]

Also I completely agree with:

> Implementing another Table AM just to implement another TOAST strategy= seems too
> much, the TAM API is very heavy and complex, and you would have to add= it as a contrib.

This is what I meant above when talking about the framework for
simplifying this task:

> It looks like the idea should be actually turned inside out. I.e. what=
> would be nice to have is some sort of _framework_ that helps TableAM > authors to implement TOAST (alternatively, the rest of the TableAM
> except for TOAST) if the TableAM is similar to the default one.