Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1omx3n-0003Kk-1P for pgsql-hackers@arkaria.postgresql.org; Mon, 24 Oct 2022 13:00:07 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1omx3l-0002IS-Cf for pgsql-hackers@arkaria.postgresql.org; Mon, 24 Oct 2022 13:00:05 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1omx3k-0002IJ-Ss for pgsql-hackers@lists.postgresql.org; Mon, 24 Oct 2022 13:00:05 +0000 Received: from mail-ed1-x52f.google.com ([2a00:1450:4864:20::52f]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1omx3i-000289-16 for pgsql-hackers@postgresql.org; Mon, 24 Oct 2022 13:00:03 +0000 Received: by mail-ed1-x52f.google.com with SMTP id r14so30927542edc.7 for ; Mon, 24 Oct 2022 06:00:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=C3J8wosBIaORSy8bZMsFhXSK5bW6Vlijs2ZxNEc812U=; b=AG+GDyUaQcnfpPsW9Xsbbe6FEyJRfu7UlxaTBDV+rfeCTypHX1gVnlhVVwCLwDQhbT AtqxnDg4FR2JovhGx0muz63RZKF6eDbgISSJsldwxxavtphnFgpMrW+qlVnwYT3WMx1r CT4Jh/qASmVn7rSG0hLMa1u3E4IXv5V3I/8S9iAsm5rSr9slXj5lHfWQyFHTDgaf7vv6 pXhZOJlHxP7KN+QyTbzKAZ077WiZnPyTK5h4b1CvXa0O5oED6QXlppPSatcMZ/8MWbUg i492dBkY+LGGXn1mYqLV9yLlVO7JDVPhZ4+Sj0sFymUAe+gDxTnQVEXfRqGO5/l2Ah4Z YKUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=C3J8wosBIaORSy8bZMsFhXSK5bW6Vlijs2ZxNEc812U=; b=dyCFF8n5pW97YxykZmZrm3F5rsncMRBaeZIw7rXV2Utm4lP+PTKnlDmU7bPffjZhAE Kj6GPT1X3Y0c7NcDjKBunN52efJZL8u/bCa/4hBpE2KpOUL0tCFynXitrPifnxf6evbH vOhJzAiJEaLpzrVBtPAtcWSA2TYKebZOI9z7J2uEYcIe30XsqnPDjDY99VC3zJKVyTaX CkZeUjPmyJ0PG6YibsN4/wEyCaojemhzDbpsPHgly5v6LvQfS6X8U78WR9NJT0oVl5CU KAaqHGfWV481M96oya97z1ExOOGjOimW4gZi2FuztVt/DspE8vtNMkfX8AP7mLchlqmK i8ug== X-Gm-Message-State: ACrzQf2oxX3xqlaZ37ImaclsxJJ2iubWnh+Mdyi/H1wrok8NfDWFZEO7 hbvdX8AN6fMMsZEX6AXxLhyTDv1b5ScS51d2gqI= X-Google-Smtp-Source: AMsMyM50oLa3EED9q5ZHTvldq8QbLNAX3jGyjFwVaYlwfnZ296iiCuIrY1PyVoMDgkL3OuJlB+RhmZoYogLH79lh1R4= X-Received: by 2002:a17:907:9493:b0:796:1166:70c4 with SMTP id dm19-20020a170907949300b00796116670c4mr20962979ejc.59.1666616400500; Mon, 24 Oct 2022 06:00:00 -0700 (PDT) MIME-Version: 1.0 References: <20220802153704.lhrqpqdrrlpandh7@awork3.anarazel.de> In-Reply-To: From: Nikita Malakhov Date: Mon, 24 Oct 2022 15:59:44 +0300 Message-ID: Subject: Re: Pluggable toaster To: Aleksander Alekseev Cc: pgsql-hackers , Jacob Champion , Andres Freund , Matthias van de Meent , Robert Haas , Greg Stark , Teodor Sigaev Content-Type: multipart/alternative; boundary="000000000000875cca05ebc75e5a" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000875cca05ebc75e5a Content-Type: text/plain; charset="UTF-8" Hi, >This is exactly the point. In order to not to create a silver bullet, >TOASTers should be limited to a single TableAM. The one we know uses >pages of a known fixed size, the one that actually requires TOAST >because pages are relatively small, etc. Currently all our TOAST implementations use Heap AM, except ones that use external (truly external, i.e. files outside DB) storage. Using Table AM Routine and routing AM methods calls via it is a topic for further discussion, if Pluggable TOAST will be committed. And even then it would be an open issue. >I believe this is the source of misunderstanding. Note that not _any_ >TableAM needs TOAST to begin with. As an example, if one chooses to >implement a column-organized TableAM that stores all text/bytea >attributes in a separate dictionary file this person doesn't need >TOAST and doesn't want to be constrained by the need of choosing one. >For this reason the "N TOASTers x M TableAMs" approach is >architecturally broken. TOAST implementation is not necessary for Table AM. And TOAST API is just an optional open interface - SET TOASTER is an option for CREATE/ALTER TABLE command. In previous discussion we haven't mentioned an approach "N TOASTers x M TableAMs". >To clarify: is an ability to specify TOASTers for given columns and/or >types also part of the plan? For table columns it is already supported by the syntax part of the TOAST API. For data types we reserved the validation part of the API, but this support is still a subject for discussion, although we think it will be very handy for DB users, like we issue something like: CREATE TYPE ... TOASTER=jsonb_toaster ... ; or ALTER TYPE JSONB SET TOASTER jsonb_toaster; and do not have to set special toaster for jsonb column each time we create or alter a table with it. >No. To be honest, it looks like you are merely discarding most/any >feedback the community provided so far. Very sorry to read that. Almost all of the requests in this discussion have been taken into account in patches, and the most serious one - I mean pg_attribute expansion which was mentioned by Tom Lane and Robert Haas - is being fixed right now and will be ready very soon. >I really think that pluggable TOASTers would be a great feature. >However if the goal is to get it into the core I doubt that we are >going to make much progress with the current approach. We hope we will. This feature is very demanded by end-users, and will be even more as time goes by - current TOAST limitations and how they affect DBMS performance is a serious drawback in comparison to competitors. On Mon, Oct 24, 2022 at 2:55 PM Aleksander Alekseev < aleksander@timescale.com> wrote: > Hi Nikita, > > > >I don't argue with most of what you say. I am just pointing out the > > >reason why the chosen approach "N TOASTers x M TableAMs" will not > > >work: > > > > We assume that TAM used in custom Toaster works as it is should work, > > and leave TAM internals to this TAM developer - say, we do not want to > > change internals of Heap AM. > > > > We don't want to create some kind of silver bullet. > > This is exactly the point. In order to not to create a silver bullet, > TOASTers should be limited to a single TableAM. The one we know uses > pages of a known fixed size, the one that actually requires TOAST > because pages are relatively small, etc. > > > That's what the TOAST API is - just an interface that all custom > > TOAST implementations use to have a common entry point from any TAM, > > [...] > > I believe this is the source of misunderstanding. Note that not _any_ > TableAM needs TOAST to begin with. As an example, if one chooses to > implement a column-organized TableAM that stores all text/bytea > attributes in a separate dictionary file this person doesn't need > TOAST and doesn't want to be constrained by the need of choosing one. > > For this reason the "N TOASTers x M TableAMs" approach is > architecturally broken. > > > keep in mind that different kinds of data require very > > different approach to external storage - say, JSON TOAST works with > > maps of keys and values, [...] > > To clarify: is an ability to specify TOASTers for given columns and/or > types also part of the plan? > > > Have I answered your question? Please don't hesitate to point to any > unclear > > parts, I'd be glad to explain that. > > No. To be honest, it looks like you are merely discarding most/any > feedback the community provided so far. > > I really think that pluggable TOASTers would be a great feature. > However if the goal is to get it into the core I doubt that we are > going to make much progress with the current approach. > > -- > Best regards, > Aleksander Alekseev > -- Regards, Nikita Malakhov Postgres Professional https://postgrespro.ru/ --000000000000875cca05ebc75e5a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

>This is exactl= y the point. In order to not to create a silver bullet,
>TOASTers sho= uld be limited to a single TableAM. The one we know uses
>pages of a = known fixed size, the one that actually requires TOAST
>because pages= are relatively small, etc.

Currently all our = TOAST implementations use Heap AM, except ones
that use external = (truly external, i.e. files outside DB) storage. Using Table AM
R= outine and routing AM methods calls via it is a topic for further discussio= n,
if Pluggable TOAST will be committed. And even then it would b= e an open issue.

>I believe this is the source = of misunderstanding. Note that not _any_
>TableAM needs TOAST to begi= n with. As an example, if one chooses to
>implement a column-organize= d TableAM that stores all text/bytea
>attributes in a separate dictio= nary file this person doesn't need
>TOAST and doesn't want to= be constrained by the need of choosing one.

>For this reason the= "N TOASTers x M TableAMs" approach is
>architecturally bro= ken.

TOAST implementation is not necessary for= Table AM. And TOAST API is just
an optional open interface - SET= TOASTER is an option for CREATE/ALTER=C2=A0
TABLE command. In pr= evious discussion we haven't mentioned an approach
"N TO= ASTers x M TableAMs".

>To clarify: is = an ability to specify TOASTers for given columns and/or
>types also p= art of the plan?

For table columns it is alrea= dy supported by the syntax part of the TOAST API.
For data types = we reserved the validation part of the API, but this support is still a
subject for discussion, although we think it will be very handy for = DB users, like
we issue something like:
CREATE TYPE ...= TOASTER=3Djsonb_toaster ... ;
or
ALTER TYPE JSONB SET = TOASTER jsonb_toaster;
and do not have to set special toaster for= jsonb column each time we create
or alter a table with it.
=

>No. To be honest, it looks like you are merely disc= arding most/any
>feedback the community provided so far.

Very = sorry to read that. Almost all of the requests in this discussion have been= taken
into account in patches, and the most serious one - I mean= pg_attribute expansion
which was mentioned by Tom Lane and Rober= t Haas - is being fixed right now and
will be ready very soon.

>I really think that pluggable TOASTers would be a great fea= ture.
>However if the goal is to get it into the core I doubt that we= are
>going to make much progress with the current approach.

We hope we will. This feature is very demanded by end-= users, and will be even more
as time goes by - current TOAST limi= tations and how they affect DBMS performance is
a serious drawbac= k in=C2=A0comparison to competitors.

On Mon, Oct 24, 2022 at 2:55 PM A= leksander Alekseev <aleksand= er@timescale.com> wrote:
Hi Nikita,

> >I don't argue with most of what you say. I am just pointing ou= t the
> >reason why the chosen approach "N TOASTers x M TableAMs"= will not
> >work:
>
> We assume that TAM used in custom Toaster works as it is should work,<= br> > and leave TAM internals to this TAM developer - say, we do not want to=
> change internals of Heap AM.
>
> We don't want to create some kind of silver bullet.

This is exactly the point. In order to not to create a silver bullet,
TOASTers should be limited to a single TableAM. The one we know uses
pages of a known fixed size, the one that actually requires TOAST
because pages are relatively small, etc.

> That's what the TOAST API is - just an interface that all custom > TOAST implementations use to have a common entry point from any TAM, > [...]

I believe this is the source of misunderstanding. Note that not _any_
TableAM needs TOAST to begin with. As an example, if one chooses to
implement a column-organized TableAM that stores all text/bytea
attributes in a separate dictionary file this person doesn't need
TOAST and doesn't want to be constrained by the need of choosing one.
For this reason the "N TOASTers x M TableAMs" approach is
architecturally broken.

> keep in mind that different kinds of data require very
> different approach to external storage - say, JSON TOAST works with > maps of keys and values, [...]

To clarify: is an ability to specify TOASTers for given columns and/or
types also part of the plan?

> Have I answered your question? Please don't hesitate to point to a= ny unclear
> parts, I'd be glad to explain that.

No. To be honest, it looks like you are merely discarding most/any
feedback the community provided so far.

I really think that pluggable TOASTers would be a great feature.
However if the goal is to get it into the core I doubt that we are
going to make much progress with the current approach.

--
Best regards,
Aleksander Alekseev


--
Regards,
Nikita Malakhov
Postgres Professional=C2=A0
https://postgrespro.ru/
--000000000000875cca05ebc75e5a--