Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1offR3-0003X1-FF for pgsql-hackers@arkaria.postgresql.org; Tue, 04 Oct 2022 10:46:01 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1offR1-0004Ja-78 for pgsql-hackers@arkaria.postgresql.org; Tue, 04 Oct 2022 10:45:59 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1offR0-0004BQ-KX for pgsql-hackers@lists.postgresql.org; Tue, 04 Oct 2022 10:45:58 +0000 Received: from mail-ej1-x635.google.com ([2a00:1450:4864:20::635]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1offQt-0004ES-Ju for pgsql-hackers@postgresql.org; Tue, 04 Oct 2022 10:45:57 +0000 Received: by mail-ej1-x635.google.com with SMTP id 13so27982108ejn.3 for ; Tue, 04 Oct 2022 03:45:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=0YCGJmRv7njZ4FTTQHl5pioISO7UVVhzmqLOj8Rn3Is=; b=mo6zy22pPSW/s8epVWXDexpaXO/VKDnT5LoYN3J9h59tYxkRY10CfVe7pyTfqFCU4u A4O/Mxu2rozieOKie9za9IeYVlOAvNPlKFJT6OWdKuQ6l2KsythGgMQs9JxadAwx6o6p TJk3uVSdwV+y8QiuWXx3mlz+l2/KZYUeRfp+DvNI9LMRk9dd5GzVijTnDwrjFpYcy6nY kUvZuhyrbBmb2LoCdw16mC9/EBLB+1PY+R7G8q/de9DUcUUJOMLLlxB2mVQFKP3ebV6J bFKXy6uR5YMkk6SGA6EyYiRQ+SY2mWRtii8Mhp/N2/y3gEYvdqNFGwFIqiNBc7AsHtlX JMFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=0YCGJmRv7njZ4FTTQHl5pioISO7UVVhzmqLOj8Rn3Is=; b=syrla20jYPWnE2rxqeZtxeInvN72mHiLH6pfb+syDs3f7DJV8zcKnd/F/imvxJppcI qmOHDwxGI6MBsHitiMsAzSL7oRMVAMablNJ0NRZfI40F0pBCe5h7H5piv2u542gJx+pT znycPkWl94+KG7lMMnMLgK+CpEyXFKriTw0p/6keSpPvQGnS0bsm6+AhDeBUAi9o5cyD 76pjc6DDlnHiChyO66zSXL7LNKrEu5g9o5VFrK+raA7cTT38fcdWtevcpWXIsJZnwcJ/ Y3MfkOQqSwdzhgrE/4h1DZ1VF81yxLdXiB0PfdN32YoTAjAmJAVU9EIMjpZRgs/pIQ1L UjoQ== X-Gm-Message-State: ACrzQf1251kc6tSgaMCf1Ma9dl3xqeI+jIXtGdaBHWf/A0YfsylG99Vg HCmyfspCS7OPLVcV93EoDfd6S/C5/MFzaHEO5KI= X-Google-Smtp-Source: AMsMyM6DInnzEXTyr7CVc6jMjzeHbNRrPf6nq2ebSNVYwsgiJBkvnbQ6kb6/MzOlLBShPpX9BGsgZgyXA0FyEq5fN2k= X-Received: by 2002:a17:907:724a:b0:782:3754:ecb3 with SMTP id ds10-20020a170907724a00b007823754ecb3mr19090202ejc.282.1664880349943; Tue, 04 Oct 2022 03:45:49 -0700 (PDT) MIME-Version: 1.0 References: <20220802153704.lhrqpqdrrlpandh7@awork3.anarazel.de> In-Reply-To: From: Nikita Malakhov Date: Tue, 4 Oct 2022 13:45:38 +0300 Message-ID: Subject: Re: Pluggable toaster To: Jacob Champion Cc: Aleksander Alekseev , pgsql-hackers , Andres Freund , Matthias van de Meent , Robert Haas , Greg Stark , Teodor Sigaev Content-Type: multipart/alternative; boundary="000000000000da1fc805ea33296b" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000da1fc805ea33296b Content-Type: text/plain; charset="UTF-8" Hi hackers! Now cfbot is happy, but there were warnings due to recent changes in PointerGetDatum function, so here's corrected patchset. Patchset consists of: v20-0001-toaster-interface.patch - Pluggable TOAST API interface along with reference TOAST mechanics - new API is introduced but reference TOAST is still unchanged; v20-0002-toaster-default.patch - Default TOAST re-implemented using Toaster API - reference TOAST is re-implemented via new API; v20-0003-toaster-docs.patch - Pluggable TOAST API documentation package Actual GitHub branch resides at https://github.com/postgrespro/postgres/tree/toasterapi_clean On Tue, Oct 4, 2022 at 1:02 AM Nikita Malakhov wrote: > Hi hackers! > Cfbot failed in meson build with previous patchsets, so I've rebased them > onto the latest master and added necessary meson build info. > > Patchset consists of: > v19-0001-toaster-interface.patch - Pluggable TOAST API interface along > with reference TOAST mechanics - new API is introduced but > reference TOAST is still unchanged; > v19-0002-toaster-default.patch - Default TOAST re-implemented using > Toaster API - reference TOAST is re-implemented via new API; > v19-0003-toaster-docs.patch - Pluggable TOAST API documentation package > > Actual GitHub branch resides at > https://github.com/postgrespro/postgres/tree/toasterapi_clean > > On Tue, Sep 27, 2022 at 12:26 AM Nikita Malakhov > wrote: > >> Hi, >> Meson build for the patchset failed, meson build files attached and >> README/Doc package >> reworked with more detailed explanation of virtual function table along >> with other corrections. >> >> On Sun, Sep 25, 2022 at 1:41 AM Nikita Malakhov >> wrote: >> >>> Hi hackers! >>> Last patchset has an invalid patch file - v16-0003-toaster-docs.patch. >>> Here's corrected patchset, >>> sorry for the noise. >>> >>> On Sat, Sep 24, 2022 at 3:50 PM Nikita Malakhov >>> wrote: >>> >>>> Hi hackers! >>>> >>>> Cfbot is still not happy with the patchset, so I'm attaching a rebased >>>> one, rebased onto the current >>>> master (from today). The third patch contains documentation package, >>>> and the second one contains large >>>> README.toastapi file providing additional in-depth docs for developers. >>>> >>>> Comments would be greatly appreciated. >>>> >>>> Again, after checking patch sources I have a strong opinion that it >>>> needs some refactoring - >>>> move all files related to TOAST implementation into new folder >>>> /backend/access/toast where >>>> Generic (default) Toaster resides. >>>> >>>> Patchset consists of: >>>> v16-0001-toaster-interface.patch - Pluggable TOAST API interface along >>>> with reference TOAST mechanics; >>>> v16-0002-toaster-default.patch - Default TOAST re-implemented using >>>> Toaster API; >>>> v16-0003-toaster-docs.patch - Pluggable TOAST API documentation package >>>> >>>> Actual GitHub branch resides at >>>> https://github.com/postgrespro/postgres/tree/toasterapi_clean >>>> >>>> On Fri, Sep 23, 2022 at 10:54 PM Nikita Malakhov >>>> wrote: >>>> >>>>> Hi hackers! >>>>> >>>>> Cfbot is not happy with previous patchset, so I'm attaching new one, >>>>> rebased onto current master >>>>> (15b4). Also providing patch with documentation package, and the >>>>> second one contains large >>>>> README.toastapi file providing additional in-depth docs for developers. >>>>> >>>>> Comments would be greatly appreciated. >>>>> >>>>> Also, after checking patch sources I have a strong opinion that it >>>>> needs some refactoring - >>>>> move all files related to TOAST implementation into new folder >>>>> /backend/access/toast where >>>>> Generic (default) Toaster resides. >>>>> >>>>> Patchset consists of: >>>>> v15-0001-toaster-interface.patch - Pluggable TOAST API interface along >>>>> with reference TOAST mechanics; >>>>> v15-0002-toaster-default.patch - Default TOAST re-implemented using >>>>> Toaster API; >>>>> v15-0003-toaster-docs.patch - Pluggable TOAST API documentation package >>>>> >>>>> On Tue, Sep 13, 2022 at 7:50 PM Jacob Champion < >>>>> jchampion@timescale.com> wrote: >>>>> >>>>>> On Mon, Sep 12, 2022 at 11:45 PM Nikita Malakhov >>>>>> wrote: >>>>>> > It would be more clear for complex data types like JSONB, where >>>>>> developers could >>>>>> > need some additional functionality to work with internal >>>>>> representation of data type, >>>>>> > and its full potential is revealed in our JSONB toaster extension. >>>>>> The JSONB toaster >>>>>> > is still in development but we plan to make it available soon. >>>>>> >>>>>> Okay. It'll be good to have that, because as it is now it's hard to >>>>>> see the whole picture. >>>>>> >>>>>> > On installing dummy_toaster contrib: I've just checked it by making >>>>>> a patch from commit >>>>>> > and applying onto my clone of master and 2 patches provided in >>>>>> previous email without >>>>>> > any errors and sll checks passed - applying with git am, configure >>>>>> with debug, cassert, >>>>>> > depend and enable-tap-tests flags and run checks. >>>>>> > Please advice what would cause such a behavior? >>>>>> >>>>>> I don't think the default pg_upgrade tests will upgrade contrib >>>>>> objects (there are instructions in src/bin/pg_upgrade/TESTING that >>>>>> cover manual dumps, if you prefer that method). My manual steps were >>>>>> roughly >>>>>> >>>>>> =# CREATE EXTENSION dummy_toaster; >>>>>> =# CREATE TABLE test (t TEXT >>>>>> STORAGE external >>>>>> TOASTER dummy_toaster_handler); >>>>>> =# \q >>>>>> $ initdb -D newdb >>>>>> $ pg_ctl -D olddb stop >>>>>> $ pg_upgrade -b /bin -B /bin -d >>>>>> ./olddb -D ./newdb >>>>>> >>>>>> (where /bin is on the PATH, so we're using the right >>>>>> binaries). >>>>>> >>>>>> Thanks, >>>>>> --Jacob >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Nikita Malakhov >>>>> Postgres Professional >>>>> https://postgrespro.ru/ >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Nikita Malakhov >>>> Postgres Professional >>>> https://postgrespro.ru/ >>>> >>> >>> >>> -- >>> Regards, >>> Nikita Malakhov >>> Postgres Professional >>> https://postgrespro.ru/ >>> >> >> >> -- >> Regards, >> Nikita Malakhov >> Postgres Professional >> https://postgrespro.ru/ >> > > > -- > Regards, > Nikita Malakhov > Postgres Professional > https://postgrespro.ru/ > -- Regards, Nikita Malakhov Postgres Professional https://postgrespro.ru/ --000000000000da1fc805ea33296b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi hackers!
Now cfbot=C2=A0is happy, = but there were warnings due to recent=C2=A0changes in PointerGetDatum funct= ion,=C2=A0so here's corrected=C2=A0patchset.

<= div dir=3D"ltr">
Patchset consists o= f:
v20-0001-toaster-interface.patch -=C2=A0Pluggable= =C2=A0TOAST=C2=A0API interface along with reference=C2=A0TOAST=C2=A0mechani= cs - new API is introduced but
reference TOAST is still unchanged= ;
v20-0002-toaster-default.patch -=C2=A0Default=C2=A0TOAST=C2= =A0re-implemented using=C2=A0Toaster=C2=A0API - reference TOAST is re-imple= mented via new API;
v20-0003-toaster-docs.patch -=C2=A0Pluggable=C2=A0TO= AST=C2=A0API documentation package


On Tue, Oct 4, 2022 at 1:02 AM Nikita Malak= hov <hukutoc@gmail.com> wrot= e:
Hi hackers!
Cfbot failed in meson build with previ= ous patchsets, so I've rebased them onto the latest master and added ne= cessary meson build info.

Patchset consists of:
v19-0001-toaster-interface.patch -=C2=A0Pluggable=C2=A0TOAST=C2=A0API interface along with reference=C2=A0TOAST=C2=A0mechanics - new API is introduced but
refer= ence TOAST is still unchanged;
v19-0002-toaster-= default.patch -=C2=A0Default=C2=A0TOAST=C2=A0re-implemented us= ing=C2=A0Toaster=C2=A0API - reference TOAST is re-implemented = via new API;
v19-0003-toaster-docs.patch -=C2=A0Plugg= able=C2=A0TOAST=C2=A0API documentation package

Actual GitHub branch resides at

On Tue, Sep 27, 2022 at 12:26 AM Nikita Malakhov <hukutoc@gmail.com> wrote:
Hi,
Meson build for the patchset failed, meson build files = attached and README/Doc package
reworked with more detailed expla= nation of virtual function table along with other corrections.
<= div>
On Sun, Sep 25, 2022 at 1:41 AM Nikita Malakhov <hukutoc@gmail.com> wrote:
Hi hackers!
Last patchset has an invalid patch file - v16-0= 003-toaster-docs.patch. Here's corrected patchset,
sorry for = the noise.

On Sat, Sep 24, 2022 at 3:50 PM Nikita Malakhov <hukutoc@gmail.com> w= rote:
Hi hackers!

<= /div>
Cfbot is still not happy with the patchset, so I'm attaching = a rebased one, rebased onto the current
master (from today). The = third patch contains documentation package, and the second one contains lar= ge=C2=A0
README.toastapi file providing additional in-depth docs = for developers.

Comments would be greatly apprecia= ted.

Again, after checking patch sources I have a = strong opinion that it needs some refactoring -=C2=A0
move all fi= les related to TOAST implementation into new folder /backend/access/toast w= here
Generic (default) Toaster resides.

= Patchset consists of:
v16-0001-toaster-interface.patch -=C2= =A0Pluggable TOAST API interface along with reference TOAST mechanics;
<= /div>
v16-0002-toaster-default.patch -=C2=A0Default TOAST re-implemente= d using Toaster API;
v16-0003-toaster-docs.patch -=C2=A0Pluggable TOAST = API documentation package

Actual GitHub branch res= ides at

On Fri, Sep 23, 2022 at 10:54 PM Nikita = Malakhov <hukutoc= @gmail.com> wrote:
Hi hackers!

Cfbot is not happy with previous patchset, so I'm attaching new one, r= ebased onto current master
(15b4). Also providing patch with docu= mentation package, and the second one contains large=C2=A0
README= .toastapi file providing additional in-depth docs for developers.

Comments would be greatly appreciated.

=
Also, after checking patch sources I have a strong opinion that it nee= ds some refactoring -=C2=A0
move all files related to TOAST imple= mentation into new folder /backend/access/toast where
Generic (de= fault) Toaster resides.

Patchset consists of:
v15-0001-toaster-interface.patch -=C2=A0Pluggable TOAST API int= erface along with reference TOAST mechanics;
v15-0002-toaster= -default.patch -=C2=A0Default TOAST re-implemented using Toaster API;
v1= 5-0003-toaster-docs.patch -=C2=A0Pluggable TOAST API documentation package<= br>

On Tue, Sep 13, 2022 at 7:50 PM Jacob Champion <jchampion@timescale.com> wro= te:
On Mon, Sep = 12, 2022 at 11:45 PM Nikita Malakhov <hukutoc@gmail.com> wrote:
> It would be more clear for complex data types like JSONB, where develo= pers could
> need some additional functionality to work with internal representatio= n of data type,
> and its full potential is revealed in our JSONB toaster extension. The= JSONB toaster
> is still in development but we plan to make it available soon.

Okay. It'll be good to have that, because as it is now it's hard to=
see the whole picture.

> On installing dummy_toaster contrib: I've just checked it by makin= g a patch from commit
> and applying onto my clone of master and 2 patches provided in previou= s email without
> any errors and sll checks passed - applying with git am, configure wit= h debug, cassert,
> depend and enable-tap-tests flags and run checks.
> Please advice what would cause such a behavior?

I don't think the default pg_upgrade tests will upgrade contrib
objects (there are instructions in src/bin/pg_upgrade/TESTING that
cover manual dumps, if you prefer that method). My manual steps were
roughly

=C2=A0 =C2=A0 =3D# CREATE EXTENSION dummy_toaster;
=C2=A0 =C2=A0 =3D# CREATE TABLE test (t TEXT
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 STORAGE external
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 TOASTER dummy_toaster_handler); =C2=A0 =C2=A0 =3D# \q
=C2=A0 =C2=A0 $ initdb -D newdb
=C2=A0 =C2=A0 $ pg_ctl -D olddb stop
=C2=A0 =C2=A0 $ pg_upgrade -b <install path>/bin -B <install path&= gt;/bin -d
./olddb -D ./newdb

(where <install path>/bin is on the PATH, so we're using the righ= t binaries).

Thanks,
--Jacob


--
Regards,
Nikita Malakhov
Postgres Professional=C2=A0


--
Regards,
Nikita Malakhov
Postgres Professional=C2=A0


--
Regards,
Nikita Malakhov
Postgres Professional=C2=A0


--
Regards,
Nikita Malakhov
Postgres Professional=C2=A0


--
Regards,
Nikita Malakhov
Postgres Professional=C2=A0


--
Regards,
Nikita Malakhov
Postgres Professional=C2=A0
https://postgrespro.ru/
--000000000000da1fc805ea33296b--