Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rAC9r-005JWa-9V for pgsql-hackers@arkaria.postgresql.org; Mon, 04 Dec 2023 16:50:59 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rAC9p-000UrH-Rc for pgsql-hackers@arkaria.postgresql.org; Mon, 04 Dec 2023 16:50:57 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rAC9p-000Ur9-Dh for pgsql-hackers@lists.postgresql.org; Mon, 04 Dec 2023 16:50:57 +0000 Received: from mail-lj1-x22b.google.com ([2a00:1450:4864:20::22b]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1rAC9m-008mUs-G6 for pgsql-hackers@postgresql.org; Mon, 04 Dec 2023 16:50:56 +0000 Received: by mail-lj1-x22b.google.com with SMTP id 38308e7fff4ca-2c9eca5bbaeso28028561fa.3 for ; Mon, 04 Dec 2023 08:50:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701708653; x=1702313453; darn=postgresql.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=aaKVTXw0lPAKmXcy9pcpawFqDE90TwCYnXZP7rzOlqc=; b=EwJYNUEKFBF/6SehQz8xGf9OArmuB8dFIEgYQe99/spBfZuwd2+Vu8VDYzIVQYQaOE oZjbLpIGifVr8M4xZE2be/ktPyigDx2Rm57gEaBrDfYwwdEKHUZahhZEfoQI+Icv+RM6 p0dTRJZ/j0wDWJz45wmasrYQrjs8lcE8K9v1TDw4VxH8fvvX/MOCj0Kt+xD9P1gQab2r a/la7yYUdqj3BFDYSjA+ueBbqPqg4mVUWufoFsftkR5+vGf+X8rfNWxTf45+YJH5z76t kTUpxL1YcJk9OB4bBm4rSg7JwttVPG7JYj9qiMr/gJgcyauHFQsQCdnQ6bCQBVRmsiYs HJEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701708653; x=1702313453; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aaKVTXw0lPAKmXcy9pcpawFqDE90TwCYnXZP7rzOlqc=; b=jo+ZQ+/InMvybmF1ZqnZsYDKHGa+3a+Zy8Awc2hyYRlpJJdlZ4iv17Y4+NEDRvLpK2 pLn3CfSdbuVm5aWOAGu8hFXBh7z5FQYR4O16ox6kO4ZXua+0EfIUnPxJum70ZKrbGOuw etSq8+yvKi1k3W5/aGaYpPgK3qK7jSbq3PoUi0u8bllhcu+JDyr1xf4mXdxUHDlUD7L7 gS4F3oKgmHZOKN5+9zKUN0HCYRukXbanMQN7nHf715rpQgO39ygrTdGldTaGSbbGC8Eq NsIfRl0fzFtXw4iwGIAv+E8yfu0EK6isHIdfhmmDw2kkAA5m26HpkomDzBVS9PtYuII3 J9MQ== X-Gm-Message-State: AOJu0YwwO4b0mT6EGomk9Viujt5SvuHvKdgV/EUz7rvz3WgqyXEFULp/ JeSbAnUz43GhP6awnK2/RNNDrt2gFDQbXfImpU1kFcEgiC4= X-Google-Smtp-Source: AGHT+IEj2/uL3jJkf8+sM4o3wbZoB7UipaspPyUfjJ742uT/5oQUAXkVQwdPoXbHH0i/hq6eoSOo1GRcm06gFQ8v6+A= X-Received: by 2002:a05:651c:101:b0:2ca:cb5:2ec1 with SMTP id a1-20020a05651c010100b002ca0cb52ec1mr213277ljb.50.1701708652474; Mon, 04 Dec 2023 08:50:52 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Kirill Reshke Date: Mon, 4 Dec 2023 19:50:41 +0300 Message-ID: Subject: Fwd: Extensible storage manager API - SMGR hook Redux To: pgsql-hackers@postgresql.org Content-Type: multipart/alternative; boundary="000000000000bdf6a7060bb1eba1" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000bdf6a7060bb1eba1 Content-Type: text/plain; charset="UTF-8" Sorry for double-posting, I accidentally replied to Matthias, not the mailing list :( ---------- Forwarded message --------- From: Kirill Reshke Date: Mon, 4 Dec 2023 at 19:46 Subject: Re: Extensible storage manager API - SMGR hook Redux To: Matthias van de Meent Hi! On Fri, 30 Jun 2023 at 15:27, Matthias van de Meent < boekewurm+postgres@gmail.com> wrote: > Hi hackers, > > At Neon, we've been working on removing the file system dependency > from PostgreSQL and replacing it with a distributed storage layer. For > now, we've seen most success in this by replacing the implementation > of the smgr API, but it did require some core modifications like those > proposed early last year by Anastasia [0]. > > As mentioned in the previous thread, there are several reasons why you > would want to use a non-default storage manager: storage-level > compression, encryption, and disk limit quotas [0]; offloading of cold > relation data was also mentioned [1]. > > In the thread on Anastasia's patch, Yura Sokolov mentioned that > instead of a hook-based smgr extension, a registration-based smgr > would be preferred, with integration into namespaces. Please find > attached an as of yet incomplete patch that starts to do that. > > The patch is yet incomplete (as it isn't derived from Anastasia's > patch), but I would like comments on this regardless, as this is a > fairly fundamental component of PostgreSQL that is being modified, and > it is often better to get comments early in the development cycle. One > significant issue that I've seen so far are that catcache is not > guaranteed to be available in all backends that need to do smgr > operations, and I've not yet found a good solution. > > Changes compared to HEAD: > - smgrsw is now dynamically allocated and grows as new storage > managers are loaded (during shared_preload_libraries) > - CREATE TABLESPACE has new optional syntax USING smgrname (option [, ...]) > - tablespace storage is (planned) fully managed by smgr through some > new smgr apis > > Changes compared to Anastasia's patch: > - extensions do not get to hook and replace the api of the smgr code > directly - they are hidden behind the smgr registry. > > Successes: > - 0001 passes tests (make check-world) > - 0002 builds without warnings (make) > > TODO: > - fix dependency failures when catcache is unavailable > - tablespace redo is currently broken with 0002 > - fix tests for 0002 > - ensure that pg_dump etc. works with the new tablespace storage manager > options > > Looking forward to any comments, suggestions and reviews. > > Kind regards, > > Matthias van de Meent > Neon (https://neon.tech/) > > > [0] > https://www.postgresql.org/message-id/CAP4vRV6JKXyFfEOf%3Dn%2Bv5RGsZywAQ3CTM8ESWvgq%2BS87Tmgx_g%40mail.gmail.com > [1] > https://www.postgresql.org/message-id/D365F19F-BC3E-4F96-A91E-8DB13049749E@yandex-team.ru So, 0002 patch uses the `get_tablespace` function, which searches Catalog to tablespace SMGR id. I wonder how `smgr_redo` would work with it? Is it possible to query the system catalog during crash recovery? As far as i understand the answer is "no", correct me if I'm wrong. Furthermore, why do we only allow tablespace to have its own SMGR implementation, can we have per-relation SMGR? Maybe we can do it in a way similar to custom RMGR (meaning, write SMGR OID into WAL and use it in crash recovery etc.)? --000000000000bdf6a7060bb1eba1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Sorry for double-posting, I accidentally replied to Matthi= as, not the mailing list :(

---------- Forwarded message ---------
Fro= m: Kirill Reshke <reshkekirill@gmail.= com>
Date: Mon, 4 Dec 2023 at 19:46
Subject: Re: Extens= ible storage manager API - SMGR hook Redux
To: Matthias van de Meent <= ;boekewurm+postgres@gmail= .com>


Hi!

On Fri, 30 Jun 2023 at 15:27, Matthias van de Meent <boekewurm+postgres@gmai= l.com> wrote:
Hi hackers,

At Neon, we've been working on removing the file system dependency
from PostgreSQL and replacing it with a distributed storage layer. For
now, we've seen most success in this by replacing the implementation of the smgr API, but it did require some core modifications like those
proposed early last year=C2=A0 by Anastasia [0].

As mentioned in the previous thread, there are several reasons why you
would want to use a non-default storage manager: storage-level
compression, encryption, and disk limit quotas [0]; offloading of cold
relation data was also mentioned [1].

In the thread on Anastasia's patch, Yura Sokolov mentioned that
instead of a hook-based smgr extension, a registration-based smgr
would be preferred, with integration into namespaces. Please find
attached an as of yet incomplete patch that starts to do that.

The patch is yet incomplete (as it isn't derived from Anastasia's patch), but I would like comments on this regardless, as this is a
fairly fundamental component of PostgreSQL that is being modified, and
it is often better to get comments early in the development cycle. One
significant issue that I've seen so far are that catcache is not
guaranteed to be available in all backends that need to do smgr
operations, and I've not yet found a good solution.

Changes compared to HEAD:
- smgrsw is now dynamically allocated and grows as new storage
managers are loaded (during shared_preload_libraries)
- CREATE TABLESPACE has new optional syntax USING smgrname (option [, ...])=
- tablespace storage is (planned) fully managed by smgr through some
new smgr apis

Changes compared to Anastasia's patch:
- extensions do not get to hook and replace the api of the smgr code
directly - they are hidden behind the smgr registry.

Successes:
- 0001 passes tests (make check-world)
- 0002 builds without warnings (make)

TODO:
- fix dependency failures when catcache is unavailable
- tablespace redo is currently broken with 0002
- fix tests for 0002
- ensure that pg_dump etc. works with the new tablespace storage manager op= tions

Looking forward to any comments, suggestions and reviews.

Kind regards,

Matthias van de Meent
Neon (h= ttps://neon.tech/)


[0] https://www.postgresql.org/message-id/CAP4vRV6JKXyFfEOf%3Dn= %2Bv5RGsZywAQ3CTM8ESWvgq%2BS87Tmgx_g%40mail.gmail.com
[1] https:/= /www.postgresql.org/message-id/D365F19F-BC3E-4F96-A91E-8DB13049749E@yandex-= team.ru

So, 0002 patch uses the `get_ta= blespace` function, which searches Catalog to tablespace SMGR id. I wonder = how `smgr_redo` would work with it?
Is it possible to query the s= ystem catalog during crash recovery? As far as i understand the answer is &= quot;no", correct me if I'm wrong.
Furthermore, why= do we only allow tablespace to have its own SMGR implementation, can we ha= ve per-relation SMGR? Maybe we can do it in a way similar to custom RMGR (m= eaning, write SMGR OID into WAL and use it in crash recovery etc.)?
--000000000000bdf6a7060bb1eba1--