Feedback-ID: i7c334307:Fastmail
Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\))
Subject: Re: RFC: Additional Directory for Extensions
From: "David E. Wheeler" <david@justatheory.com>
In-Reply-To: <0d384836-7e6e-4932-af3b-8dad1f6fee43@eisentraut.org>
Date: Mon, 11 Nov 2024 13:15:53 -0500
Cc: Craig Ringer <craig.ringer@enterprisedb.com>,
 Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>,
 Robert Haas <robertmhaas@gmail.com>,
 Alvaro Herrera <alvherre@alvh.no-ip.org>,
 PostgreSQL-development <pgsql-hackers@postgresql.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <0D89C6B2-F294-4389-9DDA-07C381995292@justatheory.com>
References: <CA+TgmoaMk==vOUmWwJ5ZA-EueNqa9RQaWMEiHCF1ZKW_YCcj8g@mail.gmail.com>
 <202406251012.brdp2dobkf7n@alvherre.pgsql>
 <CA+TgmoYcAmhnSUHKdJsS1wOR-9+y6dbLhfJ3Ad+iZEDmA1mPXg@mail.gmail.com>
 <FE1622BC-81BB-487B-BF0A-4D29906F8F47@justatheory.com>
 <C0393253-B71B-4AF9-87DA-A831AC8B3836@justatheory.com>
 <CA+VUV5oWkMVvtNB0BG2CM8SrmOkeW6fEtXGehbniHqR9MhMzVw@mail.gmail.com>
 <CAGRY4nxP6A5Dz23g+aGD-agdVwUj_qrG6szd-mWc0E5OFMBg4w@mail.gmail.com>
 <B86AF59F-5B65-4CC9-91A4-B79A089C2876@justatheory.com>
 <CAGRY4ny=Oy-F0qCRFQa=XtEqyUbMygvbqqKahAvbae7v4LTOrQ@mail.gmail.com>
 <AF4A50CD-C5CF-406D-9700-1B889F40E8FB@justatheory.com>
 <CAGRY4nx8Fmea4udk--PKdvfgpNdqC+rcueAmGwuw2PD9dv+zqQ@mail.gmail.com>
 <D30A91FA-A6D4-4737-941F-0BBB2984B730@justatheory.com>
 <0d384836-7e6e-4932-af3b-8dad1f6fee43@eisentraut.org>
To: Peter Eisentraut <peter@eisentraut.org>
Archived-At: <https://www.postgresql.org/message-id/0D89C6B2-F294-4389-9DDA-07C381995292%40justatheory.com>
Precedence: bulk

On Nov 11, 2024, at 02:16, Peter Eisentraut <peter@eisentraut.org> =
wrote:

> I implemented a patch along the lines Craig had suggested.

Oh, nice, thank you.

> It's a new GUC variable that is a path for extension control files.  =
It's called extension_control_path, and it works exactly the same way as =
dynamic_library_path.  Except that the magic token is called $system =
instead of $libdir.

I assume we can bikeshed these names later. :-)

> In fact, most of the patch is refactoring the routines in dfmgr.c to =
not hardcode dynamic_library_path but allow searching for any file in =
any path.  Once a control file is found, the other extension support =
files (script files and auxiliary control files) are looked for in the =
same directory.

What about shared libraries files?

> This works pretty much fine for the use cases that have been presented =
here, including installing extensions outside of the core installation =
tree (for CNPG and Postgres.app) and for testing uninstalled extensions =
(for Debian).

If I understand correctly, shared modules still lie in =
dynamic_library_path, yes? That makes things a bit more complicated, as =
the CNPG use case has to set up multiple persistent volumes to persist =
files put into various directories, notably sharedir and pkglibdir.

> - You can install extensions into alternative directories using PGXS =
like
>=20
>    make install datadir=3D/else/where/share pkglibdir=3D/else/where/lib
>=20
> This works.  I was hoping it would work to use
>=20
>    make install prefix=3D/else/where
>=20
> but that doesn't because of some details in Makefile.global.  I think =
we can tweak that a little bit to make that work too.

In the broader extension organization RFC I=E2=80=99ve been working =
up[1], I propose a new Makefile prefix for the destination directory for =
an extension. It hinges on the idea that an extension has all of its =
files organized in a directory with the extension name, rather than =
separate params for data, pkglib, bin, etc.

> - With the current patch, if you install into =
datadir=3D/else/where/share, then you need to set =
extension_control_path=3D/else/where/share/extension.  This is a bit =
confusing.  Maybe we want to make the "extension" part implicit.

+1

> - The biggest problem is that many extensions set in their control =
file
>=20
>    module_pathname =3D '$libdir/foo'
>=20
> This disables the use of dynamic_library_path, so this whole idea of =
installing an extension elsewhere won't work that way.  The obvious =
solution is that extensions change this to just 'foo'.  But this will =
require a lot updating work for many extensions, or a lot of patching by =
packagers.

Since that=E2=80=99s set at build/install time, couldn=E2=80=99t the =
definition of `$libdir` here be changed to mean =E2=80=9Cthe directory =
into which it=E2=80=99s being installed right now?=E2=80=9D. Doesn=E2=80=99=
t seem necessary to search a path if the specific location is set at =
install time.

> Maybe we could devise some sort of rule that if the extension control =
file is found via the path outside of $system, then the leading $libdir =
is ignored.
> <v0-0001-extension_control_path.patch>

This is what I propose,[1] yes. If we redefine the organization of =
extension files to live in a single directory named for an extension, =
then once you=E2=80=99ve found the control files all the other files are =
there, too. But `$libdir` is presumably still meaningful, since the =
first time you call an extension function in a new connection, it just =
tries to load that location, it doesn=E2=80=99t go through the control =
file search process, AFAIK.

Perhaps I misunderstand, but I would like to talk through the =
implications of a more radical rethinking of extension file location =
along the lines of the other thread[2] and the RFC I=E2=80=99m working =
up based on them both[1], especially since there are a few other use =
cases that inform it.

A notable one is the idea Gabriele shared with me in Athens to be able =
to add an extension to a running CNPG pod by simply mounting a read-only =
volume with all the files it requires.

Best,

David

[1]: https://github.com/theory/justatheory/pull/7/files