Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\))
Subject: Re: RFC: Additional Directory for Extensions
From: "David E. Wheeler" <david@justatheory.com>
In-Reply-To: <CAGRY4nxP6A5Dz23g+aGD-agdVwUj_qrG6szd-mWc0E5OFMBg4w@mail.gmail.com>
Date: Mon, 26 Aug 2024 10:06:59 -0400
Cc: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>,
 Robert Haas <robertmhaas@gmail.com>,
 Alvaro Herrera <alvherre@alvh.no-ip.org>,
 PostgreSQL-development <pgsql-hackers@postgresql.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <B86AF59F-5B65-4CC9-91A4-B79A089C2876@justatheory.com>
References: <CA+TgmoaMk==vOUmWwJ5ZA-EueNqa9RQaWMEiHCF1ZKW_YCcj8g@mail.gmail.com>
 <202406251012.brdp2dobkf7n@alvherre.pgsql>
 <CA+TgmoYcAmhnSUHKdJsS1wOR-9+y6dbLhfJ3Ad+iZEDmA1mPXg@mail.gmail.com>
 <FE1622BC-81BB-487B-BF0A-4D29906F8F47@justatheory.com>
 <C0393253-B71B-4AF9-87DA-A831AC8B3836@justatheory.com>
 <CA+VUV5oWkMVvtNB0BG2CM8SrmOkeW6fEtXGehbniHqR9MhMzVw@mail.gmail.com>
 <CAGRY4nxP6A5Dz23g+aGD-agdVwUj_qrG6szd-mWc0E5OFMBg4w@mail.gmail.com>
To: Craig Ringer <craig.ringer@enterprisedb.com>
Archived-At: <https://www.postgresql.org/message-id/B86AF59F-5B65-4CC9-91A4-B79A089C2876%40justatheory.com>
Precedence: bulk

Hi Hackers,

Apologies for the delay in reply; I=E2=80=99ve been at the XOXO Festival =
and almost completely unplugged for the first time in ages. Happy to see =
this thread coming alive, though. Thank you Gabriele, Craig, and Jelte!

On Aug 21, 2024, at 19:07, Craig Ringer <craig.ringer@enterprisedb.com> =
wrote:

> So IMO this should be a _path_ to search for extension control files
> and SQL scripts.
>=20
> If the current built-in default extension dir was exposed as a var
> $extdir like we do for $libdir, this might look something like this
> for local development and testing while working with a packaged
> postgres build:
>=20
>    SET extension_search_path =3D $extsdir, /opt/myapp/extensions,
> /usr/local/postgres/my-custom-extension/extensions;
>    SET dynamic_library_path =3D $libdir, /opt/myapp/lib,
> /usr/local/postgres/my-custom-extension/lib

I would very much like something like this, but I=E2=80=99m not sure how =
feasible it is for a few reasons. The first, and most important, is that =
extensions are not limited to just a control file and SQL file. They =
also very often include:

* one or more shared library files
* documentation files
* binary files

And maybe more? How many of these directories might an extension install =
files into:

=E2=9C=A6 =E2=9D=AF pg_config | grep DIR | awk '{print $1}'
BINDIR
DOCDIR
HTMLDIR
INCLUDEDIR
PKGINCLUDEDIR
INCLUDEDIR-SERVER
LIBDIR
PKGLIBDIR
LOCALEDIR
MANDIR
SHAREDIR
SYSCONFDIR

I would assume BINDIR, DOCDIR, HTMLDIR, PKGLIBDIR, MANDIR, SHAREDIR, and =
perhaps LOCALEDIR.

But even if it=E2=80=99s just one or two, the only proper way an =
extension directory would work, IME, is to define a directory-based =
structure for extensions, where every file for an extension is in a =
directory named for the extension, and subdirectories are defined for =
each of the above requisite file types. Something like:

extension_name
=E2=94=9C=E2=94=80=E2=94=80 control.ini
=E2=94=9C=E2=94=80=E2=94=80 bin
=E2=94=9C=E2=94=80=E2=94=80 doc
=E2=94=9C=E2=94=80=E2=94=80 html
=E2=94=9C=E2=94=80=E2=94=80 lib
=E2=94=9C=E2=94=80=E2=94=80 local
=E2=94=9C=E2=94=80=E2=94=80 man
=E2=94=94=E2=94=80=E2=94=80 share

This would allow multiple paths to work and keep all the files for an =
extension bundled together. It could also potentially allow for multiple =
versions of an extension to be installed at once, if we required the =
version to be part of the directory name.

I think this would be a much nicer layout for packaging, installing, and =
managing extensions versus the current method of strewing files around =
to a slew of different directories. But it would come at some cost, in =
terms of backward with the existing layout (or migration to it), =
significant modification of the server to use the new layout (and =
extension_search_path), and other annoyances like PATH and MANPATH =
management.

Long term I think it would be worthwhile, but the current patch feels =
like a decent interim step we could live with, solving most of the =
integration problems (immutable servers, packaging testing, etc.) at the =
cost of a slightly unexpected directory layout. What I mean by that is =
that the current patch is pretty much just using extension_destdir as a =
prefix to all of those directories from pg_config, so they never have to =
change, but it does mean that you end up installing extensions in =
something like

/mnt/extensions/pg16/usr/share/postgresql/16
/mnt/extensions/pg16/usr/include/postgresql

etc.

Best,

David