Feedback-ID: i7c334307:Fastmail
From: "David E. Wheeler" <david@justatheory.com>
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3818.100.11.1.3\))
Subject: RFC: Extension Packaging & Lookup
Message-Id: <2CAD6FA7-DC25-48FC-80F2-8F203DECAE6A@justatheory.com>
Date: Thu, 10 Oct 2024 22:34:08 +0200
To: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Archived-At: <https://www.postgresql.org/message-id/2CAD6FA7-DC25-48FC-80F2-8F203DECAE6A%40justatheory.com>
Precedence: bulk

Hackers,

Back at the end of August, I promised[1]:

> I=E2=80=99ll try to put some thought into a more formal proposal in a =
new thread next week. Unless your Gabriele beats me to it =F0=9F=98=82.

I guess I should get off my butt and do it. So let=E2=80=99s do this. =
Here=E2=80=99s what I propose.

*   When an extension is installed, all of its files should live in a =
single directory. These include:

    *   The Control file in directory describes extension
    *   Subdirectories for SQL, shared libraries, docs, binaries
        (also locales and tsearch dictionaries?)

*   Next, there should be an extension lookup path. The first item in =
the path is the compile-time default, and ideally would include only =
core extensions. Subsequent paths would be set by a GUC, similar to =
dynamic_library_path, but only for extensions (including their shared =
libraries).

*   Modify PGXS (or create a new installer CLI used by PGXS?) to install =
an extension according to this pattern. Allow the specification of a =
prefix. This should differ from the current `PREFIX`, in that the values =
of `sharedir`, `pkglibdir`, etc. would not be fully-duplicated under the =
prefix, but point to a directory used in the extension path. For =
example, when installing an extension need =E2=80=9Cpair", something =
like

        make install BASE_DIR=3D/opt/pg/extension

    Would create `/opt/pg/extension/pair`, rather than =
`/opt/pg/extension/$(pg_config --sharedir)/extension/pair`.

*   Perhaps there could also be an option to symlink binary files or man =
pages to keep paths simple.

*   For CREATE EXTENSION, Postgres would look for an extension on the =
file system by directory name in each of the extension paths instead of =
control file name. It would then find the control file in that directory =
and the necessary SQL and shared library files in the `sql` and `lib` =
subdirectories of that directory.

*   Binary-only extensions might also be installed here; the difference =
is they have no control file. The LOAD command and =
shared_preload_libraries would need to know to look here, too.

The basic idea, then, is three-fold:

1.  This pattern is more like a packaging pattern than CREATE =
EXTENSION-specific, since it includes other types of extensions

2.  All the files for a given extension live within a single directory, =
making it easier to reason about what=E2=80=99s installed and what=E2=80=99=
s not.

3.  These extension packages can live in multiple paths.

Some examples. Core extensions, like citext, would live in, say, =
$(pg_config --extensiondir)/citext), and have a structure such as:

```
citext
=E2=94=9C=E2=94=80=E2=94=80 citext.control
=E2=94=9C=E2=94=80=E2=94=80 lib
=E2=94=82   =E2=94=9C=E2=94=80=E2=94=80 citext.dylib
=E2=94=82   =E2=94=94=E2=94=80=E2=94=80 bitcode
=E2=94=82   =E2=94=9C=E2=94=80=E2=94=80 citext
=E2=94=82   =E2=94=82   =E2=94=94=E2=94=80=E2=94=80 citext.bc
=E2=94=82   =E2=94=94=E2=94=80=E2=94=80 citext.index.bc
=E2=94=94=E2=94=80=E2=94=80 sql
    =E2=94=9C=E2=94=80=E2=94=80 citext--1.0--1.1.sql
    =E2=94=9C=E2=94=80=E2=94=80 citext--1.1--1.2.sql
    =E2=94=9C=E2=94=80=E2=94=80 citext--1.2--1.3.sql
    =E2=94=9C=E2=94=80=E2=94=80 citext--1.3--1.4.sql
    =E2=94=9C=E2=94=80=E2=94=80 citext--1.4--1.5.sql
    =E2=94=9C=E2=94=80=E2=94=80 citext--1.4.sql
    =E2=94=94=E2=94=80=E2=94=80 citext--1.5--1.6.sql
```

Third-party extensions would live in one or more other directories on =
the file system, unknown at compile time, but set in the extension path =
GUC and accessible to/owned by the Postgres system user. Let=E2=80=99s =
say we set `/opt/pgxn` as one of the paths. Within that directory, we =
might have a directory for a pure SQL extension in a a directory named =
=E2=80=9Cpair=E2=80=9D that looks like this:

```
pair
=E2=94=9C=E2=94=80=E2=94=80 LICENSE.md
=E2=94=9C=E2=94=80=E2=94=80 README.md
=E2=94=9C=E2=94=80=E2=94=80 pair.control
=E2=94=9C=E2=94=80=E2=94=80 doc
=E2=94=82   =E2=94=9C=E2=94=80=E2=94=80 html
=E2=94=82   =E2=94=82   =E2=94=94=E2=94=80=E2=94=80 pair.html
=E2=94=82   =E2=94=94=E2=94=80=E2=94=80 pair.md
=E2=94=94=E2=94=80=E2=94=80 sql
=E2=94=9C=E2=94=80=E2=94=80 pair--1.0--1.1.sql
=E2=94=94=E2=94=80=E2=94=80 pair--1.1.sql
```

A binary application like pg_top would live in the pg_top directory, =
structured something like:

```
pg_top
=E2=94=9C=E2=94=80=E2=94=80 HISTORY.rst
=E2=94=9C=E2=94=80=E2=94=80 INSTALL.rst
=E2=94=9C=E2=94=80=E2=94=80 LICENSE
=E2=94=9C=E2=94=80=E2=94=80 README.rst
=E2=94=9C=E2=94=80=E2=94=80 bin
|   =E2=94=94=E2=94=80=E2=94=80 pg_top
=E2=94=94=E2=94=80=E2=94=80 doc
    =E2=94=94=E2=94=80=E2=94=80 man
        =E2=94=94=E2=94=80=E2=94=80 man3
            =E2=94=94=E2=94=80=E2=94=80 pg_top.3
```

And a C extension like semver would live in the semver directory and be =
structured something like:

```
semver
=E2=94=9C=E2=94=80=E2=94=80 LICENSE
=E2=94=9C=E2=94=80=E2=94=80 README.md
=E2=94=9C=E2=94=80=E2=94=80 semver.control
=E2=94=9C=E2=94=80=E2=94=80 doc
=E2=94=82   =E2=94=94=E2=94=80=E2=94=80 semver.md
=E2=94=9C=E2=94=80=E2=94=80 lib
=E2=94=82   =E2=94=9C=E2=94=80=E2=94=80 semver.dylib
=E2=94=82   =E2=94=9C=E2=94=80=E2=94=80 bitcode
=E2=94=82   =E2=94=94=E2=94=80=E2=94=80 semver
=E2=94=82   =E2=94=82   =E2=94=94=E2=94=80=E2=94=80 semver.bc
=E2=94=82   =E2=94=94=E2=94=80=E2=94=80 semver.index.bc
=E2=94=94=E2=94=80=E2=94=80 sql
    =E2=94=9C=E2=94=80=E2=94=80 semver--1.0--1.1.sql
    =E2=94=94=E2=94=80=E2=94=80 semver--1.1.sql
```

Thoughts?

Best,

David

[1]: =
https://www.postgresql.org/message-id/D30A91FA-A6D4-4737-941F-0BBB2984B730=
%40justatheory.com