Subject: Re: RFC: Additional Directory for Extensions

From: Peter Eisentraut <[email-masked]>
Date: Tue, 12 Nov 2024 14:25:30 +0100

On 11.11.24 19:15, David E. Wheeler wrote:
>> In fact, most of the patch is refactoring the routines in dfmgr.c to not hardcode dynamic_library_path but allow searching for any file in any path.  Once a control file is found, the other extension support files (script files and auxiliary control files) are looked for in the same directory.
> 
> What about shared libraries files?

Nothing changes about shared library files.  They are looked up in 
dynamic_library_path or any hardcoded file name.

>> This works pretty much fine for the use cases that have been presented here, including installing extensions outside of the core installation tree (for CNPG and Postgres.app) and for testing uninstalled extensions (for Debian).
> 
> If I understand correctly, shared modules still lie in dynamic_library_path, yes? That makes things a bit more complicated, as the CNPG use case has to set up multiple persistent volumes to persist files put into various directories, notably sharedir and pkglibdir.

No, you can also install them into a common directory and mount that 
one.  For example, you install the extension at build time into 
/tmp/foo/{lib,share/extension}, you package that up as a disk image, 
mount it at /opt/extensions/myext, and then you can point 
extension_control_path at /opt/extensions/myext/lib and 
dynamic_library_path at /opt/extensions/myext/share/extension.

>> - The biggest problem is that many extensions set in their control file
>>
>>     module_pathname = '$libdir/foo'
>>
>> This disables the use of dynamic_library_path, so this whole idea of installing an extension elsewhere won't work that way.  The obvious solution is that extensions change this to just 'foo'.  But this will require a lot updating work for many extensions, or a lot of patching by packagers.
> 
> Since that’s set at build/install time, couldn’t the definition of `$libdir` here be changed to mean “the directory into which it’s being installed right now?”. Doesn’t seem necessary to search a path if the specific location is set at install time.

No, this is not set at build or install time.  This is for typical 
extensions hardcoded, and $libdir is resolved by the PostgreSQL server 
at run time.

> Perhaps I misunderstand, but I would like to talk through the implications of a more radical rethinking of extension file location along the lines of the other thread[2] and the RFC I’m working up based on them both[1], especially since there are a few other use cases that inform it.

I'm aware of that thread, but I think that is looking like a much larger 
project than what I'm proposing here.

---

From: "David E. Wheeler" <[email-masked]>
Date: Tue, 12 Nov 2024 09:44:52 -0500

On Nov 12, 2024, at 08:25, Peter Eisentraut <[email-masked]> wrote:

> No, you can also install them into a common directory and mount that one.  For example, you install the extension at build time into /tmp/foo/{lib,share/extension}, you package that up as a disk image, mount it at /opt/extensions/myext, and then you can point extension_control_path at /opt/extensions/myext/lib and dynamic_library_path at /opt/extensions/myext/share/extension.

Ah, I see, then you just have to set both GUCs to subdirectories of the one volume.

> No, this is not set at build or install time.  This is for typical extensions hardcoded, and $libdir is resolved by the PostgreSQL server at run time.

I see, so that they could be moved and, as long as dynamic_library_path is updated, would still be findable.

>>> This disables the use of dynamic_library_path, so this whole idea of installing an extension elsewhere won't work that way.  The obvious solution is that extensions change this to just 'foo'.  Bu…

Yeah, '$libdir/foo' has been the documented way to do it for quite some time, as I recall. Perhaps the behavior of the MODULE_PATHNAME replacement function could be changed to omit $libdir when writing the SQL files?

> I'm aware of that thread, but I think that is looking like a much larger project than what I'm proposing here.

Fair enough. Once we get to some consensus on a design there (and I’ve continued to iterate on it elsewhere[1]), I doubt it’d take much to use this patch as the first step toward it.

Best,

David

[1]: https://github.com/theory/justatheory/pull/7/files

---

From: "David E. Wheeler" <[email-masked]>
Date: Mon, 18 Nov 2024 14:19:07 -0500

Hi Peter,

Making another pass at this proposal, I’m a bit confused by this issue:

Elsewhere you write:

> Nothing changes about shared library files.  They are looked up in dynamic_library_path or any hardcoded file name.

And also point out that the way to install them is:

```
make install datadir=/else/where/share pkglibdir=/else/where/lib
```

So as long as dynamic_library_path includes /else/where/lib it should work, just as before, no?

Best,

David

---

From: Peter Eisentraut <[email-masked]>
Date: Wed, 20 Nov 2024 10:05:32 +0100

> So as long as dynamic_library_path includes /else/where/lib it should work, just as before, no?

The path is only consulted if the specified name does not contain a 
slash.  So if you do LOAD 'foo', the path is consulted, but if you do 
LOAD '$libdir/foo', it is not.  The problem I'm describing is that most 
extensions use the latter style, per current recommendation in the 
documentation.

---

From: "David E. Wheeler" <[email-masked]>
Date: Wed, 20 Nov 2024 13:04:17 -0500

On Nov 20, 2024, at 04:05, Peter Eisentraut <[email-masked]> wrote:

> The path is only consulted if the specified name does not contain a slash.  So if you do LOAD 'foo', the path is consulted, but if you do LOAD '$libdir/foo', it is not.  The problem I'm describing is that most extensions use the latter style, per current recommendation in the documentation.

I see; some details here:

  https://www.postgresql.org/docs/current/xfunc-c.html#XFUNC-C-DYNLOAD

And I suppose the `directory` control file variable and `MODULEDIR` make variable make that necessary. 

Maybe $libdir should be stripped out when installing extensions to work with this patch?

Best,

David