public inbox for [email protected]
help / color / mirror / Atom feedpg18 patch: separate package for llvm/jit
4+ messages / 2 participants
[nested] [flat]
* pg18 patch: separate package for llvm/jit
@ 2025-02-24 21:48 Jeremy Schneider <[email protected]>
2025-02-28 15:02 ` Re: pg18 patch: separate package for llvm/jit Christoph Berg <[email protected]>
0 siblings, 1 reply; 4+ messages in thread
From: Jeremy Schneider @ 2025-02-24 21:48 UTC (permalink / raw)
To: [email protected]
Hello Debian hackers,
Last month I asked if we could make LLVM/JIT a choice for installation,
similar to procedural languages like perl, python and tcl. [1] Today it
is part of the main postgresql package.
I would like to propose this change for Postgres 18.
Our motivation is that LLVM/JIT alone increases the total size of the
docker image by 67% (not that far from doubling it), and with many
installations and frequent deployments and updates across a large
fleet, this space savings will be worthwhile when LLVM/JIT is not
needed.
After last month's discussion, in a short thread on the hackers list, I
learned that Postgres explicitly supports building with LLVM/JIT
support and running without the library files, which enables separating
LLVM/JIT into a separate package. [2]
I also learned that the PGDG rpm packagers have already been using this
approach for some time. [3]
Attached are a few patches for debian packaging which I used to
initially test the same pattern as rpm packaging. There are a few small
outstanding items before they are commit-ready, but this enabled me to
confirm that the approach works with debian's build system. Not only
did it successfully remove the LLVM/JIT dependency from the main
package and attach it to the separate package, but it even reduced the
Installed-Size from 57726kb down to 34335kb (which I did not expect).
Those bitcode files are a lot bigger than I realized.
I named the new package postgresql-17-jit-llvm to align with the
current "provides" naming
A few outstanding items to make these commit-ready:
1. Update the provides/requires for postgresql and jit-llvm (postgresql
should still recommend LLVM/JIT)
2. Figure out what happens with this new package on architectures that
don't support LLVM/JIT (empty package? README explanation?)
3. Make sure the bitcode directory itself is created by the main
package so that extensions can use the directory
4. Remove the perl, python and tcl bitcode files from the llvm package
(they are included in the PL packages)
Note that patches are needed against both postgresql-common and
postgresql repositories on salsa (hence two attachments)
Thanks
-Jeremy
1: https://postgr.es/m/20250109005301.3b145092%40jeremy-ThinkPad-T430s
2: https://postgr.es/m/20250111183709.4691fd17%40ardentperf.com
3:
https://git.postgresql.org/gitweb/?p=pgrpms.git;a=blob;f=rpm/redhat/main/non-common/postgresql-17/ma...
===== without patches:
dpkg -I ../pg17/postgresql-17_17.4-1_amd64.deb
new Debian package, version 2.0.
size 17867268 bytes: control archive=41861 bytes.
1524 bytes, 23 lines control
146703 bytes, 1571 lines md5sums
190 bytes, 13 lines * postinst #!/bin/sh
2642 bytes, 86 lines * postrm #!/bin/sh
349 bytes, 18 lines * preinst #!/bin/sh
199 bytes, 16 lines * prerm #!/bin/sh
4554 bytes, 29 lines templates
Package: postgresql-17
Version: 17.4-1
Architecture: amd64
Maintainer: Debian PostgreSQL Maintainers
<[email protected]> Installed-Size: 57726
Depends: locales | locales-all, postgresql-client-17,
postgresql-common (>= 252~), ssl-cert, tzdata, debconf (>= 0.5) |
debconf-2.0, libc6 (>= 2.38), libgcc-s1 (>= 3.3.1), libgssapi-krb5-2
(>= 1.14+dfsg), libicu74 (>= 74.1-1~), libldap2 (>= 2.6.2), libllvm19,
liblz4-1 (>= 0.0~r130), libpam0g (>= 0.99.7.1), libpq5 (>= 17~~),
libselinux1 (>= 3.1~), libssl3t64 (>= 3.0.0), libstdc++6 (>= 5.2),
libsystemd0, libuuid1 (>= 2.16), libxml2 (>= 2.7.4), libxslt1.1 (>=
1.1.25), libzstd1 (>= 1.5.5), zlib1g (>= 1:1.1.4) Recommends: sysstat
Breaks: dbconfig-common (<< 2.0.22~) Provides: postgresql-17-jit-llvm
(= 19), postgresql-contrib-17 Section: database Priority: optional
Homepage: http://www.postgresql.org/ Description: The World's Most
Advanced Open Source Relational Database PostgreSQL, also known as
Postgres, is a free and open-source relational database management
system (RDBMS) emphasizing extensibility and SQL compliance. It
features transactions with Atomicity, Consistency, Isolation,
Durability (ACID) properties, automatically updatable views,
materialized views, triggers, foreign keys, and stored procedures. It
is designed to handle a range of workloads, from single machines to
data warehouses or Web services with many concurrent users. . This
package provides the database server for PostgreSQL 17.
Postgresql-Catversion: 202406281-1700
===== with patches:
dpkg -I postgresql-17_17.4-1_amd64.deb
new Debian package, version 2.0.
size 7875442 bytes: control archive=18570 bytes.
1470 bytes, 23 lines control
50801 bytes, 602 lines md5sums
190 bytes, 13 lines * postinst #!/bin/sh
2642 bytes, 86 lines * postrm #!/bin/sh
349 bytes, 18 lines * preinst #!/bin/sh
199 bytes, 16 lines * prerm #!/bin/sh
4554 bytes, 29 lines templates
Package: postgresql-17
Version: 17.4-1
Architecture: amd64
Maintainer: Debian PostgreSQL Maintainers
<[email protected]> Installed-Size: 34335
Depends: locales | locales-all, postgresql-client-17,
postgresql-common (>= 252~), ssl-cert, tzdata, debconf (>= 0.5) |
debconf-2.0, libc6 (>= 2.38), libgssapi-krb5-2 (>= 1.14+dfsg), libicu74
(>= 74.1-1~), libldap2 (>= 2.6.2), liblz4-1 (>= 0.0~r130), libpam0g (>=
0.99.7.1), libpq5 (>= 17~~), libselinux1 (>= 3.1~), libssl3t64 (>=
3.0.0), libsystemd0, libuuid1 (>= 2.16), libxml2 (>= 2.7.4), libxslt1.1
(>= 1.1.25), libzstd1 (>= 1.5.5), zlib1g (>= 1:1.1.4) Recommends:
sysstat Breaks: dbconfig-common (<< 2.0.22~) Provides:
postgresql-17-jit-llvm (= 19), postgresql-contrib-17 Section: database
Priority: optional Homepage: http://www.postgresql.org/ Description:
The World's Most Advanced Open Source Relational Database PostgreSQL,
also known as Postgres, is a free and open-source relational database
management system (RDBMS) emphasizing extensibility and SQL compliance.
It features transactions with Atomicity, Consistency, Isolation,
Durability (ACID) properties, automatically updatable views,
materialized views, triggers, foreign keys, and stored procedures. It
is designed to handle a range of workloads, from single machines to
data warehouses or Web services with many concurrent users. . This
package provides the database server for PostgreSQL 17.
Postgresql-Catversion: 202406281-1700
Attachments:
[text/x-patch] postgresql-01-jit-llvm.patch (2.4K, 2-postgresql-01-jit-llvm.patch)
download | inline diff:
diff --git a/debian/control b/debian/control
index 66e3184..96d3fcc 100644
--- a/debian/control
+++ b/debian/control
@@ -298,3 +298,20 @@ Description: PL/Tcl procedural language for PostgreSQL 17
PostgreSQL 17 functions that use the languages pltcl or pltclu.
.
PostgreSQL is an object-relational SQL database management system.
+
+Package: postgresql-17-jit-llvm
+Architecture: any
+Depends:
+ postgresql-17 (= ${binary:Version}),
+ ${misc:Depends},
+ ${shlibs:Depends},
+Provides:
+ postgresql-jit-llvm,
+Description: JIT/LLVM support for PostgreSQL 17
+ Just-in-Time (JIT) compilation is the process of turning some
+ form of interpreted program evaluation into a native program,
+ and doing so at run time. In some cases this can provide
+ performance benefits.
+ .
+ PostgreSQL is an object-relational SQL database management system.
+
diff --git a/debian/postgresql-17-jit-llvm.install b/debian/postgresql-17-jit-llvm.install
new file mode 100755
index 0000000..2c9d974
--- /dev/null
+++ b/debian/postgresql-17-jit-llvm.install
@@ -0,0 +1,5 @@
+#!/usr/bin/dh-exec
+
+usr/lib/postgresql/*/lib/llvmjit.so
+[!alpha !hppa !hurd-amd64 !hurd-i386 !ia64 !kfreebsd-amd64 !kfreebsd-i386 !loong64 !m68k !powerpc !riscv64 !sh4 !sparc64 !x32] <!pkg.postgresql.nollvm> usr/lib/postgresql/*/lib/bitcode
+[!alpha !hppa !hurd-amd64 !hurd-i386 !ia64 !kfreebsd-amd64 !kfreebsd-i386 !loong64 !m68k !powerpc !riscv64 !sh4 !sparc64 !x32] <!pkg.postgresql.nollvm> usr/lib/postgresql/*/lib/llvmjit_types.bc
diff --git a/debian/postgresql-17-jit-llvm.lintian-overrides b/debian/postgresql-17-jit-llvm.lintian-overrides
new file mode 100644
index 0000000..db59083
--- /dev/null
+++ b/debian/postgresql-17-jit-llvm.lintian-overrides
@@ -0,0 +1,20 @@
+# We test for /usr/bin/pg_dropcluster, but run it without path
+command-with-path-in-maintainer-script
+
+# The World's Most Advanced Open Source Relational Database
+description-synopsis-starts-with-article
+
+# We ship binaries and libs in subdirs of /usr/lib/postgresql
+executable-in-usr-lib
+repeated-path-segment lib *
+
+# These are PostgreSQL server plugins; some need no external libraries
+hardening-no-fortify-functions [usr/lib/postgresql/*/lib/*]
+library-not-linked-against-libc [usr/lib/postgresql/*/lib/*]
+shared-library-lacks-prerequisites [usr/lib/postgresql/*/lib/*]
+
+# We use debconf in postrm only
+no-debconf-config
+
+# We store the PostgreSQL catalog version in a custom control field
+unknown-field *Postgresql-Catversion
[text/x-patch] postgresql-common-01-jit-llvm.patch (905B, 3-postgresql-common-01-jit-llvm.patch)
download | inline diff:
diff --git a/server/postgresql.mk b/server/postgresql.mk
index 91dc8b2..4883c82 100644
--- a/server/postgresql.mk
+++ b/server/postgresql.mk
@@ -242,6 +242,11 @@ override_dh_install-arch:
bash -c "rm -v debian/postgresql-$(MAJOR_PKG)/usr/lib/postgresql/$(MAJOR_VER)/lib/{plperl,plpython,pltcl,*_pl}*"
bash -c "rm -rfv debian/postgresql-$(MAJOR_PKG)/usr/lib/postgresql/$(MAJOR_VER)/lib/bitcode/*{plperl,plpython,pltcl}*"
+ # these are shipped in the jit/llvm package
+ bash -c "rm -v debian/postgresql-$(MAJOR_PKG)/usr/lib/postgresql/$(MAJOR_VER)/lib/llvmjit.so"
+ bash -c "rm -v debian/postgresql-$(MAJOR_PKG)/usr/lib/postgresql/$(MAJOR_VER)/lib/llvmjit_types.bc"
+ bash -c "rm -rfv debian/postgresql-$(MAJOR_PKG)/usr/lib/postgresql/$(MAJOR_VER)/lib/bitcode"
+
# record catversion in a file
echo $(CATVERSION) > debian/postgresql-$(MAJOR_PKG)/usr/share/postgresql/$(MAJOR_VER)/catalog_version
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pg18 patch: separate package for llvm/jit
2025-02-24 21:48 pg18 patch: separate package for llvm/jit Jeremy Schneider <[email protected]>
@ 2025-02-28 15:02 ` Christoph Berg <[email protected]>
2025-03-02 00:51 ` Re: pg18 patch: separate package for llvm/jit Jeremy Schneider <[email protected]>
0 siblings, 1 reply; 4+ messages in thread
From: Christoph Berg @ 2025-02-28 15:02 UTC (permalink / raw)
To: Jeremy Schneider <[email protected]>; +Cc: [email protected]
Re: Jeremy Schneider
> I would like to propose this change for Postgres 18.
Hi,
I committed a change to the PG18 packaging that implements that split.
The new package is called postgresql-18-jit.
There is more work to do on each of the extension packages currently
depending on "postgresql-18-jit-llvm (= llvmversion)", that needs to be
converted to "Breaks: postgresql-18-jit-llvm (<< llvmversion)". (This
is also the reason the jit package is not called like that because the
version number there is not the PG version number.)
This will likely happen when extensions are moved to PG18 in
September.
I'm unsure if the split should be backported to PG 17 and earlier
since it will affect production systems in some way.
Christoph
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pg18 patch: separate package for llvm/jit
2025-02-24 21:48 pg18 patch: separate package for llvm/jit Jeremy Schneider <[email protected]>
2025-02-28 15:02 ` Re: pg18 patch: separate package for llvm/jit Christoph Berg <[email protected]>
@ 2025-03-02 00:51 ` Jeremy Schneider <[email protected]>
2025-03-02 01:06 ` Re: pg18 patch: separate package for llvm/jit Jeremy Schneider <[email protected]>
0 siblings, 1 reply; 4+ messages in thread
From: Jeremy Schneider @ 2025-03-02 00:51 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; +Cc: [email protected]
On Fri, 28 Feb 2025 16:02:42 +0100
Christoph Berg <[email protected]> wrote:
> Re: Jeremy Schneider
> > I would like to propose this change for Postgres 18.
>
> Hi,
>
> I committed a change to the PG18 packaging that implements that split.
> The new package is called postgresql-18-jit.
wow!! i did not expect this 🙂
> There is more work to do on each of the extension packages currently
> depending on "postgresql-18-jit-llvm (= llvmversion)", that needs to
> be converted to "Breaks: postgresql-18-jit-llvm (<< llvmversion)".
> (This is also the reason the jit package is not called like that
> because the version number there is not the PG version number.)
>
> This will likely happen when extensions are moved to PG18 in
> September.
i'll take a look and learn a bit more about it.
one question - the commit message says "Closes: #927182" and i'm
curious what that number is a reference to? It doesn't seem to be an
issue or PR that I found on salsa.debian.org but maybe I haven't looked
in the right place yet.
> I'm unsure if the split should be backported to PG 17 and earlier
> since it will affect production systems in some way.
FWIW, i'm not a debian packaging expert but i was assuming this would
not be backported. i don't think we'd want existing installation
scripts to change behavior on a postgres minor update (eg. newly
provisioned servers no longer getting JIT installed by default, when
previously provisioned servers of the same major version have it)
-Jeremy
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pg18 patch: separate package for llvm/jit
2025-02-24 21:48 pg18 patch: separate package for llvm/jit Jeremy Schneider <[email protected]>
2025-02-28 15:02 ` Re: pg18 patch: separate package for llvm/jit Christoph Berg <[email protected]>
2025-03-02 00:51 ` Re: pg18 patch: separate package for llvm/jit Jeremy Schneider <[email protected]>
@ 2025-03-02 01:06 ` Jeremy Schneider <[email protected]>
0 siblings, 0 replies; 4+ messages in thread
From: Jeremy Schneider @ 2025-03-02 01:06 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; +Cc: [email protected]
On Sat, 1 Mar 2025 16:51:49 -0800
Jeremy Schneider <[email protected]> wrote:
> one question - the commit message says "Closes: #927182" and i'm
> curious what that number is a reference to? It doesn't seem to be an
> issue or PR that I found on salsa.debian.org but maybe I haven't
> looked in the right place yet.
nevermind, that was obvious
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=927182
also cf
https://www.debian.org/doc/manuals/debmake-doc/ch06.en.html#changelog
Crazy that bug has been open since 2019. I'll go learn a bit more about
the debian bug tracking system.
I've now bookmarked
https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=postgresql-17
This raises a higher level question... is this mailing list the right
place for this kind of discussion?
In my first pass through, I didn't see anything at
https://wiki.postgresql.org/wiki/Apt or
https://wiki.postgresql.org/wiki/Apt/FAQ about "where is the best place
to ask questions" or "how to check if a bug/issue was already reported"
or "how to raise a question about new feature or change discussions"
-Jeremy
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2025-03-02 01:06 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-02-24 21:48 pg18 patch: separate package for llvm/jit Jeremy Schneider <[email protected]>
2025-02-28 15:02 ` Christoph Berg <[email protected]>
2025-03-02 00:51 ` Jeremy Schneider <[email protected]>
2025-03-02 01:06 ` Jeremy Schneider <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox