From: Peter Eisentraut <peter@eisentraut.org>
To: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: make dist using git archive
Date: Mon, 22 Jan 2024 08:31:59 +0100
Message-ID: <40e80f77-a294-4f29-a16f-e21bc7bc75fc@eisentraut.org> (raw)
One of the goals is to make the creation of the distribution tarball
more directly traceable to the git repository. That is why we removed
the "make distprep" step.
Here I want to take another step in that direction, by changing "make
dist" to directly use "git archive", rather than the custom shell script
it currently runs.
The simple summary is that it would run
git archive --format tar.gz --prefix postgresql-17devel/ HEAD -o
postgresql-17devel.tar.gz
(with appropriate version numbers, of course), and that's the tarball we
would ship.
There are analogous commands for other compression formats.
The actual command gets subtly more complicated if you need to run this
in a separate build directory. In my attached patch, the make version
doesn't support vpath at the moment, just so that it's easier to
understand for now. The meson version is a bit hairy.
I have studied and tested this quite a bit, and I have found that the
archives produced this way are deterministic and reproducible, meaning
for a given commit the result file should always be bit-for-bit identical.
The exception is that if you use a git version older than 2.38.0, gzip
records the platform in the archive, so you'd get a different output on
Windows vs. macOS vs. "UNIX" (everything else). In git 2.38.0, this was
changed so that everything is recorded as "UNIX" now. This is just
something to keep in mind. This issue is specific to the gzip format,
it does not affect other compression formats.
Meson has its own distribution building command (meson dist), but opted
against using that. The main problem is that the way they have
implemented it, it is not deterministic in the above sense. (Another
point is of course that we probably want a "make" version for the time
being.)
But the target name "dist" in meson is reserved for that reason, so I
needed to call the custom target "pgdist".
I did take one idea from meson: It runs a check before git archive that
the checkout is clean. That way, you avoid mistakes because of
uncommitted changes. This works well in my "make" implementation. In
the meson implementation, I had to find a workaround, because a
custom_target cannot have a dependency on a run_target. As also
mentioned above, the whole meson implementation is a bit ugly.
Anyway, with the attached patch you can do
make dist
or
meson compile -C build pgdist
and it produces the same set of tarballs as before, except it's done
differently.
The actual build scripts need some fine-tuning, but the general idea is
correct, I think.
Attachments:v0-0001-make-dist-uses-git-archive.patch (text/plain, 3.2K) download
From 4b128faca90238d0a0bb6949a8050c2501d1bd67 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Sat, 20 Jan 2024 21:54:36 +0100
Subject: [PATCH v0] make dist uses git archive
---
GNUmakefile.in | 34 ++++++++++++----------------------
meson.build | 38 ++++++++++++++++++++++++++++++++++++++
2 files changed, 50 insertions(+), 22 deletions(-)
diff --git a/GNUmakefile.in b/GNUmakefile.inindex eba569e930e..3e04785ada2 100644--- a/GNUmakefile.in+++ b/GNUmakefile.in@@ -87,29 +87,19 @@ update-unicode: | submake-generated-headers submake-libpgport
distdir = postgresql-$(VERSION)
dummy = =install=
+GIT = git+
dist: $(distdir).tar.gz $(distdir).tar.bz2
- rm -rf $(distdir)--$(distdir).tar: distdir- $(TAR) chf $@ $(distdir)--.INTERMEDIATE: $(distdir).tar--distdir-location:- @echo $(distdir)--distdir:- rm -rf $(distdir)* $(dummy)- for x in `cd $(top_srcdir) && find . \( -name CVS -prune \) -o \( -name .git -prune \) -o -print`; do \- file=`expr X$$x : 'X\./\(.*\)'`; \- if test -d "$(top_srcdir)/$$file" ; then \- mkdir "$(distdir)/$$file" && chmod 777 "$(distdir)/$$file"; \- else \- ln "$(top_srcdir)/$$file" "$(distdir)/$$file" >/dev/null 2>&1 \- || cp "$(top_srcdir)/$$file" "$(distdir)/$$file"; \- fi || exit; \- done- $(MAKE) -C $(distdir) distclean++.PHONY: check-dirty-index+check-dirty-index:+ $(GIT) diff-index --quiet HEAD++$(distdir).tar.gz: check-dirty-index+ $(GIT) archive --format tar.gz --prefix $(distdir)/ HEAD -o $@++$(distdir).tar.bz2: check-dirty-index+ $(GIT) -c tar.tar.bz2.command='$(BZIP2) -c' archive --format tar.bz2 --prefix $(distdir)/ HEAD -o $@
distcheck: dist
rm -rf $(dummy)
diff --git a/meson.build b/meson.buildindex c317144b6bc..f0d870c5192 100644--- a/meson.build+++ b/meson.build@@ -3347,6 +3347,44 @@ run_target('help',+###############################################################+# Distribution archive+###############################################################++git = find_program('git', required: false, native: true, disabler: true)+bzip2 = find_program('bzip2', required: false, native: true, disabler: true)++distdir = meson.project_name() + '-' + meson.project_version()++check_dirty_index = run_target('check-dirty-index',+ command: [git, 'diff-index', '--quiet', 'HEAD'])++tar_gz = custom_target('tar.gz',+ build_always_stale: true,+ command: [git, '-C', '@SOURCE_ROOT@', 'archive',+ '--format', 'tar.gz',+ '--prefix', distdir + '/',+ '-o', '@BUILD_ROOT@/@OUTPUT@',+ 'HEAD', '.'],+ install: false,+ output: distdir + '.tar.gz',+)++tar_bz2 = custom_target('tar.bz2',+ build_always_stale: true,+ command: [git, '-C', '@SOURCE_ROOT@', '-c', 'tar.tar.bz2.command=' + bzip2.path() + ' -c', 'archive',+ '--format', 'tar.bz2',+ '--prefix', distdir + '/',+ '-o', '@BUILD_ROOT@/@OUTPUT@',+ 'HEAD', '.'],+ install: false,+ output: distdir + '.tar.bz2',+)++alias_target('pgdist', [check_dirty_index, tar_gz, tar_bz2])+++
###############################################################
# The End, The End, My Friend
###############################################################
--
2.43.0
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-hackers@postgresql.org
Cc: peter@eisentraut.org
Subject: Re: make dist using git archive
In-Reply-To: <40e80f77-a294-4f29-a16f-e21bc7bc75fc@eisentraut.org>
References: <40e80f77-a294-4f29-a16f-e21bc7bc75fc@eisentraut.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox