pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Peter Eisentraut <peter@eisentraut.org>
To: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: make dist using git archive
Date: Mon, 22 Jan 2024 08:31:59 +0100
Message-ID: <40e80f77-a294-4f29-a16f-e21bc7bc75fc@eisentraut.org> (raw)

One of the goals is to make the creation of the distribution tarball 
more directly traceable to the git repository.  That is why we removed 
the "make distprep" step.

Here I want to take another step in that direction, by changing "make 
dist" to directly use "git archive", rather than the custom shell script 
it currently runs.

The simple summary is that it would run

git archive --format tar.gz --prefix postgresql-17devel/ HEAD -o 
postgresql-17devel.tar.gz

(with appropriate version numbers, of course), and that's the tarball we 
would ship.

There are analogous commands for other compression formats.

The actual command gets subtly more complicated if you need to run this 
in a separate build directory.  In my attached patch, the make version 
doesn't support vpath at the moment, just so that it's easier to 
understand for now.  The meson version is a bit hairy.

I have studied and tested this quite a bit, and I have found that the 
archives produced this way are deterministic and reproducible, meaning 
for a given commit the result file should always be bit-for-bit identical.

The exception is that if you use a git version older than 2.38.0, gzip 
records the platform in the archive, so you'd get a different output on 
Windows vs. macOS vs. "UNIX" (everything else).  In git 2.38.0, this was 
changed so that everything is recorded as "UNIX" now.  This is just 
something to keep in mind.  This issue is specific to the gzip format, 
it does not affect other compression formats.

Meson has its own distribution building command (meson dist), but opted 
against using that.  The main problem is that the way they have 
implemented it, it is not deterministic in the above sense.  (Another 
point is of course that we probably want a "make" version for the time 
being.)

But the target name "dist" in meson is reserved for that reason, so I 
needed to call the custom target "pgdist".

I did take one idea from meson: It runs a check before git archive that 
the checkout is clean.  That way, you avoid mistakes because of 
uncommitted changes.  This works well in my "make" implementation.  In 
the meson implementation, I had to find a workaround, because a 
custom_target cannot have a dependency on a run_target.  As also 
mentioned above, the whole meson implementation is a bit ugly.

Anyway,  with the attached patch you can do

     make dist

or

     meson compile -C build pgdist

and it produces the same set of tarballs as before, except it's done 
differently.

The actual build scripts need some fine-tuning, but the general idea is 
correct, I think.

Attachments:

v0-0001-make-dist-uses-git-archive.patch (text/plain, 3.2K) download From 4b128faca90238d0a0bb6949a8050c2501d1bd67 Mon Sep 17 00:00:00 2001 From: Peter Eisentraut <peter@eisentraut.org> Date: Sat, 20 Jan 2024 21:54:36 +0100 Subject: [PATCH v0] make dist uses git archive --- GNUmakefile.in | 34 ++++++++++++---------------------- meson.build | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+), 22 deletions(-) diff --git a/GNUmakefile.in b/GNUmakefile.in index eba569e930e..3e04785ada2 100644 --- a/GNUmakefile.in +++ b/GNUmakefile.in @@ -87,29 +87,19 @@ update-unicode: | submake-generated-headers submake-libpgport distdir = postgresql-$(VERSION) dummy = =install= +GIT = git + dist: $(distdir).tar.gz $(distdir).tar.bz2 - rm -rf $(distdir) - -$(distdir).tar: distdir - $(TAR) chf $@ $(distdir) - -.INTERMEDIATE: $(distdir).tar - -distdir-location: - @echo $(distdir) - -distdir: - rm -rf $(distdir)* $(dummy) - for x in `cd $(top_srcdir) && find . \( -name CVS -prune \) -o \( -name .git -prune \) -o -print`; do \ - file=`expr X$$x : 'X\./\(.*\)'`; \ - if test -d "$(top_srcdir)/$$file" ; then \ - mkdir "$(distdir)/$$file" && chmod 777 "$(distdir)/$$file"; \ - else \ - ln "$(top_srcdir)/$$file" "$(distdir)/$$file" >/dev/null 2>&1 \ - || cp "$(top_srcdir)/$$file" "$(distdir)/$$file"; \ - fi || exit; \ - done - $(MAKE) -C $(distdir) distclean + +.PHONY: check-dirty-index +check-dirty-index: + $(GIT) diff-index --quiet HEAD + +$(distdir).tar.gz: check-dirty-index + $(GIT) archive --format tar.gz --prefix $(distdir)/ HEAD -o $@ + +$(distdir).tar.bz2: check-dirty-index + $(GIT) -c tar.tar.bz2.command='$(BZIP2) -c' archive --format tar.bz2 --prefix $(distdir)/ HEAD -o $@ distcheck: dist rm -rf $(dummy) diff --git a/meson.build b/meson.build index c317144b6bc..f0d870c5192 100644 --- a/meson.build +++ b/meson.build @@ -3347,6 +3347,44 @@ run_target('help', +############################################################### +# Distribution archive +############################################################### + +git = find_program('git', required: false, native: true, disabler: true) +bzip2 = find_program('bzip2', required: false, native: true, disabler: true) + +distdir = meson.project_name() + '-' + meson.project_version() + +check_dirty_index = run_target('check-dirty-index', + command: [git, 'diff-index', '--quiet', 'HEAD']) + +tar_gz = custom_target('tar.gz', + build_always_stale: true, + command: [git, '-C', '@SOURCE_ROOT@', 'archive', + '--format', 'tar.gz', + '--prefix', distdir + '/', + '-o', '@BUILD_ROOT@/@OUTPUT@', + 'HEAD', '.'], + install: false, + output: distdir + '.tar.gz', +) + +tar_bz2 = custom_target('tar.bz2', + build_always_stale: true, + command: [git, '-C', '@SOURCE_ROOT@', '-c', 'tar.tar.bz2.command=' + bzip2.path() + ' -c', 'archive', + '--format', 'tar.bz2', + '--prefix', distdir + '/', + '-o', '@BUILD_ROOT@/@OUTPUT@', + 'HEAD', '.'], + install: false, + output: distdir + '.tar.bz2', +) + +alias_target('pgdist', [check_dirty_index, tar_gz, tar_bz2]) + + + ############################################################### # The End, The End, My Friend ############################################################### -- 2.43.0

view thread (30+ messages)  latest in thread

Message-ID: <40e80f77-a294-4f29-a16f-e21bc7bc75fc@eisentraut.org>
Permalink:  ../40e80f77-a294-4f29-a16f-e21bc7bc75fc@eisentraut.org/
Also on:    postgresql.org/message-id/40e80f77-a294-4f29-a16f-e21bc7bc75fc@eisentraut.org

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-hackers@postgresql.org
  Cc: peter@eisentraut.org
  Subject: Re: make dist using git archive
  In-Reply-To: <40e80f77-a294-4f29-a16f-e21bc7bc75fc@eisentraut.org>
  References: <40e80f77-a294-4f29-a16f-e21bc7bc75fc@eisentraut.org>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox