Subject: Re: Tarball builds in the new world order

From: Tom Lane <[email-masked]>
Date: Fri, 26 Apr 2024 15:24:08 -0400

Concretely, I'm proposing the attached.  Peter didn't like
PG_COMMIT_HASH, so I have PG_COMMIT_REFSPEC below, but I'm not
wedded to that if a better name is proposed.

			regards, tom lane

diff --git a/GNUmakefile.in b/GNUmakefile.in
index 30553b2a95..27357e5e3b 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -102,10 +102,18 @@ distdir-location:
 # on, Unix machines.
 
 $(distdir).tar.gz:
-	$(GIT) -C $(srcdir) -c core.autocrlf=false archive --format tar.gz -9 --prefix $(distdir)/ HEAD -o $(abs_top_builddir)/$@
+ifneq ($(PG_COMMIT_REFSPEC),)
+	$(GIT) -C $(srcdir) -c core.autocrlf=false archive --format tar.gz -9 --prefix $(distdir)/ $(PG_COMMIT_REFSPEC) -o $(abs_top_builddir)/$@
+else
+	@echo "Please specify PG_COMMIT_REFSPEC." && exit 1
+endif
 
 $(distdir).tar.bz2:
-	$(GIT) -C $(srcdir) -c core.autocrlf=false -c tar.tar.bz2.command='$(BZIP2) -c' archive --format tar.bz2 --prefix $(distdir)/ HEAD -o $(abs_top_builddir)/$@
+ifneq ($(PG_COMMIT_REFSPEC),)
+	$(GIT) -C $(srcdir) -c core.autocrlf=false -c tar.tar.bz2.command='$(BZIP2) -c' archive --format tar.bz2 --prefix $(distdir)/ $(PG_COMMIT_REFSPEC) -o $(abs_top_builddir)/$@
+else
+	@echo "Please specify PG_COMMIT_REFSPEC." && exit 1
+endif
 
 distcheck: dist
 	rm -rf $(dummy)
--- mk-one-release.orig	2024-04-23 17:30:08.983226671 -0400
+++ mk-one-release	2024-04-26 15:17:29.713669677 -0400
@@ -39,13 +39,17 @@ mkdir pgsql
 git archive ${gitref} | tar xf - -C pgsql
 
 # Include the git ref in the output tarballs
+# (This has no effect with v17 and up; instead we rely on "git archive"
+# to include the commit hash in the tar header)
 echo ${gitref} >pgsql/.gitrevision
 
 cd pgsql
 ./configure
 
 # Produce .tar.gz and .tar.bz2 files
-make dist
+# (With v17 and up, this will repeat the "git archive" call; the contents
+# of the working directory don't matter directly to the results.)
+make dist PG_COMMIT_REFSPEC=${gitref}
 
 # Generate md5 and sha256 sums, and copy files to output
 for x in *.tar.*; do

---

From: Alvaro Herrera <[email-masked]>
Date: Sun, 28 Apr 2024 21:16:48 +0200

>  echo ${gitref} >pgsql/.gitrevision

Why is it that the .gitrevision file is only created here, instead of
being added to the tarball that "git archive" produces?  Adding an
argument like
	--add-virtual-file $(distdir)/.gitrevision:$(GIT_REFSPEC)

to the git archive call should suffice.

-- 
Álvaro Herrera               48°01'N 7°57'E  —  https://www.EnterpriseDB.com/
"I can't go to a restaurant and order food because I keep looking at the
fonts on the menu.  Five minutes later I realize that it's also talking
about food" (Donald Knuth)

---

From: Tom Lane <[email-masked]>
Date: Sun, 28 Apr 2024 15:44:10 -0400

> to the git archive call should suffice.

I think we don't want to do that.  In the first place, it's redundant
because "git archive" includes the commit hash in the tar header,
and in the second place it gets away from the concept that the tarball
contains exactly what is in our git tree.

Now admittedly, if anyone's built tooling that relies on the presence
of the .gitrevision file, they might prefer that we keep on including
it.  But I'm not sure anyone has, and in any case I think switching
to the git-approved way of incorporating the hash is the best thing
in the long run.

What I'm thinking of doing, as soon as we've sorted the tarball
creation process, is to make a test tarball available to the
packagers group so that anyone interested can start working on
updating their packaging process for the new approach.  Hopefully,
if anyone's especially unhappy about omitting .gitrevision, they'll
speak up.

			regards, tom lane

---

From: Peter Eisentraut <[email-masked]>
Date: Mon, 29 Apr 2024 14:36:29 +0200

> wedded to that if a better name is proposed.

This seems ok to me, but note that we do have an equivalent 
implementation in meson.  If we don't want to update that in a similar 
way, maybe we should disable it.

---

From: Peter Eisentraut <[email-masked]>
Date: Mon, 29 Apr 2024 14:39:57 +0200

> wedded to that if a better name is proposed.

Um, "refspec" leads me here 
<https://git-scm.com/book/en/v2/Git-Internals-The-Refspec>, which seems 
like the wrong concept.  I think the more correct concept is "revision" 
(https://git-scm.com/docs/gitrevisions), so something like PG_GIT_REVISION?

---

From: Tom Lane <[email-masked]>
Date: Mon, 29 Apr 2024 12:14:10 -0400

> way, maybe we should disable it.

OK.  After poking at that for awhile, it seemed like "default to
HEAD" fits into meson a lot better than "throw an error if the
variable isn't set", so I switched to doing it like that.
One reason is that AFAICT you can only set the variable during
"meson setup" not during "ninja".  This won't matter to the
tarball build script, which does a one-off configuration run
anyway.  But for manual use, a movable target like HEAD might be
more convenient given that behavior.

I tested this by building tarballs using the makefiles on a RHEL8
box, and using meson on my MacBook (with recent MacPorts tools).
I got bit-for-bit identical files, which I found rather impressive
given the gap between the platforms.  Maybe this "reproducible builds"
wheeze will actually work.

I also changed the variable name to PG_GIT_REVISION per your
other suggestion.

			regards, tom lane

diff --git a/GNUmakefile.in b/GNUmakefile.in
index 30553b2a95..9e41794f60 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -87,6 +87,9 @@ update-unicode: | submake-generated-headers submake-libpgport
 distdir	= postgresql-$(VERSION)
 dummy	= =install=
 
+# git revision to be packaged
+PG_GIT_REVISION ?= HEAD
+
 GIT = git
 
 dist: $(distdir).tar.gz $(distdir).tar.bz2
@@ -102,10 +105,10 @@ distdir-location:
 # on, Unix machines.
 
 $(distdir).tar.gz:
-	$(GIT) -C $(srcdir) -c core.autocrlf=false archive --format tar.gz -9 --prefix $(distdir)/ HEAD -o $(abs_top_builddir)/$@
+	$(GIT) -C $(srcdir) -c core.autocrlf=false archive --format tar.gz -9 --prefix $(distdir)/ $(PG_GIT_REVISION) -o $(abs_top_builddir)/$@
 
 $(distdir).tar.bz2:
-	$(GIT) -C $(srcdir) -c core.autocrlf=false -c tar.tar.bz2.command='$(BZIP2) -c' archive --format tar.bz2 --prefix $(distdir)/ HEAD -o $(abs_top_builddir)/$@
+	$(GIT) -C $(srcdir) -c core.autocrlf=false -c tar.tar.bz2.command='$(BZIP2) -c' archive --format tar.bz2 --prefix $(distdir)/ $(PG_GIT_REVISION) -o $(abs_top_builddir)/$@
 
 distcheck: dist
 	rm -rf $(dummy)
diff --git a/meson.build b/meson.build
index cdfd31377d..1c0579d5a6 100644
--- a/meson.build
+++ b/meson.build
@@ -3469,6 +3469,8 @@ bzip2 = find_program('bzip2', required: false, native: true)
 
 distdir = meson.project_name() + '-' + meson.project_version()
 
+pg_git_revision = get_option('PG_GIT_REVISION')
+
 # Note: core.autocrlf=false is needed to avoid line-ending conversion
 # in case the environment has a different setting.  Without this, a
 # tarball created on Windows might be different than on, and unusable
@@ -3483,7 +3485,7 @@ tar_gz = custom_target('tar.gz',
             '-9',
             '--prefix', distdir + '/',
             '-o', join_paths(meson.build_root(), '@OUTPUT@'),
-            'HEAD', '.'],
+            pg_git_revision],
   output: distdir + '.tar.gz',
 )
 
@@ -3497,7 +3499,7 @@ if bzip2.found()
               '--format', 'tar.bz2',
               '--prefix', distdir + '/',
               '-o', join_paths(meson.build_root(), '@OUTPUT@'),
-              'HEAD', '.'],
+              pg_git_revision],
     output: distdir + '.tar.bz2',
   )
 else
diff --git a/meson_options.txt b/meson_options.txt
index 249ecc5ffd..246cecf382 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -55,6 +55,9 @@ option('atomics', type: 'boolean', value: true,
 option('spinlocks', type: 'boolean', value: true,
   description: 'Use spinlocks')
 
+option('PG_GIT_REVISION', type: 'string', value: 'HEAD',
+  description: 'git revision to be packaged by pgdist target')
+
 
 # Compilation options
 
--- mk-one-release.orig	2024-04-23 17:30:08.983226671 -0400
+++ mk-one-release	2024-04-29 12:10:38.106723387 -0400
@@ -39,13 +39,17 @@ mkdir pgsql
 git archive ${gitref} | tar xf - -C pgsql

 cd pgsql
 ./configure
 
 # Produce .tar.gz and .tar.bz2 files
-make dist
+# (With v17 and up, this will repeat the "git archive" call; the contents
+# of the working directory don't matter directly to the results.)
+make dist PG_GIT_REVISION=${gitref}

---

From: Peter Eisentraut <[email-masked]>
Date: Fri, 3 May 2024 15:57:35 +0200

> more convenient given that behavior.

This patch looks good to me.

---

From: Tom Lane <[email-masked]>
Date: Fri, 03 May 2024 11:09:32 -0400

> This patch looks good to me.

Pushed, thanks.

			regards, tom lane