Mime-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8; format=Flowed
Date: Wed, 24 Jan 2024 11:57:08 -0600
Message-Id: <CYN4PXK5J7KP.247K22MR0MM1G@neon.tech>
Subject: Re: make dist using git archive
To: "Peter Eisentraut" <peter@eisentraut.org>
From: "Tristan Partin" <tristan@neon.tech>
Cc: "pgsql-hackers" <pgsql-hackers@postgresql.org>
References: <40e80f77-a294-4f29-a16f-e21bc7bc75fc@eisentraut.org>
 <CYLI6IZ5RAVR.3RBBF4F3P8MHX@neon.tech>
 <02714192-6eba-4145-ac1c-0a245801014f@eisentraut.org>
 <CYN2MF9LPZFX.3P1LF1G55RXM4@neon.tech>
In-Reply-To: <CYN2MF9LPZFX.3P1LF1G55RXM4@neon.tech>
Archived-At: 
 <https://www.postgresql.org/message-id/CYN4PXK5J7KP.247K22MR0MM1G%40neon.tech>
Precedence: bulk

On Wed Jan 24, 2024 at 10:18 AM CST, Tristan Partin wrote:
> On Tue Jan 23, 2024 at 3:30 AM CST, Peter Eisentraut wrote:
> > On 22.01.24 21:04, Tristan Partin wrote:
> > > I am not really following why we can't use the builtin Meson dist=20
> > > command. The only difference from my testing is it doesn't use a=20
> > > --prefix argument.
> >
> > Here are some problems I have identified:
> >
> > 1. meson dist internally runs gzip without the -n option.  That makes=
=20
> > the tar.gz archive include a timestamp, which in turn makes it not=20
> > reproducible.

It doesn't look like Python provides the facilities to affect this.

> > 2. Because gzip includes a platform indicator in the archive, the=20
> > produced tar.gz archive is not reproducible across platforms.  (I don't=
=20
> > know if gzip has an option to avoid that.  git archive uses an internal=
=20
> > gzip implementation that handles this.)

Same reason as above.

> > 3. Meson does not support tar.bz2 archives.

Submitted https://github.com/mesonbuild/meson/pull/12770.

> > 4. Meson uses git archive internally, but then unpacks and repacks the=
=20
> > archive, which loses the ability to use git get-tar-commit-id.

Because Meson allows projects to distribute arbitrary files via=20
meson.add_dist_script(), and can include subprojects via `meson dist=20
--include-subprojects`, this doesn't seem like an easily solvable=20
problem.

> > 5. I have found that the tar archives created by meson and git archive=
=20
> > include the files in different orders.  I suspect that the Python=20
> > tarfile module introduces some either randomness or platform dependency=
.

Seems likely.

> > 6. meson dist is also slower because of the additional work.

Not easily solvable due to 4.

> > 7. meson dist produces .sha256sum files but we have called them .sha256=
.=20
> >   (This is obviously trivial, but it is something that would need to be=
=20
> > dealt with somehow nonetheless.)
> >
> > Most or all of these issues are fixable, either upstream in Meson or by=
=20
> > adjusting our own requirements.  But for now this route would have some=
=20
> > significant disadvantages.
>
> Thanks Peter. I will bring these up with upstream!

I think the solution to point 4 is to not unpack/repack if there are no=20
dist scripts and/or subprojects to distribute. I can take a look at=20
this later. I think this would also solve points 1, 2, 5, and 6 because=20
at that point meson is just calling git-archive.

--=20
Tristan Partin
Neon (https://neon.tech)