pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
testing dist tarballs
6+ messages / 4 participants
[nested] [flat]

* testing dist tarballs
@ 2023-05-24 21:24 Andres Freund <andres@anarazel.de>
  2023-05-24 21:35 ` Re: testing dist tarballs Tom Lane <tgl@sss.pgh.pa.us>
  2023-05-25 14:46 ` Re: testing dist tarballs Christoph Berg <myon@debian.org>
  2023-05-26 07:02 ` Re: testing dist tarballs Peter Eisentraut <peter@eisentraut.org>
  0 siblings, 3 replies; 6+ messages in thread

From: Andres Freund @ 2023-05-24 21:24 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Christoph Berg <myon@debian.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Hi,

On 2023-05-23 14:51:03 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > I guess I need to go and check how long the "release" tarball generation
> > takes...
> 
> It's quick except for the documentation-generating steps.  Maybe
> we could test that part only once?

First thing I noticed that 'make dist' doesn't work in a vpath, failing in a
somewhat obscure way (likely because in a vpath build the the copy from the
source dir doesn't include GNUMakefile). Do we expect it to work?


Besides docs, the slowest part appears to be gzip --best and then bzip2, as
those runs serially and takes 11 and 13 seconds respectively here...

The first thing I tried was:
  make -j8 dist GZIP=pigz BZIP2=pbzip2

unfortunately that results in

pigz: abort: cannot provide files in GZIP environment variable

echo GZIP=pigz >> src/Makefile.custom
echo BZIP2=pbzip2 >> src/Makefile.custom

reduces that to

real	1m6.472s
user	1m28.316s
sys	0m5.340s

real	0m54.811s
user	1m42.078s
sys	0m6.183s

still not great...


OTOH, we currently already build the docs as part of the CompilerWarnings
test. I don't think there's a reason to test that twice?


For me make distcheck currently fails:

In file included from ../../src/include/postgres.h:46,
                 from hashfn.c:24:
../../src/include/utils/elog.h:79:10: fatal error: utils/errcodes.h: No such file or directory
   79 | #include "utils/errcodes.h"
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [<builtin>: hashfn.o] Error 1

at first I thought it was due to my use of -j8 - but it doesn't even work
without that.


That's due to MAKELEVEL:

submake-generated-headers:
ifndef NO_GENERATED_HEADERS
ifeq ($(MAKELEVEL),0)
	$(MAKE) -C $(top_builddir)/src/backend generated-headers
endif
endif

So the distcheck target needs to reset MAKELEVEL=0 - unless somebody has a
better idea?


Separately, it's somewhat confusing that we include errcodes.h etc in
src/backend/utils, rather than its final location, in src/include/utils. It
works, even without perl, because copying the file doesn't require perl, it's
just generating it...

Greetings,

Andres Freund





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: testing dist tarballs
  2023-05-24 21:24 testing dist tarballs Andres Freund <andres@anarazel.de>
@ 2023-05-24 21:35 ` Tom Lane <tgl@sss.pgh.pa.us>
  2 siblings, 0 replies; 6+ messages in thread

From: Tom Lane @ 2023-05-24 21:35 UTC (permalink / raw)
  To: Andres Freund <andres@anarazel.de>; +Cc: Christoph Berg <myon@debian.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Andres Freund <andres@anarazel.de> writes:
> First thing I noticed that 'make dist' doesn't work in a vpath, failing in a
> somewhat obscure way (likely because in a vpath build the the copy from the
> source dir doesn't include GNUMakefile). Do we expect it to work?

Don't see how it could possibly be useful in a vpath, because you'd have
the real source files and the generated files in different trees.

			regards, tom lane





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: testing dist tarballs
  2023-05-24 21:24 testing dist tarballs Andres Freund <andres@anarazel.de>
@ 2023-05-25 14:46 ` Christoph Berg <myon@debian.org>
  2 siblings, 0 replies; 6+ messages in thread

From: Christoph Berg @ 2023-05-25 14:46 UTC (permalink / raw)
  To: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Re: Andres Freund
> That's due to MAKELEVEL:
> 
> submake-generated-headers:
> ifndef NO_GENERATED_HEADERS
> ifeq ($(MAKELEVEL),0)
> 	$(MAKE) -C $(top_builddir)/src/backend generated-headers
> endif
> endif
> 
> So the distcheck target needs to reset MAKELEVEL=0 - unless somebody has a
> better idea?

Fwiw, I've had that problem as well in the Debian packages where
debian/rules is already a Makefile and calling $(MAKE) from there
trips up that logic. The workaround I used is:

override_dh_auto_build-arch:
        # set MAKELEVEL to 0 to force building submake-generated-headers in src/Makefile.global(.in)
        MAKELEVEL=0 $(MAKE) -C build/src all

...
override_dh_auto_test-arch:
ifeq (, $(findstring nocheck, $(DEB_BUILD_OPTIONS)))
        # when tests fail, print newest log files
        # initdb doesn't like LANG and LC_ALL to contradict, unset LANG and LC_CTYPE here
        # temp-install wants to be invoked from a top-level make, unset MAKELEVEL here
        # tell pg_upgrade to create its sockets in /tmp to avoid too long paths
        unset LANG LC_CTYPE MAKELEVEL; ulimit -c unlimited; \
        if ! make -C build check-world \
          $(TEMP_CONFIG) \
          PGSOCKETDIR="/tmp" \
          PG_TEST_EXTRA='ssl' \
          PROVE_FLAGS="--verbose"; \
...

(Just mentioning this, not asking it to be changed.)


Re: Tom Lane
> Andres Freund <andres@anarazel.de> writes:
> > First thing I noticed that 'make dist' doesn't work in a vpath, failing in a
> > somewhat obscure way (likely because in a vpath build the the copy from the
> > source dir doesn't include GNUMakefile). Do we expect it to work?
> 
> Don't see how it could possibly be useful in a vpath, because you'd have
> the real source files and the generated files in different trees.

I don't think "make dist" is generally expected to work in vpath
builds, that's probably one indirection layer too much. (The "make
distcheck" rule generated by automake tests vpath builds, though.)

Christoph





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: testing dist tarballs
  2023-05-24 21:24 testing dist tarballs Andres Freund <andres@anarazel.de>
@ 2023-05-26 07:02 ` Peter Eisentraut <peter@eisentraut.org>
  2023-05-27 18:47   ` Re: testing dist tarballs Andres Freund <andres@anarazel.de>
  2 siblings, 1 reply; 6+ messages in thread

From: Peter Eisentraut @ 2023-05-26 07:02 UTC (permalink / raw)
  To: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Christoph Berg <myon@debian.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

On 24.05.23 23:24, Andres Freund wrote:
> First thing I noticed that 'make dist' doesn't work in a vpath, failing in a
> somewhat obscure way (likely because in a vpath build the the copy from the
> source dir doesn't include GNUMakefile). Do we expect it to work?

I don't think so.

> Separately, it's somewhat confusing that we include errcodes.h etc in
> src/backend/utils, rather than its final location, in src/include/utils. It
> works, even without perl, because copying the file doesn't require perl, it's
> just generating it...

The "copying" is actually a symlink, right?  I don't think we want to 
ship symlinks in the tarball?






^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: testing dist tarballs
  2023-05-24 21:24 testing dist tarballs Andres Freund <andres@anarazel.de>
  2023-05-26 07:02 ` Re: testing dist tarballs Peter Eisentraut <peter@eisentraut.org>
@ 2023-05-27 18:47   ` Andres Freund <andres@anarazel.de>
  2023-05-31 10:52     ` Re: testing dist tarballs Peter Eisentraut <peter@eisentraut.org>
  0 siblings, 1 reply; 6+ messages in thread

From: Andres Freund @ 2023-05-27 18:47 UTC (permalink / raw)
  To: Peter Eisentraut <peter@eisentraut.org>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Christoph Berg <myon@debian.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Hi,

On 2023-05-26 09:02:33 +0200, Peter Eisentraut wrote:
> On 24.05.23 23:24, Andres Freund wrote:
> > First thing I noticed that 'make dist' doesn't work in a vpath, failing in a
> > somewhat obscure way (likely because in a vpath build the the copy from the
> > source dir doesn't include GNUMakefile). Do we expect it to work?
> 
> I don't think so.

Maybe we should just error out in that case, instead of failing in an obscure
way down the line?


> > Separately, it's somewhat confusing that we include errcodes.h etc in
> > src/backend/utils, rather than its final location, in src/include/utils. It
> > works, even without perl, because copying the file doesn't require perl, it's
> > just generating it...
> 
> The "copying" is actually a symlink, right?  I don't think we want to ship
> symlinks in the tarball?

Fair point - still seems we should just create the files in the right
directory instead of doing it in the wrong place and then creating symlinks to
make them accessible...

Greetings,

Andres Freund





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: testing dist tarballs
  2023-05-24 21:24 testing dist tarballs Andres Freund <andres@anarazel.de>
  2023-05-26 07:02 ` Re: testing dist tarballs Peter Eisentraut <peter@eisentraut.org>
  2023-05-27 18:47   ` Re: testing dist tarballs Andres Freund <andres@anarazel.de>
@ 2023-05-31 10:52     ` Peter Eisentraut <peter@eisentraut.org>
  0 siblings, 0 replies; 6+ messages in thread

From: Peter Eisentraut @ 2023-05-31 10:52 UTC (permalink / raw)
  To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Christoph Berg <myon@debian.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

On 27.05.23 14:47, Andres Freund wrote:
>>> Separately, it's somewhat confusing that we include errcodes.h etc in
>>> src/backend/utils, rather than its final location, in src/include/utils. It
>>> works, even without perl, because copying the file doesn't require perl, it's
>>> just generating it...
>>
>> The "copying" is actually a symlink, right?  I don't think we want to ship
>> symlinks in the tarball?
> 
> Fair point - still seems we should just create the files in the right
> directory instead of doing it in the wrong place and then creating symlinks to
> make them accessible...

Right.  I think the reason this was set up this way is that with make it 
is generally dubious to create target files outside of the current 
directory.






^ permalink  raw  reply  [nested|flat] 6+ messages in thread


end of thread, other threads:[~2023-05-31 10:52 UTC | newest]

Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-05-24 21:24 testing dist tarballs Andres Freund <andres@anarazel.de>
2023-05-24 21:35 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-05-25 14:46 ` Christoph Berg <myon@debian.org>
2023-05-26 07:02 ` Peter Eisentraut <peter@eisentraut.org>
2023-05-27 18:47   ` Andres Freund <andres@anarazel.de>
2023-05-31 10:52     ` Peter Eisentraut <peter@eisentraut.org>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox