Subject: BuildFarm client release 4.19

From: Andrew Dunstan <[email-masked]>
Date: Wed, 19 Apr 2017 09:59:29 -0400

I have released version 4.19 of the PostgreSQL Buildfarm client. It can
be downloaded from
<https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz>

Apart from some minor bug fixes, the following changes are made:

  * Include the script's path in @INC. That means you can usually run
    the script from anywhere rather than just its own directory.
  * Set TZ after "make check" is run. this makes for slightly faster
    initdb runs in later stages.
  * make TAP tests run with --timer
  * change default log_line_prefix in config file to use %p instead of
    %c. That makes it easier to tie log messages to other messages that
    mention a pid
  * Add a module to log running commands to a file as it runs and
    replace critical uses of `` with the new procedure. That means we
    have better traces if the buildfarm process crashes.
  * Improvements to TAP tests logging and coverage. Each test set (i.e
    each /t directory) is now logged separately. Coverage is extended to
    remaining test sets in src/test except SSL tests.
  * Add a module for testing ICU
  * change target URL in config file sample to use canonical name
    matching SSL certificate
  * default build_root to a directory in the same place that the script
    is found, and create it if it doesn't exist.
  * add a command-line flag to allow setting config settings
  * from-source path can now be relative
  * from-source doesn't need to specify a branch, if not clear it
    defaults to HEAD
  * from-source can now do vpath builds

These changes mean that the client is more useful for testing
development code, and also that testing config settings is much simpler.
An initial test run on a fresh installation is now as simple as:

    cp buildfarm.conf.sample build-farm.conf

    ./run_build.pl --test --verbose

To test development code, the from-source option is now much more
flexible and friendly. For example, one might do something like:

    ./run_build.pl --from-source=/path/to/postgresql \

       --config-set use_vpath=1 \

       --config-set config_opts+=--enable-tap-tests \

       --config-set locales+=en_US.utf8

If you run something like this against your development code, and it
succeeds, you can be reasonably confident that if committed it won't
sent the buildfarm all red.

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

-- 
Sent via buildfarm-members mailing list ([email-masked])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members

---

From: Tom Lane <[email-masked]>
Date: Wed, 19 Apr 2017 13:38:14 -0400

> <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz>

Nice work!

>     remaining test sets in src/test except SSL tests.

I was going to plead with buildfarm owners who use --enable-tap-tests to
update sooner rather than later, because this separate-logging behavior
is going to make it a lot less painful to decipher failures during the
TAP tests.  However, in view of my results in
https://www.postgresql.org/message-id/21358.1492622881%40sss.pgh.pa.us
I can't really recommend that owners of slower machines update just yet.
Hopefully we can do something to trim down the added runtime.

> ...

These things are pretty cool, but maybe they need to be documented at
https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto
?

			regards, tom lane

---

From: Andrew Dunstan <[email-masked]>
Date: Wed, 19 Apr 2017 13:56:37 -0400

>> <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz>

Thank you.

> Hopefully we can do something to trim down the added runtime.

As I mentioned in reply to that thread, you can disable the extra TAP
tests with this addition to the command line:

    --skip-steps=misc-check

> https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto

Indeed. I am planning to.

cheers

andrew

-- 
Sent via pgsql-hackers mailing list ([email-masked])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

---

From: Mikael Kjellström <[email-masked]>
Date: Mon, 1 May 2017 20:13:02 +0200

> <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz>

I don't know if it's only me or if others have noticed this also but I 
have the buildfarm client set up like this:

/home/pgbf
/home/pgbf/build-farm-4
/home/pgbf/buildfarm@ -> build-farm-4.19/
/home/pgbf/buildroot

and then I have a cron job that does this:

x y * * * cd /home/pgbf/buildfarm && ./run_build.pl REL9_6_STABLE

and when it tries to run I get the following error:

cannot create 
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log: 
No such file or directory
gmake is not GNU Make - please fix config file at ./run_build.pl line 343.

if I manually create the directory loach.lastrun-logs the script then 
continues normally.

I am no perl guru so I can't figure out really what is going on here.  I 
think that the directory 
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs needs to be 
created before the PGBuild/Utils.pm sub run_log tries to write to it.

/Mikael

---

From: Andrew Dunstan <[email-masked]>
Date: Mon, 1 May 2017 14:16:43 -0400

On 05/01/2017 02:13 PM, Mikael Kjellström wrote:
> I am no perl guru so I can't figure out really what is going on here. 
> I think that the directory

I will check on it. Thanks for the report.

cheers

andrew

---

From: Tom Lane <[email-masked]>
Date: Mon, 01 May 2017 14:19:15 -0400

> if I manually create the directory loach.lastrun-logs the script then 

FWIW, I haven't seen that, but probably that's because the directory
already exists in each branch's working directory.  I bet zapping the
whole working directory for a branch would reproduce it.

			regards, tom lane

---

From: Andrew Dunstan <[email-masked]>
Date: Mon, 1 May 2017 14:25:08 -0400

> created before the PGBuild/Utils.pm sub run_log tries to write to it.

OK, that's a bug. Mea culpa.

the quick fix is this patch:

    diff --git a/run_build.pl b/run_build.pl
    index aeb8966..822b4de 100755
    --- a/run_build.pl
    +++ b/run_build.pl
    @@ -1008,7 +1008,8 @@ sub writelog
     
     sub check_make
     {
    -    my @out = run_log("$make -v");
    +   # don't use run_log here - it's too early in the process
    +    my @out = `$make -v 2>&1`;
         return undef unless ($? == 0 && grep {/GNU Make/} @out);
         return 'OK';
     }

cheers

andrew

---

From: Mikael Kjellström <[email-masked]>
Date: Mon, 1 May 2017 20:34:32 +0200

> whole working directory for a branch would reproduce it.

Yes, zapping the whole working directory for each branch reproduces it 
for me every time.

/Mikael

---

From: Mikael Kjellström <[email-masked]>
Date: Mon, 1 May 2017 20:44:53 +0200

>          return undef unless ($? == 0 && grep {/GNU Make/} @out);

Nope, that didn't do it.

If I zap REL9_6_STABLE and then run:

./run_build.pl REL9_6_STABLE

I get the following output:

cannot create 
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log: 
No such file or directory
cannot create 
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log: 
No such file or directory
cannot create 
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log: 
No such file or directory
Buildfarm member loach failed on REL9_6_STABLE stage pgsql-Git

/Mikael

---

From: Mikael Kjellström <[email-masked]>
Date: Mon, 1 May 2017 20:46:40 +0200

> Nope, that didn't do it.

Or well.  It fixed the check_make bug but not the other bug with that 
the loach.lastrun-logs-directory isn't created before trying to write to it.

/Mikael

---

From: Andrew Dunstan <[email-masked]>
Date: Mon, 1 May 2017 14:56:18 -0400

> the loach.lastrun-logs-directory isn't created before trying to write

OK, coming up with a more comprehensive fix.

The obvious workaround for now is to create the directory and dont zap
it or its parents. You should only have to do it once (per branch)

cheers

andrew

---

From: Mikael Kjellström <[email-masked]>
Date: Mon, 1 May 2017 21:00:38 +0200

> OK, coming up with a more comprehensive fix.

Ok.

> it or its parents. You should only have to do it once (per branch)

Yes, I know.  That is what I have been doing so far.  But if you want to 
rerun a branch from scratch it's much easier to just do:

rm -rf buildroot/HEAD
or
rm -rf buildroot/REL9_6_STABLE

and then the buildfarm script should do the right thing and create all 
the directories that it needs.

/Mikael

---

From: Andrew Dunstan <[email-masked]>
Date: Mon, 1 May 2017 15:10:32 -0400

On 05/01/2017 03:00 PM, Mikael Kjellström wrote:
> Yes, I know.  That is what I have been doing so far.  But if you want
> to rerun a branch from scratch it's much easier to just do:
>
> rm -rf buildroot/HEAD
> or

Not sure I understand what "rerun a branch" from scratch means. If you
zap the branch directory you lose all its state. That's generally a bad
thing.

Anyway, this patch should fix it for all uses. It creates the directory
if it doesn't exist.

    diff --git a/PGBuild/Utils.pm b/PGBuild/Utils.pm
    index 91c1362..175eaa7 100644
    --- a/PGBuild/Utils.pm
    +++ b/PGBuild/Utils.pm
    @@ -14,6 +14,8 @@ See accompanying License file for license details
     use strict;
     use warnings;
     
    +use File::Path;
    +
     use Exporter   ();
     our (@ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
     
    @@ -30,8 +32,9 @@ use vars qw($VERSION); $VERSION = 'REL_4.19';
     sub run_log
     {
         my $command = shift;
    -    my $file=
    -     
    "$main::branch_root/$main::st_prefix$main::logdirname/lastcomand.log";
    +   my $filedir =
    "$main::branch_root/$main::st_prefix$main::logdirname";
    +   mkpath($filedir);
    +   my $file= "$filedir/lastcomand.log";
         unlink $file;
         system("$command > $file 2>&1");
         my @loglines;

cheers

andrew

---

From: Mikael Kjellström <[email-masked]>
Date: Mon, 1 May 2017 21:19:44 +0200

> zap the branch directory you lose all its state. That's generally a bad

I mean like the first time when you set up the buildfarm client / 
branch.  And it's not something I recomend doing all the time but some 
times when the buildfarm client doesn't clean up after it self and there 
are a lot of crap left it's "easier" to just zap everything and start 
over to get it running again.

Ok.

Thanks.  Will try it out.

/Mikael

---

From: Mikael Kjellström <[email-masked]>
Date: Mon, 1 May 2017 21:43:58 +0200

> Thanks.  Will try it out.

Just wanted to report that I've tried it and it works as expected.

Thanks for the really fast fixes.

/Mikael