Subject: Help with failures on HEAD

From: "Todd A. Cook" <[email-masked]>
Date: Wed, 8 Feb 2023 14:31:35 +0000

Hi,

Last Saturday morning, builds of HEAD on my animal mantid[1] spontaneously started failing;
see [2] for an example.  The log shown there is

Missing checked out branch bf_HEAD:
* [32mbf_HEAD[m
  bf_REL_11_STABLE[m
  bf_REL_12_STABLE[m
  bf_REL_13_STABLE[m
  bf_REL_14_STABLE[m
  bf_REL_15_STABLE[m
  master[m

and the emails I get from cron contain multiple copies of the line

Use of uninitialized value $commit in concatenation (.) or string at /u1/tac/build-farm-16/PGBuild/SCM.pm line 1381.

I tried removing buildroot/HEAD, and “./run_build.pl --nosend --nostatus --verbose HEAD” succeeded.
However, the next time cron ran the builds, all branches except REL_11_STABLE started failing the
same way as HEAD.  I then removed buildroot entirely, which fixed all the branches except HEAD.

Mantid is using v16 of the build farm client and is running CentOS 7 with perl 5.16.3.

Any advice on how to fix this would be greatly appreciated.

-- todd

[1] https://buildfarm.postgresql.org/cgi-bin/show_status.pl?member=mantid
[2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mantid&dt=2023-02-04%2008%3A07%3A02

---

From: Tom Lane <[email-masked]>
Date: Thu, 09 Feb 2023 13:41:34 -0500

> Missing checked out branch bf_HEAD:

In the past, I've been able to resolve issues similar to this one
by flushing the animal's git repo (rm -rf pgmirror.git) and letting
it pull that down fresh on the next run.  Not clear if you included
that when you "removed the buildroot"?

			regards, tom lane

---

From: "Todd A. Cook" <[email-masked]>
Date: Thu, 9 Feb 2023 19:00:41 +0000

On 2/9/23, 1:41 PM, "Tom Lane" <[email-masked] <mailto:[email-masked]>> wrote:

Yes, I did "rm -rf buildroot".  After that, I verified (with "diff -r") that I had no changes
vs. the distribution tarball other than having my build-farm.conf present.

-- todd

---

From: Andrew Dunstan <[email-masked]>
Date: Fri, 10 Feb 2023 10:42:31 -0500

> vs. the distribution tarball other than having my build-farm.conf present.

Don't do that. All the state is stored under the buildroot, and you'd be 
throwing that away. 999 times out of 1000 this should do the trick:

rm -rf buildroot/[RH]*/pgsql

The other 1 time out of 1000 you could also remove the mirror like Tom 
suggests as well.

cheers

andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

---

From: "Todd A. Cook" <[email-masked]>
Date: Fri, 10 Feb 2023 20:02:44 +0000

From: Andrew Dunstan <[email-masked]>
Date: Friday, February 10, 2023 at 10:42 AM
To: "Todd A. Cook" <[email-masked]>, Tom Lane <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: Help with failures on HEAD

    On 2/9/23, 1:41 PM, "Tom Lane" <[email-masked]<mailto:[email-masked]> <mailto:[email-masked]><mailto:[email-masked]>> wrote:

Don't do that. All the state is stored under the buildroot, and you'd be throwing that away. 999 times out of 1000 this should do the trick:

The other 1 time out of 1000 you could also remove the mirror like Tom suggests as well.

Thanks, I’ll keep that in mind if this ever happens again.  However, removing those things didn’t fix the problem.  I suspect there

might be something wrong with the Perl setup on that animal, even though it hasn’t changed recently.  Unfortunately, my Perl

skills are non-existent, and I have no clue how to proceed.

-- todd

---

From: Tom Lane <[email-masked]>
Date: Fri, 10 Feb 2023 16:48:35 -0500

> skills are non-existent, and I have no clue how to proceed.

Hmph.  Wondering if something broke recently in the buildfarm code,
I tried rm -rf'ing the whole buildroot locally ... but run_branches.pl
got through fine.

A couple of ideas:

* Run through the setup process documented at

https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto

particularly "Check that required perl modules are present" ...
maybe something went missing?

* Diff your build-farm.conf against the current sample file
and make sure there are no unintentional discrepancies.

			regards, tom lane

---

From: Andrew Dunstan <[email-masked]>
Date: Tue, 14 Feb 2023 11:18:48 -0500

> vs. the distribution tarball other than having my build-farm.conf present.

It looks like you're getting some color output, probably because of some 
unwise git configuration or something that makes git think it's talking 
to a terminal. Please see if this patch works to fix things:

diff --git a/PGBuild/SCM.pm b/PGBuild/SCM.pm
index dcfd180..6dc1232 100644
--- a/PGBuild/SCM.pm
+++ b/PGBuild/SCM.pm
@@ -983,7 +983,7 @@ sub _setup_new_workdir
     # doesn't yet know about
     my @fetchlog = run_log('git fetch --prune');

-   my @branches = `git branch`;
+   my @branches = `git branch --no-color`;
     chomp @branches;
     my @colog;
     if (grep { /\bbf_$branch\b/ } @branches)

cheers

andrew

---

From: "Todd A. Cook" <[email-masked]>
Date: Tue, 14 Feb 2023 19:30:15 +0000

Date: Tuesday, February 14, 2023 at 11:18 AM

That worked.  Thanks!

-- todd