public inbox for [email protected]help / color / mirror / Atom feed
Help with failures on HEAD 8+ messages / 3 participants [nested] [flat]
* Help with failures on HEAD @ 2023-02-08 14:31 Todd A. Cook <[email protected]> 0 siblings, 1 reply; 8+ messages in thread From: Todd A. Cook @ 2023-02-08 14:31 UTC (permalink / raw) To: [email protected] <[email protected]> Hi, Last Saturday morning, builds of HEAD on my animal mantid[1] spontaneously started failing; see [2] for an example. The log shown there is Missing checked out branch bf_HEAD: * [32mbf_HEAD[m bf_REL_11_STABLE[m bf_REL_12_STABLE[m bf_REL_13_STABLE[m bf_REL_14_STABLE[m bf_REL_15_STABLE[m master[m and the emails I get from cron contain multiple copies of the line Use of uninitialized value $commit in concatenation (.) or string at /u1/tac/build-farm-16/PGBuild/SCM.pm line 1381. I tried removing buildroot/HEAD, and “./run_build.pl --nosend --nostatus --verbose HEAD” succeeded. However, the next time cron ran the builds, all branches except REL_11_STABLE started failing the same way as HEAD. I then removed buildroot entirely, which fixed all the branches except HEAD. Mantid is using v16 of the build farm client and is running CentOS 7 with perl 5.16.3. Any advice on how to fix this would be greatly appreciated. -- todd [1] https://buildfarm.postgresql.org/cgi-bin/show_status.pl?member=mantid [2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mantid&dt=2023-02-04%2008%3A07%3A02 ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Help with failures on HEAD @ 2023-02-09 18:41 Tom Lane <[email protected]> parent: Todd A. Cook <[email protected]> 0 siblings, 1 reply; 8+ messages in thread From: Tom Lane @ 2023-02-09 18:41 UTC (permalink / raw) To: Todd A. Cook <[email protected]>; +Cc: [email protected] <[email protected]> "Todd A. Cook" <[email protected]> writes: > Hi, > Last Saturday morning, builds of HEAD on my animal mantid[1] spontaneously started failing; > see [2] for an example. The log shown there is > Missing checked out branch bf_HEAD: > * [32mbf_HEAD[m > bf_REL_11_STABLE[m > bf_REL_12_STABLE[m > bf_REL_13_STABLE[m > bf_REL_14_STABLE[m > bf_REL_15_STABLE[m > master[m In the past, I've been able to resolve issues similar to this one by flushing the animal's git repo (rm -rf pgmirror.git) and letting it pull that down fresh on the next run. Not clear if you included that when you "removed the buildroot"? regards, tom lane ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Help with failures on HEAD @ 2023-02-09 19:00 Todd A. Cook <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 2 replies; 8+ messages in thread From: Todd A. Cook @ 2023-02-09 19:00 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: [email protected] <[email protected]> On 2/9/23, 1:41 PM, "Tom Lane" <[email protected] <mailto:[email protected]>> wrote: "Todd A. Cook" <[email protected] <mailto:[email protected]>> writes: > Hi, > Last Saturday morning, builds of HEAD on my animal mantid[1] spontaneously started failing; > see [2] for an example. The log shown there is > Missing checked out branch bf_HEAD: > * [32mbf_HEAD[m > bf_REL_11_STABLE[m > bf_REL_12_STABLE[m > bf_REL_13_STABLE[m > bf_REL_14_STABLE[m > bf_REL_15_STABLE[m > master[m In the past, I've been able to resolve issues similar to this one by flushing the animal's git repo (rm -rf pgmirror.git) and letting it pull that down fresh on the next run. Not clear if you included that when you "removed the buildroot"? Yes, I did "rm -rf buildroot". After that, I verified (with "diff -r") that I had no changes vs. the distribution tarball other than having my build-farm.conf present. -- todd ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Help with failures on HEAD @ 2023-02-10 15:42 Andrew Dunstan <[email protected]> parent: Todd A. Cook <[email protected]> 1 sibling, 1 reply; 8+ messages in thread From: Andrew Dunstan @ 2023-02-10 15:42 UTC (permalink / raw) To: Todd A. Cook <[email protected]>; Tom Lane <[email protected]>; +Cc: [email protected] <[email protected]> On 2023-02-09 Th 14:00, Todd A. Cook wrote: > On 2/9/23, 1:41 PM, "Tom Lane" <[email protected] <mailto:[email protected]>> wrote: > "Todd A. Cook" <[email protected] <mailto:[email protected]>> writes: > > Hi, > > Last Saturday morning, builds of HEAD on my animal mantid[1] spontaneously started failing; > > see [2] for an example. The log shown there is > > > Missing checked out branch bf_HEAD: > > * [32mbf_HEAD[m > > bf_REL_11_STABLE[m > > bf_REL_12_STABLE[m > > bf_REL_13_STABLE[m > > bf_REL_14_STABLE[m > > bf_REL_15_STABLE[m > > master[m > > In the past, I've been able to resolve issues similar to this one > by flushing the animal's git repo (rm -rf pgmirror.git) and letting > it pull that down fresh on the next run. Not clear if you included > that when you "removed the buildroot"? > > Yes, I did "rm -rf buildroot". After that, I verified (with "diff -r") that I had no changes > vs. the distribution tarball other than having my build-farm.conf present. Don't do that. All the state is stored under the buildroot, and you'd be throwing that away. 999 times out of 1000 this should do the trick: rm -rf buildroot/[RH]*/pgsql The other 1 time out of 1000 you could also remove the mirror like Tom suggests as well. cheers andrew -- Andrew Dunstan EDB:https://www.enterprisedb.com ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Help with failures on HEAD @ 2023-02-10 20:02 Todd A. Cook <[email protected]> parent: Andrew Dunstan <[email protected]> 0 siblings, 1 reply; 8+ messages in thread From: Todd A. Cook @ 2023-02-10 20:02 UTC (permalink / raw) To: Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; +Cc: [email protected] <[email protected]> From: Andrew Dunstan <[email protected]> Date: Friday, February 10, 2023 at 10:42 AM To: "Todd A. Cook" <[email protected]>, Tom Lane <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: Help with failures on HEAD On 2023-02-09 Th 14:00, Todd A. Cook wrote: On 2/9/23, 1:41 PM, "Tom Lane" <[email protected]<mailto:[email protected]> <mailto:[email protected]><mailto:[email protected]>> wrote: "Todd A. Cook" <[email protected]<mailto:[email protected]> <mailto:[email protected]><mailto:[email protected]>> writes: > Hi, > Last Saturday morning, builds of HEAD on my animal mantid[1] spontaneously started failing; > see [2] for an example. The log shown there is > Missing checked out branch bf_HEAD: > * [32mbf_HEAD[m > bf_REL_11_STABLE[m > bf_REL_12_STABLE[m > bf_REL_13_STABLE[m > bf_REL_14_STABLE[m > bf_REL_15_STABLE[m > master[m In the past, I've been able to resolve issues similar to this one by flushing the animal's git repo (rm -rf pgmirror.git) and letting it pull that down fresh on the next run. Not clear if you included that when you "removed the buildroot"? Yes, I did "rm -rf buildroot". After that, I verified (with "diff -r") that I had no changes vs. the distribution tarball other than having my build-farm.conf present. Don't do that. All the state is stored under the buildroot, and you'd be throwing that away. 999 times out of 1000 this should do the trick: rm -rf buildroot/[RH]*/pgsql The other 1 time out of 1000 you could also remove the mirror like Tom suggests as well. Thanks, I’ll keep that in mind if this ever happens again. However, removing those things didn’t fix the problem. I suspect there might be something wrong with the Perl setup on that animal, even though it hasn’t changed recently. Unfortunately, my Perl skills are non-existent, and I have no clue how to proceed. -- todd ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Help with failures on HEAD @ 2023-02-10 21:48 Tom Lane <[email protected]> parent: Todd A. Cook <[email protected]> 0 siblings, 0 replies; 8+ messages in thread From: Tom Lane @ 2023-02-10 21:48 UTC (permalink / raw) To: Todd A. Cook <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; [email protected] <[email protected]> "Todd A. Cook" <[email protected]> writes: > Thanks, I’ll keep that in mind if this ever happens again. However, removing those things didn’t fix the problem. I suspect there > might be something wrong with the Perl setup on that animal, even though it hasn’t changed recently. Unfortunately, my Perl > skills are non-existent, and I have no clue how to proceed. Hmph. Wondering if something broke recently in the buildfarm code, I tried rm -rf'ing the whole buildroot locally ... but run_branches.pl got through fine. A couple of ideas: * Run through the setup process documented at https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto particularly "Check that required perl modules are present" ... maybe something went missing? * Diff your build-farm.conf against the current sample file and make sure there are no unintentional discrepancies. regards, tom lane ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Help with failures on HEAD @ 2023-02-14 16:18 Andrew Dunstan <[email protected]> parent: Todd A. Cook <[email protected]> 1 sibling, 1 reply; 8+ messages in thread From: Andrew Dunstan @ 2023-02-14 16:18 UTC (permalink / raw) To: Todd A. Cook <[email protected]>; Tom Lane <[email protected]>; +Cc: [email protected] <[email protected]> On 2023-02-09 Th 14:00, Todd A. Cook wrote: > On 2/9/23, 1:41 PM, "Tom Lane" <[email protected] <mailto:[email protected]>> wrote: > "Todd A. Cook" <[email protected] <mailto:[email protected]>> writes: > > Hi, > > Last Saturday morning, builds of HEAD on my animal mantid[1] spontaneously started failing; > > see [2] for an example. The log shown there is > > > Missing checked out branch bf_HEAD: > > * [32mbf_HEAD[m > > bf_REL_11_STABLE[m > > bf_REL_12_STABLE[m > > bf_REL_13_STABLE[m > > bf_REL_14_STABLE[m > > bf_REL_15_STABLE[m > > master[m > > In the past, I've been able to resolve issues similar to this one > by flushing the animal's git repo (rm -rf pgmirror.git) and letting > it pull that down fresh on the next run. Not clear if you included > that when you "removed the buildroot"? > > Yes, I did "rm -rf buildroot". After that, I verified (with "diff -r") that I had no changes > vs. the distribution tarball other than having my build-farm.conf present. It looks like you're getting some color output, probably because of some unwise git configuration or something that makes git think it's talking to a terminal. Please see if this patch works to fix things: diff --git a/PGBuild/SCM.pm b/PGBuild/SCM.pm index dcfd180..6dc1232 100644 --- a/PGBuild/SCM.pm +++ b/PGBuild/SCM.pm @@ -983,7 +983,7 @@ sub _setup_new_workdir # doesn't yet know about my @fetchlog = run_log('git fetch --prune'); - my @branches = `git branch`; + my @branches = `git branch --no-color`; chomp @branches; my @colog; if (grep { /\bbf_$branch\b/ } @branches) cheers andrew -- Andrew Dunstan EDB:https://www.enterprisedb.com ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Help with failures on HEAD @ 2023-02-14 19:30 Todd A. Cook <[email protected]> parent: Andrew Dunstan <[email protected]> 0 siblings, 0 replies; 8+ messages in thread From: Todd A. Cook @ 2023-02-14 19:30 UTC (permalink / raw) To: Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; +Cc: [email protected] <[email protected]> From: Andrew Dunstan <[email protected]> Date: Tuesday, February 14, 2023 at 11:18 AM To: "Todd A. Cook" <[email protected]>, Tom Lane <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: Help with failures on HEAD diff --git a/PGBuild/SCM.pm b/PGBuild/SCM.pm index dcfd180..6dc1232 100644 --- a/PGBuild/SCM.pm +++ b/PGBuild/SCM.pm @@ -983,7 +983,7 @@ sub _setup_new_workdir # doesn't yet know about my @fetchlog = run_log('git fetch --prune'); - my @branches = `git branch`; + my @branches = `git branch --no-color`; chomp @branches; my @colog; if (grep { /\bbf_$branch\b/ } @branches) That worked. Thanks! -- todd ^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2023-02-14 19:30 UTC | newest] Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-02-08 14:31 Help with failures on HEAD Todd A. Cook <[email protected]> 2023-02-09 18:41 ` Tom Lane <[email protected]> 2023-02-09 19:00 ` Todd A. Cook <[email protected]> 2023-02-10 15:42 ` Andrew Dunstan <[email protected]> 2023-02-10 20:02 ` Todd A. Cook <[email protected]> 2023-02-10 21:48 ` Tom Lane <[email protected]> 2023-02-14 16:18 ` Andrew Dunstan <[email protected]> 2023-02-14 19:30 ` Todd A. Cook <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox