Subject: strange git problems on turaco

From: Tomas Vondra <[email-masked]>
Date: Mon, 2 Dec 2024 02:20:35 +0100

Hi,

turaco seems to be having some strange git issues - some of the
buildfarm runs fail like this:

turaco:REL_16_STABLE [22:41:11] OK
Sun Dec  1 22:41:27 2024: buildfarm run for turaco:REL_17_STABLE starting
turaco:REL_17_STABLE [22:41:27] checking out source ...
Missing checked out branch bf_REL_17_STABLE:
fatal: not a git repository (or any parent up to mount point /mnt)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
turaco:REL_17_STABLE [22:41:32] failed at stage pgsql-Git
Sun Dec  1 22:41:33 2024: buildfarm run for turaco:HEAD starting
turaco:HEAD          [22:41:33] checking out source ...

I initially suspected this might be due to aging storage (SD card on
rpi), but I replaced that, and there's nothing strange in dmesg. Also,
other branches seem to be working fine ...

Any ideas what could be causing this?

regards

-- 
Tomas Vondra

---

From: Tom Lane <[email-masked]>
Date: Sun, 01 Dec 2024 20:56:30 -0500

> buildfarm runs fail like this:

Have you tried rm -rf'ing its git repo and letting the script
check that out from scratch?  The fact that it's just the 17
branch has a whiff of repo corruption.

Andrew might correct me, but I think you have to remove
both the pgmirror.git directory and the per-branch pgsql
subdirectories to be clean.  Don't remove the various
<animal>.* status files.

			regards, tom lane

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 2 Dec 2024 03:23:10 +0100

> branch has a whiff of repo corruption.

I actually nuked the whole buildroot, because the old SD card was having
issues and I wasn't sure what might be corrupted. So it's all fresh. But
I also first ran

   ./run_branches.pl --run-all --nosend --nostatus

just to make sure everything works fine, and it did ...

> subdirectories to be clean.  Don't remove the various

Done. Let's see how quickly it breaks again.

regards

-- 
Tomas Vondra

---

From: Noah Misch <[email-masked]>
Date: Sun, 1 Dec 2024 19:46:23 -0800

> Any ideas what could be causing this?

I had this happen ~9 times on the host of my AIX buildfarm members.  Example:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mandrill&dt=2024-07-10%2019%3A51%3A28

I figured it was some system problem, so I didn't root-cause it.  I carry the
following workaround in my fork of the buildfarm client code.  The unknown
problem caused failure reports and work stoppage ~4 times before I installed
this workaround, then logs show the workaround prevented damage 5 times.  The
last "removed intruder .git" log message appeared on 2024-07-23.  There was no
kernel reboot, and logs don't point to buildfarm client processes getting
involuntary termination, either.

diff --git a/PGBuild/SCM.pm b/PGBuild/SCM.pm
index dcfd180..2cd610a 100644
--- a/PGBuild/SCM.pm
+++ b/PGBuild/SCM.pm
@@ -1059,9 +1059,19 @@ sub _update_target
 	my @gitlog;
 
 	# If a run crashed during copy_source(), repair.
-	if (-d "./git-save" && !-d "$target/.git")
+	if (-d "./git-save")
 	{
+		# As of 2024-07-13, the following has happened about four times in the
+		# last month, to different gcc111 animals.  Despite no known crash,
+		# there's a git-save directory containing the proper git repo, and
+		# there's a bogus .git missing most content.  Remove the bogus one.
+		# This is deeply hacky, but it beats buildfarm report noise and manual
+		# intervention.
+		if (rmtree("$target/.git") > 0) {
+			print "removed intruder .git\n" if $verbose;
+		}
 		move "./git-save", "$target/.git";
+		print "restored git-save\n" if $verbose;
 	}
 
 	chdir $target;

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 2 Dec 2024 14:51:29 +0100

> involuntary termination, either.

Thanks. I suspect some system issue too, but I didn't want to blame the
system without some kind of proof. I applied your patch, let's see if
that helped after a couple runs.

regards

-- 
Tomas Vondra

---

From: Andrew Dunstan <[email-masked]>
Date: Thu, 12 Dec 2024 10:43:58 -0500

> +		print "restored git-save\n" if $verbose;

[catching up a huge email backlog]

That's kinda weird. The .git directory doesn't get moved at all if you 
have vpath turned on or you're building with meson (which always does 
vpath). So that's one possible workaround.

I guess I should put something like this in the next release ... will go 
and do that.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

---

From: Andrew Dunstan <[email-masked]>
Date: Thu, 12 Dec 2024 10:46:49 -0500

> subdirectories to be clean.  Don't remove the various

In most cases you only have to remove the per-branch pgsql directory. 
I've only very occasionally seem corruption of the mirror.

cheers

andrew