Message-ID: <7762aec1-971d-4121-83df-57bb5836ef10@dunslane.net>
Date: Thu, 12 Dec 2024 10:43:58 -0500
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: strange git problems on turaco
To: Noah Misch <noah@leadboat.com>, Tomas Vondra <tomas@vondra.me>
Cc: buildfarm-members@lists.postgresql.org
References: <6a705172-5b28-4023-a40e-fb7805c717c4@vondra.me>
 <20241202034623.39@rfd.leadboat.com>
From: Andrew Dunstan <andrew@dunslane.net>
Content-Language: en-US
Autocrypt: addr=andrew@dunslane.net; keydata=
 xsBNBE7KWFkBCAClridxur2AIc7eW2AR7izbfp3EnNefie2HbLF0izW5Ik5UjX2HBXBx4syI
 gY6b0ugohXrr274+baoAlvSbq6cAoQuEVrk5IZFzt20b1Xkx65FwGSEj526yiKLocqkJceSq
 Xr9xcA5SGY+FZv441chh5SU92v4q6z+6LPpoHOh97ptAVXZYNTtU0LevyvD5lja0TzbvJm6C
 eFXitJfnm1pLEr0DGJCR/iUOl/N62Kh4855zZC7NHIjQHPOvV5Stz/l5ilDhvGVk+xkXFPys
 SjZoUr1rXhYLpiyi5sR0X9FHXT0KnGuz1F5ERO7ZTLSSQ6fJwPj6gOk9K+vvoKvoeql5ABEB
 AAHNJEFuZHJldyBEdW5zdGFuIDxhbmRyZXdAZHVuc2xhbmUubmV0PsLAmwQTAQgARQIbAwIX
 gAIZAQULCQgHAgMiAgEGFQoJCAsCBBYCAwECHgcWIQTkPlhGHfx8v0RpFaWZ+n/LWfw7gQUC
 ZFlxxwUJGVGAbgAKCRCZ+n/LWfw7gXikB/9ZdcUy6CTBFIIuL/bVsc1eLEW/gJBjJBF6HxNY
 xgEkAgXAp4Lg4A5U+QB9GouFr7+GYxF0BU4hzoGhNPUWltxnHdMWP8nC/38LAqgMi8L/bbsm
 HW5YPBdWYaAZAPJQVfOAgjTbRUb26KSprpyrrJKW0ZmrZfjhNPcQ72jpWzoPLQqx2X6B0fru
 1jq+cBh8lb6r1mJTim1T3JIn+F/v5VpdQS+EL8xqsHkfzKjIPsW3CIXpkypSk6saA55Rkkbl
 26AW8ftPVB0Q6Lnn6FLt9CP0MGNixBQ55yq8r1K+nCBvCCjvQjM8RDm0UUum0WNl+ifQgTLO
 E8TWEnwVtkBf+3QWzsBNBE7KWFkBCADRnOM0FCzsYW6jtncg+dWIagjUZpvaClmqn/sJluLa
 Q3v1VXMQJzYs3eC1gh386W+XBwLRpDj3jzH81lX+p73Re3d3oJW7X+ffsxuzu5ZVdMUkqBYo
 nkAbKxr6gyJ12F/+JkUVzLcoTN+d/7YsQvUVi7NaKH8mJgjz112O4fUe3p9wfAaFa0RXHc5S
 GPzRTYRRlv/XZBIho4J2tkZOnteZJZ+GbxQVlINt6fd8P6al3MWOvpP/ExJPguEfjOsO6Njy
 xjo3WfpD4lHMOR/Oc3/8mScEF84rF2jXbsFgelWnbPWAvXY+pD0dXOFRkagGmC/viwBDqq5b
 5tk76kKmUbZxABEBAAHCwHwEGAEIACYCGwwWIQTkPlhGHfx8v0RpFaWZ+n/LWfw7gQUCZFlx
 5wUJGVGAjgAKCRCZ+n/LWfw7gf+iB/4g8CPY5jihf5r/8EsoIGe2H+dpVmpPF8YGBzTIvCz/
 fQoOq8AX/pE76QEuFnFZWfjw+wgBXgCVmkox2Eflkk6z4ND3pcwGZ6CfCxTQCDk/dij+2DQ4
 6bmDCy/sBgcbz9mTpoLC11HLoPae6YN9nBNQRZDcEFEu54OaVOqlIdbA6m+POIBCXZdHOFc0
 WoDTgxHRzC1jgQNidyd6tKqcsVJs0dzF0oKTmFFmUAqTdJO12LBuNA1rlqrR3EtpYk8B/wtS
 5dIMD7Q8hwQpL+4C6GNpb6ZKnPkLi47pDOLhz2qBrqN+rqUEsT3YnExYpzj5yOBi+FlmV1Hw
 49QYe1sn2ZPs
In-Reply-To: <20241202034623.39@rfd.leadboat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/7762aec1-971d-4121-83df-57bb5836ef10%40dunslane.net>
Precedence: bulk


On 2024-12-01 Su 10:46 PM, Noah Misch wrote:
> On Mon, Dec 02, 2024 at 02:20:35AM +0100, Tomas Vondra wrote:
>> turaco seems to be having some strange git issues - some of the
>> buildfarm runs fail like this:
>>
>>
>> turaco:REL_16_STABLE [22:41:11] OK
>> Sun Dec  1 22:41:27 2024: buildfarm run for turaco:REL_17_STABLE starting
>> turaco:REL_17_STABLE [22:41:27] checking out source ...
>> Missing checked out branch bf_REL_17_STABLE:
>> fatal: not a git repository (or any parent up to mount point /mnt)
>> Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
>> turaco:REL_17_STABLE [22:41:32] failed at stage pgsql-Git
>> Sun Dec  1 22:41:33 2024: buildfarm run for turaco:HEAD starting
>> turaco:HEAD          [22:41:33] checking out source ...
>>
>>
>> I initially suspected this might be due to aging storage (SD card on
>> rpi), but I replaced that, and there's nothing strange in dmesg. Also,
>> other branches seem to be working fine ...
>>
>> Any ideas what could be causing this?
> I had this happen ~9 times on the host of my AIX buildfarm members.  Example:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mandrill&dt=2024-07-10%2019%3A51%3A28
>
> I figured it was some system problem, so I didn't root-cause it.  I carry the
> following workaround in my fork of the buildfarm client code.  The unknown
> problem caused failure reports and work stoppage ~4 times before I installed
> this workaround, then logs show the workaround prevented damage 5 times.  The
> last "removed intruder .git" log message appeared on 2024-07-23.  There was no
> kernel reboot, and logs don't point to buildfarm client processes getting
> involuntary termination, either.
>
> diff --git a/PGBuild/SCM.pm b/PGBuild/SCM.pm
> index dcfd180..2cd610a 100644
> --- a/PGBuild/SCM.pm
> +++ b/PGBuild/SCM.pm
> @@ -1059,9 +1059,19 @@ sub _update_target
>   	my @gitlog;
>   
>   	# If a run crashed during copy_source(), repair.
> -	if (-d "./git-save" && !-d "$target/.git")
> +	if (-d "./git-save")
>   	{
> +		# As of 2024-07-13, the following has happened about four times in the
> +		# last month, to different gcc111 animals.  Despite no known crash,
> +		# there's a git-save directory containing the proper git repo, and
> +		# there's a bogus .git missing most content.  Remove the bogus one.
> +		# This is deeply hacky, but it beats buildfarm report noise and manual
> +		# intervention.
> +		if (rmtree("$target/.git") > 0) {
> +			print "removed intruder .git\n" if $verbose;
> +		}
>   		move "./git-save", "$target/.git";
> +		print "restored git-save\n" if $verbose;
>   	}
>   
>   	chdir $target;
>
>

[catching up a huge email backlog]

That's kinda weird. The .git directory doesn't get moved at all if you 
have vpath turned on or you're building with meson (which always does 
vpath). So that's one possible workaround.

I guess I should put something like this in the next release ... will go 
and do that.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com