Date: Fri, 3 Apr 2020 23:58:30 +0200 (CEST)
From: Fabien COELHO <coelho@cri.ensmp.fr>
To: Robert Haas <robertmhaas@gmail.com>
cc: Alvaro Herrera <alvherre@2ndquadrant.com>,
    David Steele <david@pgmasters.net>, Andres Freund <andres@anarazel.de>,
    Noah Misch <noah@leadboat.com>, Stephen Frost <sfrost@snowman.net>,
    Amit Kapila <amit.kapila16@gmail.com>,
    Suraj Kharage <suraj.kharage@enterprisedb.com>,
    tushar <tushar.ahuja@enterprisedb.com>,
    Rajkumar Raghuwanshi <rajkumar.raghuwanshi@enterprisedb.com>,
    Rushabh Lathia <rushabh.lathia@gmail.com>,
    Tels <nospam-pg-abuse@bloodgate.com>,
    Andrew Dunstan <andrew.dunstan@2ndquadrant.com>,
    PostgreSQL Hackers <pgsql-hackers@postgresql.org>,
    Jeevan Chalke <jeevan.chalke@enterprisedb.com>,
    vignesh C <vignesh21@gmail.com>, Thomas Munro <thomas.munro@gmail.com>
Subject: Re: backup manifests and contemporaneous buildfarm failures
In-Reply-To: 
 <CA+TgmoZ4bAbjd_Am=UiMpBrgPEO=7YarqvaAMrQYKP1GkkOaRg@mail.gmail.com>
Message-ID: <alpine.DEB.2.21.2004032346550.16227@pseudo>
References: 
 <CA+TgmoazJ5wh+H77kjVfMnDh6WLh=oJodrwJ21SACSq5fojwPg@mail.gmail.com>
 <20200403205412.GA8279@alvherre.pgsql>
 <CA+TgmoaRbm-L1C2YkDVPx1TGN9ufxWjnaweTwKE=TQC-YOD=rg@mail.gmail.com>
 <CA+TgmoZ4bAbjd_Am=UiMpBrgPEO=7YarqvaAMrQYKP1GkkOaRg@mail.gmail.com>
User-Agent: Alpine 2.21 (DEB 202 2017-01-01)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII; format=flowed
Precedence: bulk


Hello Robert,

> Done now. Meanwhile, two more machines have reported the mysterious message:
>
> sh: ./configure: not found
>
> ...that first appeared on spurfowl a few hours ago. The other two
> machines are eelpout and elver, both of which list Thomas Munro as a
> maintainer. spurfowl lists Stephen Frost. Thomas, Stephen, can one of
> you check and see what's going on? spurfowl has failed this way four
> times now, and eelpout and elver have each failed the last two runs,
> but since there's no helpful information in the logs, it's hard to
> guess what went wrong.
>
> I'm sort of afraid that something in the new TAP tests accidentally
> removed way too many files during the cleanup phase - e.g. it decided
> the temporary directory was / and removed every file it could access,
> or something like that. It doesn't do that here, or I, uh, would've
> noticed by now. But sometimes strange things happen on other people's
> machines. Hopefully one of those strange things is not that my test
> code is single-handedly destroying the entire buildfarm, but it's
> possible.

seawasp just failed the same way. Good news, I can see "configure" under 
"HEAD/pgsql".

The only strange thing under buildroot I found is:

HEAD/pgsql.build/src/bin/pg_validatebackup/tmp_check/t_003_corruption_master_data/backup/open_directory_fails/pg_subtrans/

this last directory perms are d--------- which seems to break cleanup.

It may be a left over from a previous run which failed (possibly 21dc488 
?). I cannot see how this would be related to configure, though. Maybe 
something else fails silently and the message is about a consequence of 
the prior silent failure.

I commented out the cron job and will try to look into it on tomorrow if 
the status has not changed by then.

-- 
Fabien.