Feedback-ID: id4a34324:Fastmail
Date: Fri, 21 Feb 2025 21:14:33 -0500
From: Andres Freund <andres@anarazel.de>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, 
	Ilia Evdokimov <ilya.evdokimov@tantorlabs.com>, Andrei Lepikhov <lepihov@gmail.com>, 
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: a very significant fraction of the buildfarm is now pink
Message-ID: <h5tjttxky4wlk4easqrcaqnvly74phwttvne6crk3lm4rdjbl5@uxu33v5mxjqx>
References: <e434311c-0f84-4c2b-a10d-ecb67f1ec19d@tantorlabs.com>
 <CA+TgmobfeePej0uNfbLy0v6jUrB3D9m1Kh+QQ40_-A9KQjkqUw@mail.gmail.com>
 <f428e80d-8607-4a9d-950c-be9600f24904@gmail.com>
 <01ec86c1-e04b-4cbe-b641-22cae8b87604@tantorlabs.com>
 <CA+TgmobVWYnYDj1HZ--iqeKx+n7zTgtYCar+YfZ0V6wiX-t0Mg@mail.gmail.com>
 <73e2440e-7d8b-4860-974d-931a93326d95@gmail.com>
 <CA+TgmoYp1KvG=7L7SjiRjJrXT8kEfH10kSGp54FQd03v19OwvA@mail.gmail.com>
 <CA+Tgmoaob_3aNF5S49WFoJs6Rb37rNnd5WH_e9nXAn_jtWWCLQ@mail.gmail.com>
 <3737950.1740182671@sss.pgh.pa.us>
 <CA+TgmoZB8frH_n6tYAdFSYWUwy1wbVOzM3zHfVkHUKCj0+rHzA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CA+TgmoZB8frH_n6tYAdFSYWUwy1wbVOzM3zHfVkHUKCj0+rHzA@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/h5tjttxky4wlk4easqrcaqnvly74phwttvne6crk3lm4rdjbl5%40uxu33v5mxjqx>
Precedence: bulk

Hi,

On 2025-02-21 20:09:24 -0500, Robert Haas wrote:
> In the case of my commit today, the failures are the result of a
> 2-line regression diff with no functional impact that neither CI nor
> any of the 11 reviewers noticed. That just shouldn't be the sort of
> thing that results in somebody having to work evenings and weekends.
> Perhaps if it DIDN'T result in a committer having to work evenings and
> weekends, it wouldn't have taken 16 years for us to do something about
> that problem.

I think you're right that it really depends on what the breakage is.

If just about all animals die early in the test, it's more important the
problem gets fixed more or the faulty commit(s) reverted. For one it prevents
detecting other problems, as later, independent failures are hidden. For
another it quite likely will affect other developers "at home", too.

I'd also say that breaking CI and BF is probably something I'd consider more
urgent, as that could indicate the commit was just generally less well tested.

If, as the case here, a few animals fail with a minor regression.diffs output,
one that doesn't indicate any major issue but just some corner case change in
output, it's ok to leave it for the next day or so (*).

Greetings,

Andres Freund


(*) With some adjustments based on circumstances. If the breakage is over a
weekend, it's perhaps a bit less urgent. However, if it's the Friday before a
feature freeze, it's obviously a differnt story.