MIME-Version: 1.0
References: <CA+TgmoZv8DvmeQHQAX+rkn68id2N+DoUhFZB36d-_w1tyAeqeA@mail.gmail.com>
 <70bc1af0-6910-4f71-84ba-fa02095fe9ba@tantorlabs.com> <CAFY6G8dgp494Xi2+VUUyM2CvuSQ+3nnzWi44gURWbCHjEWW89g@mail.gmail.com>
 <fc28108a-8e7e-4f7d-b8f4-0bcc1581650d@tantorlabs.com> <CA+Tgmobgkh5CW=4Xpejw9nR6Onv8SZh0nnr+=HmNAFyyHZb0eg@mail.gmail.com>
 <3212947.1739306803@sss.pgh.pa.us> <eb73d4e1-4235-41b5-90a8-b0196550f62b@gmail.com>
 <CA+TgmobT9gPQLtw143oGnKVNtaOrTDOy+ZbLfe18MVZ9POjaWg@mail.gmail.com>
 <3365160.1739385615@sss.pgh.pa.us> <0e4c813a-18eb-41e2-8edb-89b78d43c82d@gmail.com>
 <CA+TgmoYzawdq40C5cN==LRPBvQChZ7t5XxH6wv+zMxnyM7LjBQ@mail.gmail.com>
 <e434311c-0f84-4c2b-a10d-ecb67f1ec19d@tantorlabs.com> <CA+TgmobfeePej0uNfbLy0v6jUrB3D9m1Kh+QQ40_-A9KQjkqUw@mail.gmail.com>
 <f428e80d-8607-4a9d-950c-be9600f24904@gmail.com> <01ec86c1-e04b-4cbe-b641-22cae8b87604@tantorlabs.com>
 <CA+TgmobVWYnYDj1HZ--iqeKx+n7zTgtYCar+YfZ0V6wiX-t0Mg@mail.gmail.com>
 <73e2440e-7d8b-4860-974d-931a93326d95@gmail.com> <CA+TgmoYp1KvG=7L7SjiRjJrXT8kEfH10kSGp54FQd03v19OwvA@mail.gmail.com>
 <CA+Tgmoaob_3aNF5S49WFoJs6Rb37rNnd5WH_e9nXAn_jtWWCLQ@mail.gmail.com> <3737950.1740182671@sss.pgh.pa.us>
In-Reply-To: <3737950.1740182671@sss.pgh.pa.us>
From: Robert Haas <robertmhaas@gmail.com>
Date: Fri, 21 Feb 2025 20:09:24 -0500
Message-ID: <CA+TgmoZB8frH_n6tYAdFSYWUwy1wbVOzM3zHfVkHUKCj0+rHzA@mail.gmail.com>
Subject: a very significant fraction of the buildfarm is now pink
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Ilia Evdokimov <ilya.evdokimov@tantorlabs.com>, Andrei Lepikhov <lepihov@gmail.com>, 
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://www.postgresql.org/message-id/CA%2BTgmoZB8frH_n6tYAdFSYWUwy1wbVOzM3zHfVkHUKCj0%2BrHzA%40mail.gmail.com>
Precedence: bulk

On Fri, Feb 21, 2025 at 7:04=E2=80=AFPM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> A very significant fraction of the buildfarm is now pink.
> If you don't have a fix pretty nearly ready, please revert.

When we're going to do a release, you want no commits for at least 24
hours before the release so that we can make sure the buildfarm is
clean. But when I commit something and the buildfarm fails, you want
it reverted within a handful of hours before I can even be sure of
having seen all the failures the commit caused, let alone had time to
think about what the best fix might be. It doesn't make sense to me
that we need 24 hours to be sure that the buildfarm is passing, but in
3 hours I'm supposed to see all the failures -- including the ones
that only happen on buildfarm animals that run once a day, I guess? --
and analyze them -- and decide on a fix -- naturally without any sort
of discussion because there's no time for that -- and code the fix --
and push it. I have a really hard time seeing how that's a reasonable
expectation.

I understand that the buildfarm can't be red all the time or nobody
can distinguish the problems they caused from preexisting ones. At the
same time, it seems completely unreasonable for us to say that, on the
one hand, the buildfarm has to be green at absolutely all times, and
on the other time, buildfarm owners are not required to provide any
sort of resources to test things before they are committed. IMHO, one
of those policies absolutely has to change. The current situation is
way too stressful for committers and it's burning people out and
making them unwilling to commit things -- or if they do commit things,
then they end up insta-reverting them, committing them again later,
maybe insta-reverting them a second time because they didn't actually
find all the problems the first time, and then maybe even round three,
four, or five. The commit log ends up with a bunch of garbage from the
repeated commits and reverts, and if it goes on long enough,
eventually somebody shows up to say "wow, this patch seems to be in
terrible shape, maybe it shouldn't ever be committed again" right when
the committer's stress level is already going through the ceiling. And
sometimes that is justified, but sometimes it isn't.

In the case of my commit today, the failures are the result of a
2-line regression diff with no functional impact that neither CI nor
any of the 11 reviewers noticed. That just shouldn't be the sort of
thing that results in somebody having to work evenings and weekends.
Perhaps if it DIDN'T result in a committer having to work evenings and
weekends, it wouldn't have taken 16 years for us to do something about
that problem.

--=20
Robert Haas
EDB: http://www.enterprisedb.com