MIME-Version: 1.0
References: <e434311c-0f84-4c2b-a10d-ecb67f1ec19d@tantorlabs.com>
 <CA+TgmobfeePej0uNfbLy0v6jUrB3D9m1Kh+QQ40_-A9KQjkqUw@mail.gmail.com>
 <f428e80d-8607-4a9d-950c-be9600f24904@gmail.com> <01ec86c1-e04b-4cbe-b641-22cae8b87604@tantorlabs.com>
 <CA+TgmobVWYnYDj1HZ--iqeKx+n7zTgtYCar+YfZ0V6wiX-t0Mg@mail.gmail.com>
 <73e2440e-7d8b-4860-974d-931a93326d95@gmail.com> <CA+TgmoYp1KvG=7L7SjiRjJrXT8kEfH10kSGp54FQd03v19OwvA@mail.gmail.com>
 <CA+Tgmoaob_3aNF5S49WFoJs6Rb37rNnd5WH_e9nXAn_jtWWCLQ@mail.gmail.com>
 <3737950.1740182671@sss.pgh.pa.us> <CA+TgmoZB8frH_n6tYAdFSYWUwy1wbVOzM3zHfVkHUKCj0+rHzA@mail.gmail.com>
 <h5tjttxky4wlk4easqrcaqnvly74phwttvne6crk3lm4rdjbl5@uxu33v5mxjqx> <CA+Tgmobec4-XiTZ0_VNf7G1znq3eJhjDL0c=3E1qkOXy4xtXxA@mail.gmail.com>
In-Reply-To: <CA+Tgmobec4-XiTZ0_VNf7G1znq3eJhjDL0c=3E1qkOXy4xtXxA@mail.gmail.com>
From: Jelte Fennema-Nio <postgres@jeltef.nl>
Date: Sun, 23 Feb 2025 12:30:16 +0100
Message-ID: <CAGECzQRys8t8ig7eN26rbvop+zgw06oP11L5HpYuUtA=6j_WZA@mail.gmail.com>
Subject: Re: a very significant fraction of the buildfarm is now pink
To: Robert Haas <robertmhaas@gmail.com>
Cc: Andres Freund <andres@anarazel.de>, Tom Lane <tgl@sss.pgh.pa.us>, 
	Ilia Evdokimov <ilya.evdokimov@tantorlabs.com>, Andrei Lepikhov <lepihov@gmail.com>, 
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://www.postgresql.org/message-id/CAGECzQRys8t8ig7eN26rbvop%2Bzgw06oP11L5HpYuUtA%3D6j_WZA%40mail.gmail.com>
Precedence: bulk

This was very painful to read. +1 on making changes. Both for a
culture change to require less urgency over the weekend if it's a
minor failure, and probably also a tooling change to make this less of
a problem in general.

On Sat, 22 Feb 2025 at 04:38, Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Fri, Feb 21, 2025 at 9:14=E2=80=AFPM Andres Freund <andres@anarazel.de=
> wrote:
> > I'd also say that breaking CI and BF is probably something I'd consider=
 more
> > urgent, as that could indicate the commit was just generally less well =
tested.
>
> This is possible, but I think we oversell it by quite a lot. It's
> quite possible for someone to put hundreds of hours into a patch and
> forget to check CI before pushing;
...snip...
> Twenty years ago, post-commit testing
> was probably the best you could hope for, but today it isn't.

100% agreed. As someone also working on other open source projects,
this reads incredibly foreign. This is a problem that simply cannot
happen in most projects I've worked on: GitHub will simply not allow
you to push to master/main if CI fails (at least not without an
explicit override). Postgres is the only project I work on where all
these things only run post-push to master.

To be clear, I'm not saying that we should be using GitHub for this,
although I personally think that would be a good idea given our CI is
triggered by pushes to GitHub. But it at least seems like one thing
that we need is a process change to make it less likely that people
accidentally push broken changes to master. e.g. We create a staging
repo (which could be on GitHub) and each committer has their own
branch (or branch prefix). Then when a committer pushes there, CI is
run and probably also some significant part of the build farm (and
maybe a committer could even opt in to running the full build farm).
If they pass, then the branch is automatically rebased on top of
master and pushed to the production repo its master branch.