Date: Thu, 14 Apr 2022 19:45:15 -0700
From: Noah Misch <noah@leadboat.com>
To: Andres Freund <andres@anarazel.de>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Peter Geoghegan <pg@bowt.ie>,
	David Rowley <dgrowleyml@gmail.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Intermittent buildfarm failures on wrasse
Message-ID: <20220415024515.GA862547@rfd.leadboat.com>
References: <1486022.1649952083@sss.pgh.pa.us>
 <20220414161850.ibqkevxcv6a7vdxg@alap3.anarazel.de>
 <1521388.1649953580@sss.pgh.pa.us>
 <1547912.1649971995@sss.pgh.pa.us>
 <CAH2-Wz=DHY2LV7=2aVd=qz6AbQhFK7KJRNL5ZwgB7dyno_gzdg@mail.gmail.com>
 <1553951.1649975018@sss.pgh.pa.us>
 <CAH2-WzngTjUk_65ESmrMf40WTqasrmWG0gpwnvcN+mXsoXT7qQ@mail.gmail.com>
 <CAH2-WznXXkkfE_y9ieaHkEQj31HeMAqJBtJPmDY+EpyVg1QywQ@mail.gmail.com>
 <1570894.1649986347@sss.pgh.pa.us>
 <20220415015249.kfsqxujrf2bn2wz6@alap3.anarazel.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220415015249.kfsqxujrf2bn2wz6@alap3.anarazel.de>
User-Agent: Mutt/1.5.24 (2015-08-30)
Archived-At: 
 <https://www.postgresql.org/message-id/20220415024515.GA862547%40rfd.leadboat.com>
Precedence: bulk

On Thu, Apr 14, 2022 at 06:52:49PM -0700, Andres Freund wrote:
> On 2022-04-14 21:32:27 -0400, Tom Lane wrote:
> > Peter Geoghegan <pg@bowt.ie> writes:
> > > Are you aware of Andres' commit 02fea8fd? That work prevented exactly
> > > the same set of symptoms (the same index-only scan create_index
> > > regressions),
> > 
> > Hm.  I'm starting to get the feeling that the real problem here is
> > we've "optimized" the system to the point where repeatable results
> > from VACUUM are impossible :-(
> 
> The synchronous_commit issue is an old one. It might actually be worth
> addressing it by flushing out pending async commits out instead. It just
> started to be noticeable when tenk1 load and vacuum were moved closer.
> 
> 
> What do you think about applying a polished version of what I posted in
> https://postgr.es/m/20220414164830.63rk5zqsvtqqk7qz%40alap3.anarazel.de
> ? That'd tell us a bit more about the horizon etc.

No objection.

> It's also interesting that it only happens in the installcheck cases,
> afaics, not the check ones. Although that might just be because there's
> more of them...

I suspect the failure is somehow impossible in "check".  Yesterday, I cranked
up the number of locales, so there are now a lot more installcheck.  Before
that, each farm run had one "check" and two "installcheck".  Those days saw
ten installcheck failures, zero check failures.

Like Tom, I'm failing to reproduce this outside the buildfarm client.  I wrote
a shell script to closely resemble the buildfarm installcheck sequence, but
it's lasted a dozen runs without failing.