Date: Thu, 14 Apr 2022 19:50:19 -0700
From: Noah Misch <noah@leadboat.com>
To: Tom Lane <tgl@sss.pgh.pa.us>, Andres Freund <andres@anarazel.de>
Cc: Peter Geoghegan <pg@bowt.ie>, David Rowley <dgrowleyml@gmail.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Intermittent buildfarm failures on wrasse
Message-ID: <20220415025019.GA863781@rfd.leadboat.com>
References: <20220414161850.ibqkevxcv6a7vdxg@alap3.anarazel.de>
 <1521388.1649953580@sss.pgh.pa.us>
 <1547912.1649971995@sss.pgh.pa.us>
 <CAH2-Wz=DHY2LV7=2aVd=qz6AbQhFK7KJRNL5ZwgB7dyno_gzdg@mail.gmail.com>
 <1553951.1649975018@sss.pgh.pa.us>
 <CAH2-WzngTjUk_65ESmrMf40WTqasrmWG0gpwnvcN+mXsoXT7qQ@mail.gmail.com>
 <CAH2-WznXXkkfE_y9ieaHkEQj31HeMAqJBtJPmDY+EpyVg1QywQ@mail.gmail.com>
 <1570894.1649986347@sss.pgh.pa.us>
 <20220415015249.kfsqxujrf2bn2wz6@alap3.anarazel.de>
 <20220415024515.GA862547@rfd.leadboat.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="QTprm0S8XgL7H0Dt"
Content-Disposition: inline
In-Reply-To: <20220415024515.GA862547@rfd.leadboat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Archived-At: 
 <https://www.postgresql.org/message-id/20220415025019.GA863781%40rfd.leadboat.com>
Precedence: bulk


--QTprm0S8XgL7H0Dt
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Thu, Apr 14, 2022 at 07:45:15PM -0700, Noah Misch wrote:
> On Thu, Apr 14, 2022 at 06:52:49PM -0700, Andres Freund wrote:
> > On 2022-04-14 21:32:27 -0400, Tom Lane wrote:
> > > Peter Geoghegan <pg@bowt.ie> writes:
> > > > Are you aware of Andres' commit 02fea8fd? That work prevented exactly
> > > > the same set of symptoms (the same index-only scan create_index
> > > > regressions),
> > > 
> > > Hm.  I'm starting to get the feeling that the real problem here is
> > > we've "optimized" the system to the point where repeatable results
> > > from VACUUM are impossible :-(
> > 
> > The synchronous_commit issue is an old one. It might actually be worth
> > addressing it by flushing out pending async commits out instead. It just
> > started to be noticeable when tenk1 load and vacuum were moved closer.
> > 
> > 
> > What do you think about applying a polished version of what I posted in
> > https://postgr.es/m/20220414164830.63rk5zqsvtqqk7qz%40alap3.anarazel.de
> > ? That'd tell us a bit more about the horizon etc.
> 
> No objection.
> 
> > It's also interesting that it only happens in the installcheck cases,
> > afaics, not the check ones. Although that might just be because there's
> > more of them...
> 
> I suspect the failure is somehow impossible in "check".  Yesterday, I cranked
> up the number of locales, so there are now a lot more installcheck.  Before
> that, each farm run had one "check" and two "installcheck".  Those days saw
> ten installcheck failures, zero check failures.
> 
> Like Tom, I'm failing to reproduce this outside the buildfarm client.  I wrote
> a shell script to closely resemble the buildfarm installcheck sequence, but
> it's lasted a dozen runs without failing.

But 24s after that email, it did reproduce the problem.  Same symptoms as the
last buildfarm runs, including visfrac=0.  I'm attaching my script.  (It has
various references to my home directory, so it's not self-contained.)

--QTprm0S8XgL7H0Dt
Content-Type: application/x-sh
Content-Disposition: attachment; filename="repro-wrasse-vis.sh"
Content-Transfer-Encoding: quoted-printable

# run with: env -i nice -+20 sh repro-wrasse-vis.sh=0A=0Aset -ex=0A=0Aexpor=
t CCACHE_MAXSIZE=3D1536M=0Aexport LD_LIBRARY_PATH=3D/home/nm/sw/nopath/uuid=
-64/lib:/home/nm/sw/nopath/openldap-64/lib=0Aexport PATH=3D/home/nm/bin:/ho=
me/nm/sw/bison/bin:/home/nm/sw/ccache/bin:/usr/bin:/opt/csw/bin=0Aexport PE=
RL5LIB=3D/home/nm/src/build/IPC-Run-0.99/lib=0Aexport PGCTLTIMEOUT=3D5400=
=0Aexport PGISOLATIONTIMEOUT=3D5400=0Aexport PG_TEST_TIMEOUT_DEFAULT=3D5400=
=0Aexport TMPDIR=3D/home/nm/farm/tmp=0A=0Aexport PGUSER=3Dbuildfarm=0A=0Alo=
cale=3Dko_KR.UTF-8=0Arm -rf data-$locale=0APATH=3D/export/home/nm/farm/stud=
io64v12_6/HEAD/instkeep.2022-04-14_22-58-00/bin:$PATH=0Ainitdb -A trust -U =
buildfarm --locale=3D$locale data-$locale=0Acat >>data-$locale/postgresql.c=
onf <<EOF=0Aunix_socket_directories =3D '$TMPDIR'=0Alisten_addresses =3D ''=
=0Alog_line_prefix =3D '%m [%p:%l] %q%a '=0Alog_connections =3D 'true'=0Alo=
g_disconnections =3D 'true'=0Alog_statement =3D 'all'=0Afsync =3D off=0Aaut=
hentication_timeout =3D '600s'=0Awal_receiver_timeout =3D '18000s'=0Awal_se=
nder_timeout =3D '18000s'=0Alog_autovacuum_min_duration =3D 0=0AEOF=0A=0Aex=
port PGHOST=3D$TMPDIR=0Apg_ctl -D data-$locale -l logfile -w start >startlo=
g 2>&1=0A=0A(=0A    cd /export/home/nm/farm/studio64v12_6/HEAD/pgsqlkeep.20=
22-04-14_22-58-00/src/test/regress=0A    make installcheck-parallel=0A)=0Ae=
xitcode=3D$?=0Apg_ctl -D data-$locale -l logfile -w stop=0Aexit $exitcode=0A
--QTprm0S8XgL7H0Dt--