Date: Fri, 29 Oct 2021 04:57:25 -0700
From: Noah Misch <noah@leadboat.com>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Andrey Borodin <x4mmm@yandex-team.ru>,
	PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>,
	Michael Paquier <michael@paquier.xyz>, Peter Geoghegan <pg@bowt.ie>,
	Andres Freund <andres@anarazel.de>
Subject: Re: conchuela timeouts since 2021-10-09 system upgrade
Message-ID: <20211029115725.GA309057@rfd.leadboat.com>
References: <20211017151205.GA2384251@rfd.leadboat.com>
 <9BA43784-E4A7-4638-900C-FC9868C388FA@yandex-team.ru>
 <20211019030212.GB3403374@rfd.leadboat.com>
 <20211024050028.GA3310519@rfd.leadboat.com>
 <64835CF9-D9AA-49BF-A685-01C23B1023C1@yandex-team.ru>
 <20211024161942.GB3945842@rfd.leadboat.com>
 <640FA6DC-A998-4499-B3B4-2773685B8C56@yandex-team.ru>
 <20211029025600.GB298507@rfd.leadboat.com>
 <581193.1635477596@sss.pgh.pa.us>
 <CA+hUKGKMHyKheU9G_-vVcN-WPJ-YBX=9LrHfJBVbVbAmcMtjzA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CA+hUKGKMHyKheU9G_-vVcN-WPJ-YBX=9LrHfJBVbVbAmcMtjzA@mail.gmail.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Archived-At: 
 <https://www.postgresql.org/message-id/20211029115725.GA309057%40rfd.leadboat.com>
Precedence: bulk

On Fri, Oct 29, 2021 at 04:42:31PM +1300, Thomas Munro wrote:
> On Fri, Oct 29, 2021 at 4:20 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > It indeed is looking like 7f580aa made the problem go away on conchuela,
> > but do we understand why?

I don't.

> > The only theory I can think of is "kernel bug",
> > but while that's plausible for prairiedog it seems hard to credit for a
> > late-model BSD kernel.

DragonFly BSD is a niche OS, so I'm more willing than usual to conclude that.
Could be a bug in IPC::Run or in the port of Perl to DragonFly, but those feel
less likely than the kernel.  The upgrade from DragonFly v4.4.3 to DragonFly
v6.0.0, which introduced this form of PostgreSQL test breakage, also updated
Perl from v5.20.3 to 5.32.1.

> I have yet to even log into a DBSD system (my attempt to install the
> 6.0.1 ISO on bhyve failed for lack of a driver, or something), but I
> do intend to get it working at some point.  But I can offer a poorly
> researched wildly speculative hypothesis: DBSD forked from FBSD in
> 2003.  macOS 10.3 took FBSD's kqueue code in... 2003.  So maybe a bug
> was fixed later that they both inherited?  Or perhaps that makes no
> sense, I dunno.  It'd be nice to try to write a repro and send them a
> report, if we can.

The conchuela bug and the prairiedog bug both present with a timeout in
IPC::Run::finish, but the similarity ends there.  On prairiedog, the
postmaster was stuck when it should have been reading a query from pgbench.
On conchuela, pgbench ran to completion and became a zombie, and IPC::Run got
stuck when it should have been reaping that zombie.  Good thought, however.