From: Noah Misch <noah@leadboat.com>
To: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Peter Geoghegan <pg@bowt.ie>
Cc: Andres Freund <andres@anarazel.de>
Subject: conchuela timeouts since 2021-10-09 system upgrade
Date: Sun, 24 Oct 2021 09:19:42 -0700
Message-ID: <20211024161942.GB3945842@rfd.leadboat.com> (raw)
In-Reply-To: <64835CF9-D9AA-49BF-A685-01C23B1023C1@yandex-team.ru>
References: <364BAA98-1A3F-4C39-B0D9-ED8C40EA902D@yandex-team.ru>
<20210908064515.GA2496820@rfd.leadboat.com>
<20210920044143.GA3414844@rfd.leadboat.com>
<593C6C66-F19E-412A-9896-F1F5B272424A@yandex-team.ru>
<20210925201012.GC4134968@rfd.leadboat.com>
<20211017151205.GA2384251@rfd.leadboat.com>
<9BA43784-E4A7-4638-900C-FC9868C388FA@yandex-team.ru>
<20211019030212.GB3403374@rfd.leadboat.com>
<20211024050028.GA3310519@rfd.leadboat.com>
<64835CF9-D9AA-49BF-A685-01C23B1023C1@yandex-team.ru>
On Sun, Oct 24, 2021 at 02:45:38PM +0300, Andrey Borodin wrote:
> > 24 окт. 2021 г., в 08:00, Noah Misch <noah@leadboat.com> написал(а):
> > Buildfarm member conchuela (DragonFly BSD 6.0) has gotten multiple
> > "IPC::Run: timeout on timer" in the new tests. No other animal has.
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2021-10-24%2003%3A05%3A09
> > is an example run. The pgbench queries finished quickly, but the
> > $pgbench_h->finish() apparently timed out after 180s. I guess this would be
> > consistent with pgbench blocking in write(), waiting for something to empty a
> > pipe buffer so it can write more. I thought finish() will drain any incoming
> > I/O, though. This phenomenon has been appearing regularly via
> > src/test/recovery/t/017_shm.pl[1], so this thread doesn't have a duty to
> > resolve it. A stack trace of the stuck pgbench should be informative, though.
>
> Some thoughts:
> 0. I doubt that psql\pgbench is stuck in these failures.
Got it. If pgbench is a zombie, the fault does lie in IPC::Run or the kernel.
> 1. All observed similar failures seem to be related to finish() sub of IPC::Run harness
> 2. Finish must pump any pending data from process [0]. But it can hang if process is waiting for something.
> 3. There is reported bug of finish [1]. But the description is slightly different.
Since that report is about a Perl-process child on Linux, I think we can treat
it as unrelated.
These failures started on 2021-10-09, the day conchuela updated from DragonFly
v4.4.3-RELEASE to DragonFly v6.0.0-RELEASE. It smells like a kernel bug.
Since the theorized kernel bug seems not to affect
src/test/subscription/t/015_stream.pl, I wonder if we can borrow a workaround
from other tests. One thing in common with src/test/recovery/t/017_shm.pl and
the newest failure sites is that they don't write anything to the child stdin.
Does writing e.g. a single byte (that the child doesn't use) work around the
problem? If not, does passing the script via stdin, like "pgbench -f-
<script.sql", work around the problem?
> [0] https://metacpan.org/dist/IPC-Run/source/lib/IPC/Run.pm#L3481
> [1] https://github.com/toddr/IPC-Run/issues/57
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-bugs@postgresql.org
Cc: noah@leadboat.com, x4mmm@yandex-team.ru, michael@paquier.xyz, tgl@sss.pgh.pa.us, pg@bowt.ie, andres@anarazel.de
Subject: Re: conchuela timeouts since 2021-10-09 system upgrade
In-Reply-To: <20211024161942.GB3945842@rfd.leadboat.com>
References: <364BAA98-1A3F-4C39-B0D9-ED8C40EA902D@yandex-team.ru> <20210908064515.GA2496820@rfd.leadboat.com> <20210920044143.GA3414844@rfd.leadboat.com> <593C6C66-F19E-412A-9896-F1F5B272424A@yandex-team.ru> <20210925201012.GC4134968@rfd.leadboat.com> <20211017151205.GA2384251@rfd.leadboat.com> <9BA43784-E4A7-4638-900C-FC9868C388FA@yandex-team.ru> <20211019030212.GB3403374@rfd.leadboat.com> <20211024050028.GA3310519@rfd.leadboat.com> <64835CF9-D9AA-49BF-A685-01C23B1023C1@yandex-team.ru> <20211024161942.GB3945842@rfd.leadboat.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox