From: Tom Lane <tgl@sss.pgh.pa.us>
To: mark@2ndquadrant.com
Cc: Andrew Dunstan <andrew@dunslane.net>
Cc: Thomas Munro <thomas.munro@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: stress test for parallel workers
Date: Thu, 10 Oct 2019 17:34:51 -0400
Message-ID: <5350.1570743291@sss.pgh.pa.us> (raw)
In-Reply-To: <18537.1570421268@sss.pgh.pa.us>
References: <20190723162703.GM22387@telsasoft.com>
<6762.1563902927@sss.pgh.pa.us>
<20190723174222.GO22387@telsasoft.com>
<CA+hUKGLch1bNWdG-G8YaeJbyVsper6hG86Ugx9tSWG3=a1R89Q@mail.gmail.com>
<32179.1563919918@sss.pgh.pa.us>
<CA+hUKGKZS32Lx1zViq0W8omCjHjBsN=dJZ8nX-rkhv1W=b802Q@mail.gmail.com>
<17389.1563945314@sss.pgh.pa.us>
<CA+hUKGL6cDyb2maq2P60cEsjFK=3saBCAj7sDzE3jysL-PRwqg@mail.gmail.com>
<31032.1565152159@sss.pgh.pa.us>
<CA+hUKGJNS6gZ=w=+wxjqhmck-voJ7FBuejHnw8mQgAp2yt0A9g@mail.gmail.com>
<18537.1570421268@sss.pgh.pa.us>
I wrote:
>>> Yeah, I've been wondering whether pg_ctl could fork off a subprocess
>>> that would fork the postmaster, wait for the postmaster to exit, and then
>>> report the exit status.
> [ pushed at 6a5084eed ]
> Given wobbegong's recent failure rate, I don't think we'll have to wait
> long.
Indeed, we didn't:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=wobbegong&dt=2019-10-10%2020%3A54%3A46
The tail end of the system log looks like
2019-10-10 21:00:33.717 UTC [15127:306] pg_regress/date FATAL: postmaster exited during a parallel transaction
2019-10-10 21:00:33.717 UTC [15127:307] pg_regress/date LOG: disconnection: session time: 0:00:02.896 user=fedora database=regression host=[local]
/bin/sh: line 1: 14168 Segmentation fault (core dumped) "/home/fedora/build-farm-10-clang/buildroot/HEAD/pgsql.build/tmp_install/home/fedora/build-farm-clang/buildroot/HEAD/inst/bin/postgres" -F -c listen_addresses="" -k "/tmp/pg_upgrade_check-ZrhQ4h"
postmaster exit status is 139
So that's definitive proof that the postmaster is suffering a SIGSEGV.
Unfortunately, we weren't blessed with a stack trace, even though
wobbegong is running a buildfarm client version that is new enough
to try to collect one. However, seeing that wobbegong is running
a pretty-recent Fedora release, the odds are that systemd-coredump
has commandeered the core dump and squirreled it someplace where
we can't find it.
Much as one could wish otherwise, systemd doesn't seem likely to
either go away or scale back its invasiveness; so I'm afraid we
are probably going to need to teach the buildfarm client how to
negotiate with systemd-coredump at some point. I don't much want
to do that right this minute, though.
A nearer-term solution would be to reproduce this manually and
dig into the core. Mark, are you in a position to give somebody
ssh access to wobbegong's host, or another similarly-configured VM?
(While at it, it'd be nice to investigate the infinite_recurse
failures we've been seeing on all those ppc64 critters ...)
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tgl@sss.pgh.pa.us, mark@2ndquadrant.com, andrew@dunslane.net, thomas.munro@gmail.com, pryzby@telsasoft.com
Subject: Re: stress test for parallel workers
In-Reply-To: <5350.1570743291@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox