From: Tom Lane <tgl@sss.pgh.pa.us>
To: mark@2ndquadrant.com
cc: Andrew Dunstan <andrew@dunslane.net>,
        Thomas Munro <thomas.munro@gmail.com>,
        Justin Pryzby <pryzby@telsasoft.com>,
        pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: stress test for parallel workers
In-reply-to: <18537.1570421268@sss.pgh.pa.us>
References: <20190723162703.GM22387@telsasoft.com>
 <6762.1563902927@sss.pgh.pa.us> <20190723174222.GO22387@telsasoft.com>
 <CA+hUKGLch1bNWdG-G8YaeJbyVsper6hG86Ugx9tSWG3=a1R89Q@mail.gmail.com>
 <32179.1563919918@sss.pgh.pa.us>
 <CA+hUKGKZS32Lx1zViq0W8omCjHjBsN=dJZ8nX-rkhv1W=b802Q@mail.gmail.com>
 <17389.1563945314@sss.pgh.pa.us>
 <CA+hUKGL6cDyb2maq2P60cEsjFK=3saBCAj7sDzE3jysL-PRwqg@mail.gmail.com>
 <31032.1565152159@sss.pgh.pa.us>
 <CA+hUKGJNS6gZ=w=+wxjqhmck-voJ7FBuejHnw8mQgAp2yt0A9g@mail.gmail.com>
 <18537.1570421268@sss.pgh.pa.us>
Comments: In-reply-to Tom Lane <tgl@sss.pgh.pa.us>
	message dated "Mon, 07 Oct 2019 00:07:48 -0400"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <5349.1570743291.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Thu, 10 Oct 2019 17:34:51 -0400
Message-ID: <5350.1570743291@sss.pgh.pa.us>
Precedence: bulk

I wrote:
>>> Yeah, I've been wondering whether pg_ctl could fork off a subprocess
>>> that would fork the postmaster, wait for the postmaster to exit, and t=
hen
>>> report the exit status.

> [ pushed at 6a5084eed ]
> Given wobbegong's recent failure rate, I don't think we'll have to wait
> long.

Indeed, we didn't:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=3Dwobbegong&dt=3D2=
019-10-10%2020%3A54%3A46

The tail end of the system log looks like

2019-10-10 21:00:33.717 UTC [15127:306] pg_regress/date FATAL:  postmaster=
 exited during a parallel transaction
2019-10-10 21:00:33.717 UTC [15127:307] pg_regress/date LOG:  disconnectio=
n: session time: 0:00:02.896 user=3Dfedora database=3Dregression host=3D[l=
ocal]
/bin/sh: line 1: 14168 Segmentation fault      (core dumped) "/home/fedora=
/build-farm-10-clang/buildroot/HEAD/pgsql.build/tmp_install/home/fedora/bu=
ild-farm-clang/buildroot/HEAD/inst/bin/postgres" -F -c listen_addresses=3D=
"" -k "/tmp/pg_upgrade_check-ZrhQ4h"
postmaster exit status is 139

So that's definitive proof that the postmaster is suffering a SIGSEGV.
Unfortunately, we weren't blessed with a stack trace, even though
wobbegong is running a buildfarm client version that is new enough
to try to collect one.  However, seeing that wobbegong is running
a pretty-recent Fedora release, the odds are that systemd-coredump
has commandeered the core dump and squirreled it someplace where
we can't find it.

Much as one could wish otherwise, systemd doesn't seem likely to
either go away or scale back its invasiveness; so I'm afraid we
are probably going to need to teach the buildfarm client how to
negotiate with systemd-coredump at some point.  I don't much want
to do that right this minute, though.

A nearer-term solution would be to reproduce this manually and
dig into the core.  Mark, are you in a position to give somebody
ssh access to wobbegong's host, or another similarly-configured VM?

(While at it, it'd be nice to investigate the infinite_recurse
failures we've been seeing on all those ppc64 critters ...)

			regards, tom lane