From: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
To: mark@2ndquadrant.com
Cc: Thomas Munro <thomas.munro@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: stress test for parallel workers
Date: Fri, 11 Oct 2019 11:12:28 -0400
Message-ID: <06d6be5a-7af2-ef78-0abc-6bfe500ead87@2ndQuadrant.com> (raw)
In-Reply-To: <84813a30-8c34-9c32-7ad5-90d9eefba468@2ndQuadrant.com>
References: <20190723162703.GM22387@telsasoft.com>
<6762.1563902927@sss.pgh.pa.us>
<20190723174222.GO22387@telsasoft.com>
<CA+hUKGLch1bNWdG-G8YaeJbyVsper6hG86Ugx9tSWG3=a1R89Q@mail.gmail.com>
<32179.1563919918@sss.pgh.pa.us>
<CA+hUKGKZS32Lx1zViq0W8omCjHjBsN=dJZ8nX-rkhv1W=b802Q@mail.gmail.com>
<17389.1563945314@sss.pgh.pa.us>
<CA+hUKGL6cDyb2maq2P60cEsjFK=3saBCAj7sDzE3jysL-PRwqg@mail.gmail.com>
<31032.1565152159@sss.pgh.pa.us>
<CA+hUKGJNS6gZ=w=+wxjqhmck-voJ7FBuejHnw8mQgAp2yt0A9g@mail.gmail.com>
<18537.1570421268@sss.pgh.pa.us>
<5350.1570743291@sss.pgh.pa.us>
<84813a30-8c34-9c32-7ad5-90d9eefba468@2ndQuadrant.com>
On 10/10/19 6:01 PM, Andrew Dunstan wrote:
> On 10/10/19 5:34 PM, Tom Lane wrote:
>> I wrote:
>>>>> Yeah, I've been wondering whether pg_ctl could fork off a subprocess
>>>>> that would fork the postmaster, wait for the postmaster to exit, and then
>>>>> report the exit status.
>>> [ pushed at 6a5084eed ]
>>> Given wobbegong's recent failure rate, I don't think we'll have to wait
>>> long.
>> Indeed, we didn't:
>>
>> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=wobbegong&dt=2019-10-10%2020%3A54%3A46
>>
>> The tail end of the system log looks like
>>
>> 2019-10-10 21:00:33.717 UTC [15127:306] pg_regress/date FATAL: postmaster exited during a parallel transaction
>> 2019-10-10 21:00:33.717 UTC [15127:307] pg_regress/date LOG: disconnection: session time: 0:00:02.896 user=fedora database=regression host=[local]
>> /bin/sh: line 1: 14168 Segmentation fault (core dumped) "/home/fedora/build-farm-10-clang/buildroot/HEAD/pgsql.build/tmp_install/home/fedora/build-farm-clang/buildroot/HEAD/inst/bin/postgres" -F -c listen_addresses="" -k "/tmp/pg_upgrade_check-ZrhQ4h"
>> postmaster exit status is 139
>>
>> So that's definitive proof that the postmaster is suffering a SIGSEGV.
>> Unfortunately, we weren't blessed with a stack trace, even though
>> wobbegong is running a buildfarm client version that is new enough
>> to try to collect one. However, seeing that wobbegong is running
>> a pretty-recent Fedora release, the odds are that systemd-coredump
>> has commandeered the core dump and squirreled it someplace where
>> we can't find it.
>
>
> At least on F29 I have set /proc/sys/kernel/core_pattern and it works.
I have done the same on this machine. wobbegong runs every hour, so
let's see what happens next. With any luck the buildfarm will give us a
stack trace without needing further action.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: andrew.dunstan@2ndquadrant.com, tgl@sss.pgh.pa.us, mark@2ndquadrant.com, thomas.munro@gmail.com, pryzby@telsasoft.com
Subject: Re: stress test for parallel workers
In-Reply-To: <06d6be5a-7af2-ef78-0abc-6bfe500ead87@2ndQuadrant.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox