Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <pgsql-hackers-owner+archive@lists.postgresql.org>)
	id 1iIx7B-0001DT-TU
	for pgsql-hackers@arkaria.postgresql.org; Fri, 11 Oct 2019 15:46:02 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.89)
	(envelope-from <pgsql-hackers-owner+archive@lists.postgresql.org>)
	id 1iIx79-00055a-L0
	for pgsql-hackers@arkaria.postgresql.org; Fri, 11 Oct 2019 15:45:59 +0000
Received: from makus.postgresql.org ([2001:4800:3e1:1::229])
	by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <tgl@sss.pgh.pa.us>)
	id 1iIx79-00055R-8c
	for pgsql-hackers@lists.postgresql.org; Fri, 11 Oct 2019 15:45:59 +0000
Received: from sss.pgh.pa.us ([66.207.139.130])
	by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
	(Exim 4.92)
	(envelope-from <tgl@sss.pgh.pa.us>)
	id 1iIx6p-0005b3-Vd
	for pgsql-hackers@postgresql.org; Fri, 11 Oct 2019 15:45:57 +0000
Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1])
	by sss.pgh.pa.us (8.14.4/8.14.4) with ESMTP id x9BFjV5c020033;
	Fri, 11 Oct 2019 11:45:31 -0400
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
cc: mark@2ndquadrant.com, Thomas Munro <thomas.munro@gmail.com>,
        Justin Pryzby <pryzby@telsasoft.com>,
        Andres Freund <andres@anarazel.de>,
        pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: stress test for parallel workers
In-reply-to: <06d6be5a-7af2-ef78-0abc-6bfe500ead87@2ndQuadrant.com>
References: <20190723162703.GM22387@telsasoft.com>
 <6762.1563902927@sss.pgh.pa.us> <20190723174222.GO22387@telsasoft.com>
 <CA+hUKGLch1bNWdG-G8YaeJbyVsper6hG86Ugx9tSWG3=a1R89Q@mail.gmail.com>
 <32179.1563919918@sss.pgh.pa.us>
 <CA+hUKGKZS32Lx1zViq0W8omCjHjBsN=dJZ8nX-rkhv1W=b802Q@mail.gmail.com>
 <17389.1563945314@sss.pgh.pa.us>
 <CA+hUKGL6cDyb2maq2P60cEsjFK=3saBCAj7sDzE3jysL-PRwqg@mail.gmail.com>
 <31032.1565152159@sss.pgh.pa.us>
 <CA+hUKGJNS6gZ=w=+wxjqhmck-voJ7FBuejHnw8mQgAp2yt0A9g@mail.gmail.com>
 <18537.1570421268@sss.pgh.pa.us> <5350.1570743291@sss.pgh.pa.us>
 <84813a30-8c34-9c32-7ad5-90d9eefba468@2ndQuadrant.com>
 <06d6be5a-7af2-ef78-0abc-6bfe500ead87@2ndQuadrant.com>
Comments: In-reply-to Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
	message dated "Fri, 11 Oct 2019 11:12:28 -0400"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <20031.1570808731.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Fri, 11 Oct 2019 11:45:31 -0400
Message-ID: <20032.1570808731@sss.pgh.pa.us>
List-Id: <pgsql-hackers.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-hackers@lists.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-hackers>
Precedence: bulk

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>> At least on F29 I have set /proc/sys/kernel/core_pattern and it works.

FWIW, I'm not excited about that as a permanent solution.  It requires
root privilege, and it affects the whole machine not only the buildfarm,
and making it persist across reboots is even more invasive.

> I have done the same on this machine. wobbegong runs every hour, so
> let's see what happens next. With any luck the buildfarm will give us a
> stack trace without needing further action.

I already collected one manually.  It looks like this:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  sigusr1_handler (postgres_signal_arg=3D10) at postmaster.c:5114
5114    {
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.26-30.fc27=
.ppc64le
(gdb) bt
#0  sigusr1_handler (postgres_signal_arg=3D10) at postmaster.c:5114
#1  <signal handler called>
#2  0x00007fff93923ca4 in sigprocmask () from /lib64/libc.so.6
#3  0x00000000103fad08 in reaper (postgres_signal_arg=3D<optimized out>)
    at postmaster.c:3215
#4  <signal handler called>
#5  0x00007fff93923ca4 in sigprocmask () from /lib64/libc.so.6
#6  0x00000000103f9f98 in sigusr1_handler (postgres_signal_arg=3D<optimize=
d out>)
    at postmaster.c:5275
#7  <signal handler called>
#8  0x00007fff93923ca4 in sigprocmask () from /lib64/libc.so.6
#9  0x00000000103fad08 in reaper (postgres_signal_arg=3D<optimized out>)
    at postmaster.c:3215
#10 <signal handler called>
#11 sigusr1_handler (postgres_signal_arg=3D10) at postmaster.c:5114
#12 <signal handler called>
#13 0x00007fff93923ca4 in sigprocmask () from /lib64/libc.so.6
#14 0x00000000103f9f98 in sigusr1_handler (postgres_signal_arg=3D<optimize=
d out>)
    at postmaster.c:5275
#15 <signal handler called>
#16 0x00007fff93923ca4 in sigprocmask () from /lib64/libc.so.6
#17 0x00000000103fad08 in reaper (postgres_signal_arg=3D<optimized out>)
    at postmaster.c:3215
...
#572 <signal handler called>
#573 0x00007fff93923ca4 in sigprocmask () from /lib64/libc.so.6
#574 0x00000000103f9f98 in sigusr1_handler (
    postgres_signal_arg=3D<optimized out>) at postmaster.c:5275
#575 <signal handler called>
#576 0x00007fff93923ca4 in sigprocmask () from /lib64/libc.so.6
#577 0x00000000103fad08 in reaper (postgres_signal_arg=3D<optimized out>)
    at postmaster.c:3215
#578 <signal handler called>
#579 sigusr1_handler (postgres_signal_arg=3D10) at postmaster.c:5114
#580 <signal handler called>
#581 0x00007fff93a01514 in select () from /lib64/libc.so.6
#582 0x00000000103f7ad8 in ServerLoop () at postmaster.c:1682
#583 PostmasterMain (argc=3D<optimized out>, argv=3D<optimized out>)
    at postmaster.c:1391
#584 0x0000000000000000 in ?? ()

What we've apparently got here is that signals were received
so fast that the postmaster ran out of stack space.  I remember
Andres complaining about this as a theoretical threat, but I
hadn't seen it in the wild before.

I haven't finished investigating though, as there are some things
that remain to be explained.  The dependency on having
force_parallel_mode =3D regress makes sense now, because the extra
traffic to launch and reap all those parallel workers would
increase the stress on the postmaster (and it seems likely that
this stack trace corresponds exactly to alternating launch and
reap signals).  But why does it only happen during the pg_upgrade
test --- plain "make check" ought to be about the same?  I also
want to investigate why clang builds seem more prone to this
than gcc builds on the same machine; that might just be down to
more or less stack consumption, but it bears looking into.

			regards, tom lane