pg.ddx.io pgsql-hackers@postgresql.org mailing list archive help / color / mirror / Atom feed From: Jonathan Rudenberg <jonathan@titanous.com>
To: Thomas Munro <thomas.munro@enterprisedb.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Andreas Seltenreich <seltenreich@gmx.de>
Cc: Pg Hackers <pgsql-hackers@postgresql.org>
Subject: Re: [sqlsmith] Unpinning error in parallel worker
Date: Tue, 24 Apr 2018 16:15:31 -0400
Message-ID: <1524600931.3892926.1349473696.17D84D5D@webmail.messagingengine.com> (raw )
In-Reply-To: <CAEepm=0Y_nw=kp-YZtnkvhySYxu4PPONWeF4ap=M05g-STgKbg@mail.gmail.com >
References: <87h92g83t3.fsf@credativ.de >
<CAEepm=1sLjHyauGZoXM3KCT2Gv+CHPEWQcxob_1Oo0KshRp7-Q@mail.gmail.com >
<CAEepm=2VvKCVXjg-REHTikoBUOhtfvfCyiJrT=zHDJyvMNd9pw@mail.gmail.com >
<CAEepm=23=vGz=CgVurPxBGV6SeOJ7YxSaAJKr_aH+f2cV4zrow@mail.gmail.com >
<CA+TgmoZfPOu62bR71bahf90ivOUxXpYOh0RqDiue0+dVVPNrWg@mail.gmail.com >
<1523998351.1231357.1341528496.28EBAF7F@webmail.messagingengine.com >
<CAEepm=0gtExezsnVabv79hKSzn61dbdEbzWRxqyJf1nf8hzppQ@mail.gmail.com >
<1524006065.2172367.1341656968.532A8113@webmail.messagingengine.com >
<CAEepm=1Hft9m2wp86++S86Yrn6C8O2sms4=B=66-scdQGQt7PA@mail.gmail.com >
<1524008630.2186289.1341689208.31A2DB11@webmail.messagingengine.com >
<CAEepm=0awU3+3mdpBTREjSFg1Km1Lx3ikO5NF-fML4tirQMBmw@mail.gmail.com >
<1524579700.3786618.1349037056.03F1C731@webmail.messagingengine.com >
<CAEepm=0Y_nw=kp-YZtnkvhySYxu4PPONWeF4ap=M05g-STgKbg@mail.gmail.com >
On Tue, Apr 24, 2018, at 16:06, Thomas Munro wrote:
> On Wed, Apr 25, 2018 at 2:21 AM, Jonathan Rudenberg
> <jonathan@titanous.com> wrote:
> > This issue happened again in production, here are the stack traces from three we grabbed before nuking the >400 hanging backends.
> >
> > [...]
> > #4 0x000055fccb93b21c in LWLockAcquire+188() at /usr/lib/postgresql/10/bin/postgres at lwlock.c:1233
> > #5 0x000055fccb925fa7 in dsm_create+151() at /usr/lib/postgresql/10/bin/postgres at dsm.c:493
> > #6 0x000055fccb6f2a6f in InitializeParallelDSM+511() at /usr/lib/postgresql/10/bin/postgres at parallel.c:266
> > [...]
>
> Thank you. These stacks are all blocked trying to acquire
> DynamicSharedMemoryControlLock. My theory is that they can't because
> one backend -- the one that emitted the error "FATAL: cannot unpin a
> segment that is not pinned" -- is deadlocked against itself. After
> emitting that error you can see from Andreas's "seabisquit" stack that
> that shmem_exit() runs dsm_backend_shutdown() which runs dsm_detach()
> which tries to acquire DynamicSharedMemoryControlLock again, even
> though we already hold it at that point.
>
> I'll write a patch to fix that unpleasant symptom. While holding
> DynamicSharedMemoryControlLock we shouldn't raise any errors without
> releasing it first, because the error handling path will try to
> acquire it again. That's a horrible failure mode as you have
> discovered.
>
> But that isn't the root problem: we shouldn't be raising that error,
> and I'd love to see the stack of the one process that did that and
> then self-deadlocked. I will have another go at trying to reproduce
> it here today.
Thanks for the update!
We have turned off parallel queries (using max_parallel_workers_per_gather = 0) for now, as the production impact of this bug is unfortunately quite problematic.
What will this failure look like with the patch you've proposed?
Thanks again,
Jonathan
view thread (20+ messages) latest in thread
Message-ID: <1524600931.3892926.1349473696.17D84D5D@webmail.messagingengine.com>
Permalink: ../1524600931.3892926.1349473696.17D84D5D@webmail.messagingengine.com/
Also on: postgresql.org/message-id/1524600931.3892926.1349473696.17D84D5D@webmail.messagingengine.com
copy link · copy postgr.es
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: jonathan@titanous.com, thomas.munro@enterprisedb.com, robertmhaas@gmail.com, seltenreich@gmx.de
Subject: Re: [sqlsmith] Unpinning error in parallel worker
In-Reply-To: <1524600931.3892926.1349473696.17D84D5D@webmail.messagingengine.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox