From: Antonin Houska <ah@cybertec.at>
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: Nathan Bossart <nathandbossart@gmail.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Cc: alvherre@kurilemu.de
Subject: Re: REPACK (CONCURRENTLY) backend waits indefinitely when decoding worker fails to start
Date: Mon, 31 Aug 2026 12:31:35 +0200
Message-ID: <27780.1788172295@localhost> (raw)
In-Reply-To: <CALj2ACUpxBdkAfvfT4=DsQ4k5Faz7W9Sqk27j+Y4HWexPSvUdA@mail.gmail.com>
References: <CALj2ACVAxA9HxvFe8HSspTJ-UO4Aoz=kuQdZBeLrod0gqUxH3g@mail.gmail.com>
<apBqNzV4N2CKa1ug@nathan>
<CALj2ACUpxBdkAfvfT4=DsQ4k5Faz7W9Sqk27j+Y4HWexPSvUdA@mail.gmail.com>
Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote:
> On Thu, Aug 27, 2026 at 9:47 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
> >
> > On Thu, Aug 27, 2026 at 09:41:49AM -0700, Bharath Rupireddy wrote:
> > > REPACK (CONCURRENTLY) starts a decoding bgworker and then waits in
> > > start_repack_decoding_worker() for the worker to set a shared-memory
> > > flag. The wait has no liveness check on the worker itself. If the
> > > worker never reaches that point (e.g., fork failure under memory
> > > pressure, or when BecomeLockGroupMember() returns false, or early exit
> > > before the shm_mq error redirect is set up), the backend waits
> > > indefinitely with no way out other than cancellation. I reproduced
> > > this with an induced fork failure, so I think we need to tighten this
> > > for both PG19 and HEAD branches.
> >
> > Oops, I just concurrently reported this [0]. Note that teardown can
> > deadlock, too.
> >
> > [0] https://postgr.es/m/apBpOVZOyqrakEr_%40nathan
>
> Thanks. Here's my first attempt at fixing both the fork failure hang
> and the teardown deadlock. I tried to use the parallel query approach
> as much as possible.
>
> The fork failure hang can occur because the backend running concurrent
> repack sleeps on a CV and ignores the SIGUSR1 sent via bgw_notify_pid
> by the postmaster upon fork failure.
>
> The teardown deadlock can occur because the backend waits for the
> worker to exit before detaching the error queue, while the worker is
> blocked writing to a full queue, so both end up waiting on each other.
>
> Although these issues seem rare to hit, I think it's good to tighten
> the repack code because users can see them via SQL. Therefore, I think
> we need to backpatch these to PG19. Please have a look at the attached
> patch.
Thanks for the fix(es). One thing I'm not sure I understand is:
@@ -3851,9 +3900,11 @@ ProcessRepackMessages(void)
/*
* Nothing to do if we haven't launched the worker yet or have already
- * terminated it.
+ * terminated it. stop_repack_decoding_worker() detaches the error queue
+ * before clearing decoding_worker, so also bail out once error_mqh is
+ * gone.
*/
- if (decoding_worker == NULL)
+ if (decoding_worker == NULL || decoding_worker->error_mqh == NULL)
return;
/*
I don't think that stop_repack_decoding_worker() can clear ->error_mqh w/o
also clearing decoding_worker.
Other than that, I'm not sure you need to mention the condition variable in
the comments. And maybe even the mentions of parallel workers are not
necessary.
> While here, I noticed that the same wait event is used for both the
> worker startup wait and the file export wait. Ideally these would have
> separate wait events, but given that the startup wait is typically
> very short, reusing the same one seems fine.
I think that initially I also considered this situation not worth a new wait
event, but I probably had missed an existing one:
WAIT_EVENT_BGWORKER_STARTUP. It's already used for multiple workers, so we
could perhaps use it here.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: ah@cybertec.at, bharath.rupireddyforpostgres@gmail.com, nathandbossart@gmail.com, alvherre@kurilemu.de
Subject: Re: REPACK (CONCURRENTLY) backend waits indefinitely when decoding worker fails to start
In-Reply-To: <27780.1788172295@localhost>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox