agora inbox for pgsql-bugs@postgresql.org
help / color / mirror / Atom feedBUG #19557: Parallel leader stuckat ParallelFinish leaked InterruptHoldoffCount
2+ messages / 2 participants
[nested] [flat]
* BUG #19557: Parallel leader stuckat ParallelFinish leaked InterruptHoldoffCount
@ 2026-07-17 11:24 PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 2+ messages in thread
From: PG Bug reporting form @ 2026-07-17 11:24 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: jacob@brejnbjerg.com
The following bug has been logged on the website:
Bug reference: 19557
Logged by: Jacob Mørk Brejnbjerg
Email address: jacob@brejnbjerg.com
PostgreSQL version: 18.4
Operating system: Ubuntu 24.04 (kernel 6.8.0-124-generic)
Description:
A backend executing a parallel query has been stuck for 40+ hours in
wait_event IPC/ParallelFinish. All of its parallel workers exited cleanly
(exit code 0, nothing logged). pg_cancel_backend() and
pg_terminate_backend() have no effect. We kept the backend alive and
debugged it with gdb + debug symbols; the evidence shows a leaked
InterruptHoldoffCount, plus a design interaction that turns any such leak
into a permanently unkillable backend.
Possibly relevant configuration: io_method = io_uring,
max_parallel_workers_per_gather = 48,
shared_preload_libraries = 'timescaledb,pg_parquet'
(timescaledb 2.27.2, pg_parquet 0.5.1).
PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1).
The query is a COPY (SELECT <aggregation over one table> GROUP BY ...)
TO STDOUT (FORMAT parquet) issued via pg_parquet's COPY hook; the inner
select runs a parallel plan (Gather, 10+ workers planned) over
TimescaleDB compressed-hypertable chunks, i.e. an AIO-read-heavy scan.
Backtrace of the stuck leader (main thread; the only other thread is an
io_uring kernel worker, "iou-wrk-<pid>"):
#0 epoll_wait (epfd=754, events=..., maxevents=1, timeout=-1)
#1 WaitEventSetWait ()
#2 WaitLatch ()
#3 WaitForParallelWorkersToFinish ()
#4 ExecParallelFinish ()
#5-#8 (Gather shutdown path, static functions)
#9 standard_ExecutorRun ()
#10-#11 PortalRun ()
#12-#17 pg_parquet.so (COPY TO hook driving the portal)
#18 prev_ProcessUtility (timescaledb process_utility.c:124)
#19 timescaledb_ddl_command_start (...)
#20-#22 PortalRun ()
#23-#24 PostgresMain ()
Key state read from the live process via gdb:
InterruptPending = 1
ProcDiePending = 1 <- pg_terminate_backend arrived, never
serviced
QueryCancelPending = 1 <- pg_cancel_backend arrived, never serviced
InterruptHoldoffCount = 1 <- held, forever
CritSectionCount = 0
num_held_lwlocks = 0 <- NO LWLock is held
ParallelMessagePending = 1 <- worker messages queued, unprocessable
/proc signal state: SigPnd/ShdPnd are all zero - the SIGTERM/SIGINT were
delivered and consumed by the handlers (which set the flags above); the
backend wakes from WaitLatch, but CHECK_FOR_INTERRUPTS() is a no-op with
InterruptHoldoffCount > 0, so it just re-sleeps.
Why this is a permanent hang and not just an uncancellable wait:
worker-exit accounting is itself interrupt-driven. The workers' detach
notifications are only processed via CHECK_FOR_INTERRUPTS() ->
ProcessParallelMessages(), which is what clears each worker's error_mqh.
With the holdoff leaked, ProcessParallelMessages never runs
(ParallelMessagePending stays 1 - verified in the live process), so
WaitForParallelWorkersToFinish() believes the workers are still attached
and waits forever. The same leaked holdoff blocks ProcDiePending. One
leaked holdoff therefore produces both symptoms; only SIGKILL
(crash-restart) or a debugger can clear the backend.
On the origin of the leak: num_held_lwlocks = 0 rules out a stuck
LWLockAcquire, and the server log shows NO error from this backend for
its entire lifetime, which we believe rules out the
"ereport(ERROR) raised under HOLD, caught by PG_TRY" pattern (the raise
would have been logged by errfinish). That points to a straight-line
code path that called HOLD_INTERRUPTS() and returned without
RESUME_INTERRUPTS(). We audited the read path: bufmgr.c, read_stream.c,
shm_mq.c, condition_variable.c contain no bare holds; timescaledb 2.27.2
and pg_parquet 0.5.1 (and pgrx 0.16.1) contain no HOLD_INTERRUPTS at
all. The remaining bare holds on this workload's path are in
src/backend/storage/aio/aio.c (four HOLD/RESUME regions, including
pgaio_io_reclaim which runs completion callbacks under HOLD) and the
parallel-message path itself. Given io_method = io_uring and an
AIO-heavy scan we suspect an AIO path, but we could not identify the
exact site from the live process.
Frequency: once in ~6 days of heavy production use of this COPY path
(thousands of executions/day). Not deterministically reproducible so
far.
We have a gcore of the stuck backend and can keep it (and/or the live
process) available - happy to run additional gdb commands against it.
Separately, it may be worth considering a defensive measure: because
WaitForParallelWorkersToFinish depends on interrupt processing for its
own termination condition, any holdoff leak from any subsystem converts
into a silent, unkillable backend. Detecting "waiting on ParallelFinish
with INTERRUPTS_CAN_BE_PROCESSED() false and ParallelMessagePending set"
and at least logging (or treating it as an invariant violation) would
make this failure mode visible instead of silent.
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: BUG #19557: Parallel leader stuckat ParallelFinish leaked InterruptHoldoffCount
@ 2026-07-24 12:06 Rafia Sabih <rafia.pghackers@gmail.com>
parent: PG Bug reporting form <noreply@postgresql.org>
0 siblings, 0 replies; 2+ messages in thread
From: Rafia Sabih @ 2026-07-24 12:06 UTC (permalink / raw)
To: jacob@brejnbjerg.com, pgsql-bugs@lists.postgresql.org, Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>
Adding Robert and Andres because they know these areas best and can provide
some valuable insights to this issue.
On Fri, 17 Jul 2026 at 22:00, PG Bug reporting form <noreply@postgresql.org>
wrote:
> The following bug has been logged on the website:
>
> Bug reference: 19557
> Logged by: Jacob Mørk Brejnbjerg
> Email address: jacob@brejnbjerg.com
> PostgreSQL version: 18.4
> Operating system: Ubuntu 24.04 (kernel 6.8.0-124-generic)
> Description:
>
> A backend executing a parallel query has been stuck for 40+ hours in
> wait_event IPC/ParallelFinish. All of its parallel workers exited cleanly
> (exit code 0, nothing logged). pg_cancel_backend() and
> pg_terminate_backend() have no effect. We kept the backend alive and
> debugged it with gdb + debug symbols; the evidence shows a leaked
> InterruptHoldoffCount, plus a design interaction that turns any such leak
> into a permanently unkillable backend.
>
> Possibly relevant configuration: io_method = io_uring,
> max_parallel_workers_per_gather = 48,
> shared_preload_libraries = 'timescaledb,pg_parquet'
> (timescaledb 2.27.2, pg_parquet 0.5.1).
> PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1).
>
> The query is a COPY (SELECT <aggregation over one table> GROUP BY ...)
> TO STDOUT (FORMAT parquet) issued via pg_parquet's COPY hook; the inner
> select runs a parallel plan (Gather, 10+ workers planned) over
> TimescaleDB compressed-hypertable chunks, i.e. an AIO-read-heavy scan.
>
> Backtrace of the stuck leader (main thread; the only other thread is an
> io_uring kernel worker, "iou-wrk-<pid>"):
>
> #0 epoll_wait (epfd=754, events=..., maxevents=1, timeout=-1)
> #1 WaitEventSetWait ()
> #2 WaitLatch ()
> #3 WaitForParallelWorkersToFinish ()
> #4 ExecParallelFinish ()
> #5-#8 (Gather shutdown path, static functions)
> #9 standard_ExecutorRun ()
> #10-#11 PortalRun ()
> #12-#17 pg_parquet.so (COPY TO hook driving the portal)
> #18 prev_ProcessUtility (timescaledb process_utility.c:124)
> #19 timescaledb_ddl_command_start (...)
> #20-#22 PortalRun ()
> #23-#24 PostgresMain ()
>
> Key state read from the live process via gdb:
>
> InterruptPending = 1
> ProcDiePending = 1 <- pg_terminate_backend arrived, never
> serviced
> QueryCancelPending = 1 <- pg_cancel_backend arrived, never serviced
> InterruptHoldoffCount = 1 <- held, forever
> CritSectionCount = 0
> num_held_lwlocks = 0 <- NO LWLock is held
> ParallelMessagePending = 1 <- worker messages queued, unprocessable
>
> /proc signal state: SigPnd/ShdPnd are all zero - the SIGTERM/SIGINT were
> delivered and consumed by the handlers (which set the flags above); the
> backend wakes from WaitLatch, but CHECK_FOR_INTERRUPTS() is a no-op with
> InterruptHoldoffCount > 0, so it just re-sleeps.
>
> Why this is a permanent hang and not just an uncancellable wait:
> worker-exit accounting is itself interrupt-driven. The workers' detach
> notifications are only processed via CHECK_FOR_INTERRUPTS() ->
> ProcessParallelMessages(), which is what clears each worker's error_mqh.
> With the holdoff leaked, ProcessParallelMessages never runs
> (ParallelMessagePending stays 1 - verified in the live process), so
> WaitForParallelWorkersToFinish() believes the workers are still attached
> and waits forever. The same leaked holdoff blocks ProcDiePending. One
> leaked holdoff therefore produces both symptoms; only SIGKILL
> (crash-restart) or a debugger can clear the backend.
>
> On the origin of the leak: num_held_lwlocks = 0 rules out a stuck
> LWLockAcquire, and the server log shows NO error from this backend for
> its entire lifetime, which we believe rules out the
> "ereport(ERROR) raised under HOLD, caught by PG_TRY" pattern (the raise
> would have been logged by errfinish). That points to a straight-line
> code path that called HOLD_INTERRUPTS() and returned without
> RESUME_INTERRUPTS(). We audited the read path: bufmgr.c, read_stream.c,
> shm_mq.c, condition_variable.c contain no bare holds; timescaledb 2.27.2
> and pg_parquet 0.5.1 (and pgrx 0.16.1) contain no HOLD_INTERRUPTS at
> all. The remaining bare holds on this workload's path are in
> src/backend/storage/aio/aio.c (four HOLD/RESUME regions, including
> pgaio_io_reclaim which runs completion callbacks under HOLD) and the
> parallel-message path itself. Given io_method = io_uring and an
> AIO-heavy scan we suspect an AIO path, but we could not identify the
> exact site from the live process.
>
> Frequency: once in ~6 days of heavy production use of this COPY path
> (thousands of executions/day). Not deterministically reproducible so
> far.
>
> We have a gcore of the stuck backend and can keep it (and/or the live
> process) available - happy to run additional gdb commands against it.
>
> Separately, it may be worth considering a defensive measure: because
> WaitForParallelWorkersToFinish depends on interrupt processing for its
> own termination condition, any holdoff leak from any subsystem converts
> into a silent, unkillable backend. Detecting "waiting on ParallelFinish
> with INTERRUPTS_CAN_BE_PROCESSED() false and ParallelMessagePending set"
> and at least logging (or treating it as an invariant violation) would
> make this failure mode visible instead of silent.
>
>
>
>
>
--
Regards,
Rafia Sabih
CYBERTEC PostgreSQL International GmbH
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2026-07-24 12:06 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-07-17 11:24 BUG #19557: Parallel leader stuckat ParallelFinish leaked InterruptHoldoffCount PG Bug reporting form <noreply@postgresql.org>
2026-07-24 12:06 ` Rafia Sabih <rafia.pghackers@gmail.com>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox