public inbox for [email protected]
help / color / mirror / Atom feedRe: BUG #19373: One backend hanging in AioIoUringExecution blocking other backends
2+ messages / 2 participants
[nested] [flat]
* Re: BUG #19373: One backend hanging in AioIoUringExecution blocking other backends
@ 2026-01-17 00:09 surya poondla <[email protected]>
2026-01-17 09:03 ` Re: BUG #19373: One backend hanging in AioIoUringExecution blocking other backends Michael Kröll <[email protected]>
0 siblings, 1 reply; 2+ messages in thread
From: surya poondla @ 2026-01-17 00:09 UTC (permalink / raw)
To: [email protected]; [email protected]
Hi Michael,
Thank you for the detailed report.
Even though there was a *global* statement_timeout=61s configured, backends
> accessing the same table were hanging with ``LWLock AioUringCompletion``
>
The statement_timeout not interrupting and not erroring out looks weird,
and this part could be a postgres bug in itself.
> Restarting the cluster did not go through until the hanging leader PID was
> ``SIGKILL``ed
>
Am I understanding this correctly as "Normal shutdown (SIGTERM or pg_ctl
stop) did not complete, and postmaster remained waiting on
until AioIoUringExecution was force killed" ?
I’m interested in digging into this and am wondering about the below
1. What filesystem and storage was this instance running on
2. Was this a parallel sequential scan, was any index access involved?
3. By any chance do you have a reproducible test case?
4. Can you share what shared_preload_libraries you are using?
Regards,
Surya Poondla
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: BUG #19373: One backend hanging in AioIoUringExecution blocking other backends
2026-01-17 00:09 Re: BUG #19373: One backend hanging in AioIoUringExecution blocking other backends surya poondla <[email protected]>
@ 2026-01-17 09:03 ` Michael Kröll <[email protected]>
0 siblings, 0 replies; 2+ messages in thread
From: Michael Kröll @ 2026-01-17 09:03 UTC (permalink / raw)
To: surya poondla <[email protected]>; +Cc: [email protected]
On 1/17/26 01:09, surya poondla wrote:
> Restarting the cluster did not go through until the hanging leader
> PID was
> ``SIGKILL``ed
>
> Am I understanding this correctly as "Normal shutdown (SIGTERM or pg_ctl
> stop) did not complete, and postmaster remained waiting on
> until AioIoUringExecution was force killed" ?
Remained waiting until the PG backend process in wait_event
AioIoUringExecution was SIGKILLed.
> I’m interested in digging into this and am wondering about the below
> 1. What filesystem and storage was this instance running on
Two of
Disk model: INTEL SSDPE2KX010T7
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
in a software RAID 1 on ext4
> 2. Was this a parallel sequential scan, was any index access involved?
I not have the exact query plan for this one as it depends on the bind
parameters passed and there are three ARRAY type bind parameters for the
query in question.
Typically the query plan looks like this:
Gather (cost=1825.51..30731.85 rows=6391 width=678)
Workers Planned: 2
-> Nested Loop (cost=825.51..29092.75 rows=2663 width=678)
-> Parallel Bitmap Heap Scan on offer o
(cost=825.09..22161.55 rows=2676 width=542)
Recheck Cond: (id = ANY
('{3048845,2121345,2840302,2807790,3273743,2798121,2017850,3226237,1501236,2449122,2891576,2927727,3526960,3467910,2929690,3299523,3458918,2840304,1707208,2101471,245>
Filter: (vfb_in_de AND ((loc)::text = ANY
('{de,at}'::text[])))
-> Bitmap Index Scan on offer_id_npr_lzf_idx
(cost=0.00..822.62 rows=14268 width=0)
Index Cond: (id = ANY
('{3048845,2121345,2840302,2807790,3273743,2798121,2017850,3226237,1501236,2449122,2891576,2927727,3526960,3467910,2929690,3299523,3458918,2840304,1707208,2101471>
-> Index Scan using gh_haendler_pkey on gh_haendler
(cost=0.42..2.58 rows=1 width=8)
Index Cond: (h_id = o.h_id)
Filter: (COALESCE(multimerchants_template_id, o.h_id) <>
ALL ('{4957}'::integer[]))
In the case of the problematic query/params/backend there were *two*
workers with the associated leader PID found in pg_stat_activity.
> 3. By any chance do you have a reproducible test case?
Unfortunately not: We had this happily running in production for
multiple weeks without issues on three identical machines and it
happened once on one. We could not reproduce it on our
development/testing machines.
At the moment we have switched to io_method=worker which at least for
the index driven use cases on those machines won't make a big difference.
We could configure one of those three boxes again with io_uring but will
not know ahead if we'll trigger this issue ever again and would need to
have a solid specific monitoring in place before.
At the same time, we've upgraded to Kernel 6.12 by now on those boxes
and if the issue was related to an interaction dependant on the io_uring
versions, this might be another reason, we'll likely not see the same
issue again.
> 4. Can you share what shared_preload_libraries you are using?
pg_stat_statements is the only one used there.
Thank you for having a look. Sorry for not being able to provide more
specifics.
BR,
Michael
> Regards,
> Surya Poondla
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2026-01-17 09:03 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-01-17 00:09 Re: BUG #19373: One backend hanging in AioIoUringExecution blocking other backends surya poondla <[email protected]>
2026-01-17 09:03 ` Michael Kröll <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox