pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6)
4+ messages / 3 participants
[nested] [flat]
* BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6)
@ 2026-10-23 13:46 PG Bug reporting form <noreply@postgresql.org>
2026-10-25 18:31 ` Re: BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6) shihao zhong <zhong950419@gmail.com>
0 siblings, 1 reply; 4+ messages in thread
From: PG Bug reporting form @ 2026-10-23 13:46 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: barton.tomas@gmail.com
The following bug has been logged on the website:
Bug reference: 19792
Logged by: Tomas Barton
Email address: barton.tomas@gmail.com
PostgreSQL version: 18.6
Operating system: Debian 12
Description:
PostgreSQL 18.6 (Debian 18.6-1.pgdg12+2), Debian 12, kernel 6.1.187,
io_method=io_uring.
io_method = io_uring
Storage: local ext4 on md RAID, no I/O errors in dmesg
Extensions in play: pg_stat_statements (no custom AMs; the affected table
has only btree indexes)
Three backends hung indefinitely and could not be cancelled or terminated
(`statement_timeout`, `pg_cancel_backend`, `pg_terminate_backend` all
ineffective,
as expected for LWLock / buffer-IO waits):
| pid | backend | wait event | what it is doing |
|---|---|---|---|
| 3689628 | client backend (A) | IPC:BufferIo | `INSERT ... ON CONFLICT DO
UPDATE`, in `_bt_split` -> `_bt_allocbuf` -> `ReadBuffer(block 628)` ->
`StartBufferIO` -> `WaitIO`, holding exclusive content lock on leaf block
627 |
| 2213934 | autovacuum worker (B) | LWLock:BufferContent | `btvacuumcleanup`
-> `btvacuumscan` -> `btvacuumpage` -> `_bt_lockbuf(block 627, BT_READ)` |
| 2214042 | client backend (C) | LWLock:BufferContent | backward index-only
scan (`MAX(last_update)`), `_bt_search` -> `_bt_relandgetbuf` ->
`_bt_lockbuf(block 627)` |
Block 627 and 628 both belong to the btree index
`last_update_similar_items_thenlper_gte_large_recent` (relfilenode
3378615556) on table
`"db-prod".similar_items_thenlper_gte_large_recent` (oid 3371211934). The
index is on a
timestamp column, so inserts land on the rightmost leaf (627) and split
frequently. Block 628
is a free page returned by the FSM.
## Mechanism
1. Backend B (autovacuum, no dead tuples, in the index-cleanup pass) walks
the index through a
read stream. The stream issued the read of block 628 as look-ahead.
`pg_aios` shows that I/O
as handle **17445**, generation **9035984**, owner pid **2213934**, state
**SUBMITTED**.
2. Backend A, splitting leaf 627 (exclusive content lock held), got block
628 from the FSM and
called `ReadBuffer`. `StartBufferIO` found `BM_IO_IN_PROGRESS` already
set by B. At that
moment B had set the flag but **had not yet staged the I/O**, so
`buf->io_wref` was still
invalid. The `WaitIO` frame of A confirms this: its local copy is
`iow = {aio_index = 4294967295, ...}`. A therefore took the fallback
branch and went to
sleep on the buffer's condition variable (`ConditionVariableSleep(cv,
WAIT_EVENT_BUFFER_IO)`).
3. B then staged and submitted the I/O. `buffer_stage_common` sets
`buf_hdr->io_wref` but does
**not** broadcast the buffer's I/O condition variable, so A is not woken.
The buffer header
now reads `io_wref = {aio_index = 17445, generation_lower = 9035984}`
while A still sleeps.
4. B proceeded to `_bt_lockbuf(627, BT_READ)` and blocked on the content
lock held by A.
5. Under io_uring, the completion of handle 17445 is only processed by its
owner (B) or by a
backend that waits on that wait reference (`pgaio_wref_wait`). B is
blocked on an LWLock and
A never re-reads `io_wref` (nothing wakes it). The completion sits in B's
ring forever.
Backend C and any future reader of block 627 queue behind A.
In `io_method=worker` the I/O worker runs the completion callback, which
calls
`TerminateBufferIO` and broadcasts the condition variable, so the same race
is harmless there.
The relevant code in REL_18_STABLE is unchanged since the original AIO
commits: `WaitIO()`
sleeps on the buffer CV when `io_wref` is invalid, and
`buffer_stage_common()` sets `io_wref`
without a broadcast. Master (post commits 74eafea / 999dec9, March 2026) has
the same fallback;
`AsyncReadBuffers()` there even comments: "It's possible that another
backend has started IO on
the buffer but not yet set its wait reference. In this case, we have no
choice but to wait for
either the wait reference to be valid or the IO to be done." Nothing signals
the first of those
two transitions.
Possibly related (different trigger, same family): "Potential deadlock in
pgaio_io_wait()",
Thomas Munro / Heikki Linnakangas, pgsql-hackers, Aug-Oct 2025
(https://www.postgresql.org/message-id/CA%2BhUKG%2BmZYrSdnhk-XrBYO18H829K77S9gMKUsykOiTJtqB43g%40mail...).
Also BUG #19725 (io_uring read failures on 18.6).
## Suggested fix
Either broadcast `BufferDescriptorGetIOCV(buf)` after setting
`buf_hdr->io_wref` in
`buffer_stage_common()` (so a waiter in `WaitIO()` re-checks and switches to
`pgaio_wref_wait()`), or make `WaitIO()` sleep with a timeout / re-check
loop when it observes
`BM_IO_IN_PROGRESS` with an invalid `io_wref`.
## Recovery
Fast shutdown cannot complete (the backends do not reach
`CHECK_FOR_INTERRUPTS`). Immediate
shutdown or SIGKILL of the INSERT backend, followed by crash recovery.
`io_method` was changed
to `worker` on restart.
---
# Evidence
## 1. pg_stat_activity (monitoring view, before diagnosis)
```
pid | duration | cpu | mem | state | wait_event
| client | user | query
---------+-----------------+-----+---------+--------+------------------------+-------------------+-------------+--------------------------------------------------------------
2213934 | 1d 18h 18m 19s | 0% | 3668 kB | active | LWLock:BufferContent
| autovacuum worker | | autovacuum: VACUUM
db-prod.similar_items_thenlper_...
3689628 | 1d 18h 18m 18s | 0% | 29 MB | active | IPC:BufferIo
| 10.0.3.1 | recommender | INSERT INTO
"db-prod"."similar_items_thenlper_gte_...
2214042 | 1d 18h 18m 3s | 0% | 3880 kB | active | LWLock:BufferContent
| 10.0.2.13 | recommender | SET LOCAL statement_timeout = 600000;
SELECT CURRENT_TIME...
```
## 2. OS state of the three processes (no D state, nothing pending in the
kernel)
```
PID STAT WCHAN CMD
2213934 Ss futex_wait_queue postgres: 18/main: autovacuum
worker recommender
2214042 Ss futex_wait_queue postgres: 18/main: recommender
recommender 10.0.3.1(43106) SELECT
3689628 Ssl do_epoll_wait postgres: 18/main: recommender
recommender 10.0.2.13(33290) INSERT
== /proc/2213934/stack
[<0>] futex_wait_queue+0x60/0x90
[<0>] futex_wait+0x185/0x270
[<0>] do_futex+0x106/0x1b0
[<0>] __x64_sys_futex+0x8e/0x1d0
[<0>] do_syscall_64+0x5d/0xb0
[<0>] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
== /proc/3689628/stack
[<0>] do_epoll_wait+0x698/0x7d0
[<0>] __x64_sys_epoll_wait+0x6f/0x110
[<0>] do_syscall_64+0x5d/0xb0
[<0>] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
== /proc/2214042/stack
[<0>] futex_wait_queue+0x60/0x90
[<0>] futex_wait+0x185/0x270
[<0>] do_futex+0x106/0x1b0
[<0>] __x64_sys_futex+0x8e/0x1d0
[<0>] do_syscall_64+0x5d/0xb0
[<0>] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
# threads of the INSERT backend (io_uring worker idle)
== /proc/3689628/task/1773077 iou-wrk-3689628 S
[<0>] io_wqe_worker+0x1b9/0x3a0
[<0>] ret_from_fork+0x22/0x30
== /proc/3689628/task/3689628 postgres S
[<0>] do_epoll_wait+0x698/0x7d0
[<0>] __x64_sys_epoll_wait+0x6f/0x110
[<0>] do_syscall_64+0x5d/0xb0
[<0>] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
```
dmesg contains nothing since a routine md data-check on 2026-09-24.
## 3. Settings
```
io_method
-----------
io_uring
io_workers
------------
6
```
## 4. Relations involved
```
oid | relname | relkind
| relfilenode
------------+-----------------------------------------------------+---------+-------------
3371211934 | similar_items_thenlper_gte_large_recent | r
| 3371211934
3378609811 | similar_items_thenlper_gte_large_recent_pkey | i
| 3378609811
3378615556 | last_update_similar_items_thenlper_gte_large_recent | i
| 3378615556
```
## 5. pg_stat_progress_vacuum for the autovacuum (pid 2213934)
```
pid | datid | datname | relid | phase |
heap_blks_total | heap_blks_scanned | heap_blks_vacuumed |
index_vacuum_count | max_dead_tuple_bytes | dead_tuple_bytes |
num_dead_item_ids | indexes_total | indexes_processed | delay_time
---------+-------+-------------+------------+---------------------+-----------------+-------------------+--------------------+--------------------+----------------------+------------------+-------------------+---------------+-------------------+------------
2213934 | 16386 | recommender | 3371211934 | cleaning up indexes |
1204577 | 1204577 | 1204577 | 0 |
2147483648 | 0 | 0 | 2 |
1 | 0
```
## 6. pg_aios: the I/O owned by the autovacuum that nobody completes
```
pid | io_id | io_generation | state | target_desc
---------+-------+---------------+-----------+-------------------------------------------
2213934 | 17445 | 9035984 | SUBMITTED | block 628 in file
"base/16386/3378615556"
```
The INSERT's own handle (acquired before `StartBufferIO`, never used):
```
pid | io_id | io_generation | state | operation | ... |
target_desc
---------+-------+---------------+------------+-----------+-----+-------------
3689628 | 10321 | 29264366 | HANDED_OUT | | |
```
(Other rows in `pg_aios` at the time were short-lived reads from unrelated
sessions; all of
those backends had exited by the next check.)
## 7. Buffer header of the buffer the INSERT waits on (block 628), read via
gdb from pid 3689628
```
(gdb) p *(BufferDesc *)0x7fab2868ad80
$1 = {tag = {spcOid = 1663, dbOid = 16386, relNumber = 3378615556, forkNum =
MAIN_FORKNUM, blockNum = 628},
buf_id = 994552, state = {value = 2248671235}, wait_backend_pgprocno =
-1, freeNext = -2,
io_wref = {aio_index = 17445, generation_upper = 0, generation_lower =
9035984},
content_lock = {tranche = 62, state = {value = 1073741824}, waiters =
{head = -1, tail = -1}}}
(gdb) p/x ((BufferDesc *)0x7fab2868ad80)->state.value
$2 = 0x86080003
```
Decoded: refcount 3, usagecount 2, flags = BM_PERMANENT | BM_IO_IN_PROGRESS
| BM_TAG_VALID.
Not BM_VALID, no BM_IO_ERROR. `io_wref` matches handle 17445 / generation
9035984 above.
Note the contrast with the INSERT's `WaitIO` frame below, whose copy of
`iow` has
`aio_index = 4294967295` (invalid): the reference became valid after A went
to sleep.
## 8. Backtrace of the INSERT backend (pid 3689628), `bt full`
Frames 0-20 with locals. The large `io_pages` scratch array in frame 10 and
the
`events` array in frame 39 are elided as noise.
```
#0 0x00007fb342f28ef3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x0000561a358ba97b in WaitEventSetWaitBlock (nevents=1,
occurred_events=0x7ffcb93c2b50, cur_timeout=-1, set=0x561a53fcb8f0) at
./build/../src/backend/storage/ipc/waiteventset.c:1191
#2 WaitEventSetWait (set=0x561a53fcb8f0, timeout=timeout@entry=-1,
occurred_events=occurred_events@entry=0x7ffcb93c2b50,
nevents=nevents@entry=1, wait_event_info=wait_event_info@entry=134217736) at
./build/../src/backend/storage/ipc/waiteventset.c:1139
#3 0x0000561a358af5e5 in WaitLatch (latch=<optimized out>,
wakeEvents=wakeEvents@entry=33, timeout=timeout@entry=-1,
wait_event_info=wait_event_info@entry=134217736) at
./build/../src/backend/storage/ipc/latch.c:196
#4 0x0000561a358bc215 in ConditionVariableTimedSleep
(cv=cv@entry=0x7fb30e72af00, timeout=timeout@entry=-1,
wait_event_info=wait_event_info@entry=134217736) at
./build/../src/backend/storage/lmgr/condition_variable.c:163
#5 0x0000561a358bc3be in ConditionVariableSleep
(cv=cv@entry=0x7fb30e72af00,
wait_event_info=wait_event_info@entry=134217736) at
./build/../src/backend/storage/lmgr/condition_variable.c:98
#6 0x0000561a3589c191 in WaitIO (buf=buf@entry=0x7fab2868ad80) at
./build/../src/backend/storage/buffer/bufmgr.c:6032
buf_state = <optimized out>
iow = {aio_index = 4294967295, generation_upper = 0,
generation_lower = 28473836}
cv = 0x7fb30e72af00
#7 0x0000561a3589d20d in StartBufferIO (buf=0x7fab2868ad80,
forInput=forInput@entry=true, nowait=nowait@entry=false) at
./build/../src/backend/storage/buffer/bufmgr.c:6076
#8 0x0000561a3589d3ac in ReadBuffersCanStartIOOnce (nowait=false,
buffer=<optimized out>) at
./build/../src/backend/storage/buffer/bufmgr.c:1566
#9 ReadBuffersCanStartIO (nowait=false, buffer=<optimized out>) at
./build/../src/backend/storage/buffer/bufmgr.c:1594
#10 AsyncReadBuffers (operation=operation@entry=0x7ffcb93c3230,
nblocks_progress=nblocks_progress@entry=0x7ffcb93c317c) at
./build/../src/backend/storage/buffer/bufmgr.c:1871
buffers = 0x7ffcb93c322c
flags = 8
forknum = MAIN_FORKNUM
persistence = 112 'p'
nblocks_done = 0
blocknum = 628
io_buffers = 0x7ffcb93c322c
io_buffers_len = 0
ioh = 0x7fb3363fd390
ioh_flags = 1
io_pages = {... elided ...}
io_context = IOCONTEXT_NORMAL
io_object = IOOBJECT_RELATION
did_start_io = <optimized out>
#11 0x0000561a3589fcac in StartReadBuffersImpl (allow_forwarding=false,
flags=8, nblocks=0x7ffcb93c317c, blockNum=628, buffers=0x7ffcb93c322c,
operation=0x7ffcb93c3230) at
./build/../src/backend/storage/buffer/bufmgr.c:1437
actual_nblocks = 1
maxcombine = 0
#12 StartReadBuffer (operation=operation@entry=0x7ffcb93c3230,
buffer=buffer@entry=0x7ffcb93c322c, blocknum=blocknum@entry=628,
flags=flags@entry=8) at ./build/../src/backend/storage/buffer/bufmgr.c:1525
nblocks = 1
#13 0x0000561a358a10dd in ReadBuffer_common (strategy=<optimized out>,
mode=<optimized out>, blockNum=<optimized out>, forkNum=<optimized out>,
smgr_persistence=<optimized out>, smgr=<optimized out>, rel=<optimized out>)
at ./build/../src/backend/storage/buffer/bufmgr.c:1255
operation = {rel = 0x7fab1e2e94e8, smgr = 0x561a541d2150,
persistence = 112 'p', forknum = MAIN_FORKNUM, strategy = 0x0, buffers =
0x7ffcb93c322c, blocknum = 628, flags = 8, nblocks = 1, nblocks_done = 0,
io_wref = {aio_index = 4294967295, generation_upper = 0, generation_lower =
0}, io_return = {...}}
buffer = 994553
flags = 8
#14 ReadBufferExtended (reln=<optimized out>,
forkNum=forkNum@entry=MAIN_FORKNUM, blockNum=628,
mode=mode@entry=RBM_NORMAL, strategy=strategy@entry=0x0) at
./build/../src/backend/storage/buffer/bufmgr.c:815
#15 0x0000561a358a15ee in ReadBuffer (reln=<optimized out>,
blockNum=<optimized out>) at
./build/../src/backend/storage/buffer/bufmgr.c:760
#16 0x0000561a355a157c in _bt_allocbuf (rel=rel@entry=0x7fab1e2e94e8,
heaprel=heaprel@entry=0x561a54175460) at
./build/../src/backend/access/nbtree/nbtpage.c:906
buf = <optimized out>
blkno = 628
page = <optimized out>
#17 0x0000561a3559e8f1 in _bt_split (postingoff=0, nposting=0x0,
orignewitem=0x0, newitem=0x561a54403018, newitemsz=16, newitemoff=8, cbuf=0,
buf=992779, itup_key=<optimized out>, heaprel=0x561a54175460,
rel=0x7fab1e2e94e8) at ./build/../src/backend/access/nbtree/nbtinsert.c:1720
origpagenumber = 627
maxoff = 7
firstrightoff = 6
afterleftoff = 2
isleaf = true
isrightmost = true
newitemonleft = false
sbuf = 0
spage = 0x0
origpage = 0x7fad19111000 "\035\353\n"
leftpage = 0x561a543ff248 "\035\353\n"
lefthighkey = 0x561a543fdb90
#18 _bt_insertonpg (rel=0x7fab1e2e94e8, heaprel=0x561a54175460,
itup_key=<optimized out>, buf=992779, cbuf=0, stack=0x561a543fd490,
itup=0x561a54403018, itemsz=<optimized out>, newitemoff=8,
postingoff=<optimized out>, split_only_page=<optimized out>) at
./build/../src/backend/access/nbtree/nbtinsert.c:1217
isroot = false
isrightmost = true
isonly = false
#19 0x0000561a355a0571 in _bt_doinsert (rel=rel@entry=0x7fab1e2e94e8,
itup=itup@entry=0x561a54403018,
checkUnique=checkUnique@entry=UNIQUE_CHECK_NO,
indexUnchanged=indexUnchanged@entry=false,
heapRel=heapRel@entry=0x561a54175460) at
./build/../src/backend/access/nbtree/nbtinsert.c:260
insertstate = {itup = 0x561a54403018, itemsz = 16, itup_key =
0x561a542f3320, buf = 992779, bounds_valid = true, low = 8, stricthigh = 8,
postingoff = 0}
#20 0x0000561a355a6b23 in btinsert (rel=0x7fab1e2e94e8, values=<optimized
out>, isnull=<optimized out>, ht_ctid=0x561a543fcad8,
heapRel=0x561a54175460, checkUnique=UNIQUE_CHECK_NO,
indexUnchanged=<optimized out>, indexInfo=<optimized out>) at
./build/../src/backend/access/nbtree/nbtree.c:215
#21 0x0000561a35707243 in ExecInsertIndexTuples
(resultRelInfo=resultRelInfo@entry=0x561a5427da50,
slot=slot@entry=0x561a543fcaa8, estate=<optimized out>,
update=update@entry=true, noDupErr=noDupErr@entry=false,
specConflict=specConflict@entry=0x0, arbiterIndexes=<optimized out>,
onlySummarizing=<optimized out>) at
./build/../src/backend/executor/execIndexing.c:449
#22 0x0000561a357336ee in ExecUpdateEpilogue
(context=context@entry=0x7ffcb93c3c80, updateCxt=<optimized out>,
resultRelInfo=resultRelInfo@entry=0x561a5427da50,
tupleid=tupleid@entry=0x7ffcb93c3baa, oldtuple=oldtuple@entry=0x0,
slot=slot@entry=0x561a543fcaa8) at
./build/../src/backend/executor/nodeModifyTable.c:2330
#23 0x0000561a35734f27 in ExecUpdate (context=context@entry=0x7ffcb93c3c80,
resultRelInfo=resultRelInfo@entry=0x561a5427da50,
tupleid=tupleid@entry=0x7ffcb93c3baa, oldtuple=oldtuple@entry=0x0,
oldSlot=<optimized out>, oldSlot@entry=0x561a5427f348, slot=0x561a543fcaa8,
canSetTag=<optimized out>) at
./build/../src/backend/executor/nodeModifyTable.c:2709
#24 0x0000561a35735c6e in ExecOnConflictUpdate (returning=<synthetic
pointer>, canSetTag=<optimized out>, excludedSlot=<optimized out>,
conflictTid=0x7ffcb93c3baa, resultRelInfo=0x561a5427da50,
context=0x7ffcb93c3c80) at
./build/../src/backend/executor/nodeModifyTable.c:2925
#25 ExecInsert (context=context@entry=0x7ffcb93c3c80,
resultRelInfo=resultRelInfo@entry=0x561a5427da50, slot=<optimized out>,
slot@entry=0x561a543fcd00, canSetTag=<optimized out>,
inserted_tuple=inserted_tuple@entry=0x0,
insert_destrel=insert_destrel@entry=0x0) at
./build/../src/backend/executor/nodeModifyTable.c:1152
#26 0x0000561a3573714e in ExecModifyTable (pstate=0x561a5427d848) at
./build/../src/backend/executor/nodeModifyTable.c:4509
#27 0x0000561a35707eda in ExecProcNode (node=0x561a5427d848) at
./build/../src/include/executor/executor.h:315
#28 ExecutePlan (dest=0x561a5400eed8, direction=<optimized out>,
numberTuples=0, sendTuples=false, operation=CMD_INSERT,
queryDesc=0x561a5429b2b8) at ./build/../src/backend/executor/execMain.c:1711
#29 standard_ExecutorRun (queryDesc=0x561a5429b2b8, direction=<optimized
out>, count=0) at ./build/../src/backend/executor/execMain.c:366
#30 0x00007fb342c434e5 in pgss_ExecutorRun (queryDesc=0x561a5429b2b8,
direction=ForwardScanDirection, count=0) at
./build/../contrib/pg_stat_statements/pg_stat_statements.c:1036
#31 0x0000561a358db264 in ProcessQuery (plan=0x561a558c8948,
sourceText=0x561a542fa9f0 "\n INSERT INTO
\"db-prod\".\"similar_items_thenlper_gte_large_recent\" (itemid,
similar_items, last_update)\n VALUES ('AmZHZVwxWbZ5', '[{\"id\":
\"A7GBDfQ8rMhD\", \"r"..., params=0x0, queryEnv=0x0, dest=0x561a5400eed8,
qc=0x7ffcb93c4110) at ./build/../src/backend/tcop/pquery.c:161
#32 0x0000561a358dbe79 in PortalRunMulti
(portal=portal@entry=0x561a540e4a48, isTopLevel=isTopLevel@entry=true,
setHoldSnapshot=setHoldSnapshot@entry=false, dest=dest@entry=0x561a5400eed8,
altdest=altdest@entry=0x561a5400eed8, qc=qc@entry=0x7ffcb93c4110) at
./build/../src/backend/tcop/pquery.c:1272
#33 0x0000561a358dc326 in PortalRun (portal=portal@entry=0x561a540e4a48,
count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true,
dest=dest@entry=0x561a5400eed8, altdest=altdest@entry=0x561a5400eed8,
qc=qc@entry=0x7ffcb93c4110) at ./build/../src/backend/tcop/pquery.c:788
#34 0x0000561a358d8189 in exec_simple_query (query_string=0x561a542fa9f0
"...INSERT INTO \"db-prod\".\"similar_items_thenlper_gte_large_recent\"
...") at ./build/../src/backend/tcop/postgres.c:1274
#35 0x0000561a358d9e2c in PostgresMain (dbname=<optimized out>,
username=<optimized out>) at ./build/../src/backend/tcop/postgres.c:4770
#36 0x0000561a358d46bf in BackendMain (startup_data=<optimized out>,
startup_data_len=<optimized out>) at
./build/../src/backend/tcop/backend_startup.c:124
#37 0x0000561a35835745 in postmaster_child_launch (child_type=B_BACKEND,
child_slot=20, ...) at
./build/../src/backend/postmaster/launch_backend.c:290
#38 0x0000561a35839039 in BackendStartup (client_sock=0x7ffcb93c4630) at
./build/../src/backend/postmaster/postmaster.c:3569
#39 ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1703
#40 0x0000561a3583abbd in PostmasterMain (argc=argc@entry=5,
argv=argv@entry=0x561a53fca7f0) at
./build/../src/backend/postmaster/postmaster.c:1401
#41 0x0000561a3553bb7c in main (argc=5, argv=0x561a53fca7f0) at
./build/../src/backend/main/main.c:227
```
Note: `buf=992779` in frames 17-19 is block 627 (held exclusively); `buffer
= 994553` in frame
13 is block 628 (the buffer being waited on, header dumped in section 7).
## 9. Backtrace of the autovacuum worker (pid 2213934)
```
#0 0x00007fb342ea5e96 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fb342eb0d90 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x0000561a3582df72 in PGSemaphoreLock (sema=0x7fab1e8dc838) at
./build/src/backend/port/pg_sema.c:326
#3 0x0000561a358c4c0d in LWLockAcquire (lock=0x7fab2866f230,
mode=LW_SHARED) at ./build/../src/backend/storage/lmgr/lwlock.c:1299
#4 0x0000561a355a1dc9 in _bt_lockbuf (rel=rel@entry=0x561a5415a430,
buf=buf@entry=992779, access=access@entry=1) at
./build/../src/backend/access/nbtree/nbtpage.c:1042
#5 0x0000561a355a732b in btvacuumpage (vstate=vstate@entry=0x7ffcb93c3720,
buf=992779) at ./build/../src/backend/access/nbtree/nbtree.c:1383
#6 0x0000561a355a7d98 in btvacuumscan (info=info@entry=0x7ffcb93c39e0,
stats=stats@entry=0x561a54105450, callback=callback@entry=0x0,
callback_state=callback_state@entry=0x0, cycleid=cycleid@entry=0) at
./build/../src/backend/access/nbtree/nbtree.c:1312
#7 0x0000561a355a80a8 in btvacuumcleanup (stats=0x561a54105450,
info=0x7ffcb93c39e0) at ./build/../src/backend/access/nbtree/nbtree.c:1138
#8 btvacuumcleanup (info=0x7ffcb93c39e0, stats=0x0) at
./build/../src/backend/access/nbtree/nbtree.c:1098
#9 0x0000561a356f3faf in vac_cleanup_one_index
(ivinfo=ivinfo@entry=0x7ffcb93c39e0, istat=istat@entry=0x0) at
./build/../src/backend/commands/vacuum.c:2674
#10 0x0000561a35598c99 in lazy_cleanup_one_index (vacrel=0x561a541049b0,
estimated_count=true, reltuples=2712141, istat=0x0, indrel=<optimized out>)
at ./build/../src/backend/access/heap/vacuumlazy.c:3173
#11 lazy_cleanup_all_indexes (vacrel=0x561a541049b0) at
./build/../src/backend/access/heap/vacuumlazy.c:3061
#12 lazy_scan_heap (vacrel=0x561a541049b0) at
./build/../src/backend/access/heap/vacuumlazy.c:1558
#13 heap_vacuum_rel (rel=<optimized out>, params=<optimized out>,
bstrategy=<optimized out>) at
./build/../src/backend/access/heap/vacuumlazy.c:839
#14 0x0000561a356f1afa in table_relation_vacuum (bstrategy=0x561a5418f9e8,
params=0x7ffcb93c3f80, rel=0x561a54156ff0) at
./build/../src/include/access/tableam.h:1677
#15 vacuum_rel (relid=3371211934, relation=<optimized out>,
params=params@entry=0x7ffcb93c3f80,
bstrategy=bstrategy@entry=0x561a5418f9e8) at
./build/../src/backend/commands/vacuum.c:2317
#16 0x0000561a356f2f4a in vacuum (relations=0x561a5419a7d8,
params=params@entry=0x561a54198810,
bstrategy=bstrategy@entry=0x561a5418f9e8,
vac_context=vac_context@entry=0x561a5419a5e0,
isTopLevel=isTopLevel@entry=true) at
./build/../src/backend/commands/vacuum.c:649
#17 0x0000561a35831c7f in autovacuum_do_vac_analyze
(bstrategy=0x561a5418f9e8, tab=0x561a54198808) at
./build/../src/backend/postmaster/autovacuum.c:3196
#18 do_autovacuum () at ./build/../src/backend/postmaster/autovacuum.c:2457
#19 0x0000561a3583224a in AutoVacWorkerMain (startup_data=<optimized out>,
startup_data_len=<optimized out>) at
./build/../src/backend/postmaster/autovacuum.c:1590
#20 0x0000561a35835745 in postmaster_child_launch
(child_type=child_type@entry=B_AUTOVAC_WORKER, child_slot=560, ...) at
./build/../src/backend/postmaster/launch_backend.c:290
#21 0x0000561a358377bd in StartChildProcess (type=B_AUTOVAC_WORKER) at
./build/../src/backend/postmaster/postmaster.c:3962
#22 0x0000561a35839ac8 in StartAutovacuumWorker () at
./build/../src/backend/postmaster/postmaster.c:4026
#23 process_pm_pmsignal () at
./build/../src/backend/postmaster/postmaster.c:3783
#24 ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1696
#25 0x0000561a3583abbd in PostmasterMain (argc=argc@entry=5,
argv=argv@entry=0x561a53fca7f0) at
./build/../src/backend/postmaster/postmaster.c:1401
#26 0x0000561a3553bb7c in main (argc=5, argv=0x561a53fca7f0) at
./build/../src/backend/main/main.c:227
```
`buf=992779` is block 627, the leaf held exclusively by the INSERT. The read
stream feeding
`btvacuumscan` had already issued block 628 (handle 17445, section 6).
## 10. Backtrace of the SELECT backend (pid 2214042), collateral
```
#0 0x00007fb342ea5e96 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fb342eb0d90 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x0000561a3582df72 in PGSemaphoreLock (sema=0x7fab1e8d7938) at
./build/src/backend/port/pg_sema.c:326
#3 0x0000561a358c4c0d in LWLockAcquire (lock=0x7fab2866f230,
mode=LW_SHARED) at ./build/../src/backend/storage/lmgr/lwlock.c:1299
#4 0x0000561a355a1d5f in _bt_lockbuf (access=1, buf=992779,
rel=0x561a5418f170) at ./build/../src/backend/access/nbtree/nbtpage.c:1042
#5 _bt_relandgetbuf (rel=rel@entry=0x561a5418f170, obuf=<optimized out>,
blkno=<optimized out>, access=access@entry=1) at
./build/../src/backend/access/nbtree/nbtpage.c:1011
#6 0x0000561a355a9f68 in _bt_search (access=<optimized out>,
bufP=<optimized out>, key=<optimized out>, heaprel=<optimized out>,
rel=<optimized out>) at ./build/../src/backend/access/nbtree/nbtsearch.c:186
#7 _bt_search (rel=0x561a5418f170, heaprel=<optimized out>, key=<optimized
out>, bufP=0x561a541f6808, access=<optimized out>) at
./build/../src/backend/access/nbtree/nbtsearch.c:107
#8 0x0000561a355aa791 in _bt_first (scan=scan@entry=0x561a541f32c8,
dir=dir@entry=BackwardScanDirection) at
./build/../src/backend/access/nbtree/nbtsearch.c:1520
#9 0x0000561a355a6bcf in btgettuple (scan=0x561a541f32c8,
dir=BackwardScanDirection) at
./build/../src/backend/access/nbtree/nbtree.c:245
#10 0x0000561a3559b739 in index_getnext_tid (scan=scan@entry=0x561a541f32c8,
direction=direction@entry=BackwardScanDirection) at
./build/../src/backend/access/index/indexam.c:637
#11 0x0000561a3572bfd4 in IndexOnlyNext (node=0x561a541d8ed0) at
./build/../src/backend/executor/nodeIndexonlyscan.c:122
#12 0x0000561a3572edc9 in ExecProcNode (node=0x561a541d8ed0) at
./build/../src/include/executor/executor.h:315
#13 ExecLimit (pstate=0x561a541d8bc0) at
./build/../src/backend/executor/nodeLimit.c:95
#14 0x0000561a3573da1d in ExecProcNode (node=<optimized out>) at
./build/../src/include/executor/executor.h:315
#15 ExecSetParamPlan (node=0x561a541da6d0,
econtext=econtext@entry=0x561a541da230) at
./build/../src/backend/executor/nodeSubplan.c:1125
#16 0x0000561a35704c7a in ExecEvalParamExec (econtext=0x561a541da230,
op=<optimized out>, state=0x561a541da5d0) at
./build/../src/backend/executor/execExprInterp.c:3064
#17 ExecInterpExpr (state=0x561a541da5d0, econtext=0x561a541da230,
isnull=<optimized out>) at
./build/../src/backend/executor/execExprInterp.c:1322
#18 0x0000561a3573a1f5 in ExecEvalExprNoReturn (econtext=0x561a541da230,
state=0x561a541da5d0) at ./build/../src/include/executor/executor.h:419
#19 ExecEvalExprNoReturnSwitchContext (econtext=0x561a541da230,
state=0x561a541da5d0) at ./build/../src/include/executor/executor.h:460
#20 ExecProject (projInfo=0x561a541da5c8) at
./build/../src/include/executor/executor.h:492
#21 ExecResult (pstate=<optimized out>) at
./build/../src/backend/executor/nodeResult.c:135
#22 0x0000561a35707eda in ExecProcNode (node=0x561a541da128) at
./build/../src/include/executor/executor.h:315
#23 ExecutePlan (dest=0x561a541de868, direction=<optimized out>,
numberTuples=0, sendTuples=true, operation=CMD_SELECT,
queryDesc=0x561a541d8508) at ./build/../src/backend/executor/execMain.c:1711
#24 standard_ExecutorRun (queryDesc=0x561a541d8508, direction=<optimized
out>, count=0) at ./build/../src/backend/executor/execMain.c:366
#25 0x00007fb342c434e5 in pgss_ExecutorRun (queryDesc=0x561a541d8508,
direction=ForwardScanDirection, count=0) at
./build/../contrib/pg_stat_statements/pg_stat_statements.c:1036
#26 0x0000561a358dad97 in PortalRunSelect
(portal=portal@entry=0x561a540efec8, forward=forward@entry=true, count=0,
count@entry=9223372036854775807, dest=dest@entry=0x561a541de868) at
./build/../src/backend/tcop/pquery.c:921
#27 0x0000561a358dc1fb in PortalRun (portal=portal@entry=0x561a540efec8,
count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true,
dest=dest@entry=0x561a541de868, altdest=altdest@entry=0x561a541de868,
qc=qc@entry=0x7ffcb93c4110) at ./build/../src/backend/tcop/pquery.c:765
#28 0x0000561a358d8189 in exec_simple_query (query_string=0x561a5400d0a8
"SET LOCAL statement_timeout = 600000; \n SELECT\n CURRENT_TIMESTAMP,\n
MAX(last_update) as last_update_timestamp,\n (CURRENT_TIMESTAMP -
MAX(last_update))"...) at ./build/../src/backend/tcop/postgres.c:1274
#29 0x0000561a358d9e2c in PostgresMain (dbname=<optimized out>,
username=<optimized out>) at ./build/../src/backend/tcop/postgres.c:4770
#30 0x0000561a358d46bf in BackendMain (startup_data=<optimized out>,
startup_data_len=<optimized out>) at
./build/../src/backend/tcop/backend_startup.c:124
#31 0x0000561a35835745 in postmaster_child_launch (child_type=B_BACKEND,
child_slot=36, ...) at
./build/../src/backend/postmaster/launch_backend.c:290
#32 0x0000561a35839039 in BackendStartup (client_sock=0x7ffcb93c4630) at
./build/../src/backend/postmaster/postmaster.c:3569
#33 ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1703
#34 0x0000561a3583abbd in PostmasterMain (argc=argc@entry=5,
argv=argv@entry=0x561a53fca7f0) at
./build/../src/backend/postmaster/postmaster.c:1401
#35 0x0000561a3553bb7c in main (argc=5, argv=0x561a53fca7f0) at
./build/../src/backend/main/main.c:227
```
Same LWLock address (0x7fab2866f230) and same buffer (992779 = block 627) as
the autovacuum.
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6)
2026-10-23 13:46 BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6) PG Bug reporting form <noreply@postgresql.org>
@ 2026-10-25 18:31 ` shihao zhong <zhong950419@gmail.com>
2026-10-28 07:30 ` Re: BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6) Tomas Barton <barton.tomas@gmail.com>
0 siblings, 1 reply; 4+ messages in thread
From: shihao zhong @ 2026-10-25 18:31 UTC (permalink / raw)
To: barton.tomas@gmail.com; +Cc: pgsql-bugs@lists.postgresql.org
Hi Thomas,
Thanks for the report.
I'm starting to learn AIO and have no Linux machine for io_uring, so I
could not
reproduce this with a real workload. From the code, I think it goes
like this.
t1 A: an INSERT splitting leaf 627 holds its lock.
t2 B: autovacuum's btree scan starts a look-ahead read of 628.
BM_IO_IN_PROGRESS is set, but io_wref is not yet.
t3 A: gets 628 from the FSM. WaitIO() sees no io_wref, so it
sleeps on the buffer CV.
t4 B: stages the IO and sets io_wref, but does not wake A.
t5 B: waits for the lock on 627 that A holds.
t6 A and B wait on each other forever.
With io_uring, only the owner or a backend that waits through io_wref
reaps the completion, so nobody reaps it. [1] only fixes waits
through io_wref, and A never gets there.
0001 is Thomas Munro's patch from [1]. A woken A can find the IO
still staged, and waits on submit_cv.
0002 wakes the buffer CV once io_wref is set. It also wakes
submit_cv after synchronous execution, which 0001 misses.
Draft attached. Could someone check them?
[1]
https://www.postgresql.org/message-id/CA%2BhUKG%2BmZYrSdnhk-XrBYO18H829K77S9gMKUsykOiTJtqB43g%40mail...
Thanks,
Shihao
Attachments: download all (2 files, .tar.gz)
v3-0001-aio-Fix-pgaio_io_wait-for-staged-IOs.patch (application/octet-stream, 5.1K) download
From ddc0e58d3e6c694e2c83cf2de8d529a1bb041996 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sun, 3 Aug 2025 23:07:56 +1200
Subject: [PATCH v3 1/2] aio: Fix pgaio_io_wait() for staged IOs.
Previously, pgaio_io_wait()'s cases for PGAIO_HS_DEFINED and
PGAIO_HS_STAGED fell through to waiting for completion. The owner only
promises to advance it to PGAIO_HS_SUBMITTED. The waiter needs to be
prepared to call ->wait_one() itself once the IO is submitted in order
to guarantee progress and avoid deadlocks on IO methods that provide
->wait_one().
Introduce a new per-backend condition variable submit_cv, woken by by
pgaio_submit_stage(), and use it to wait for the state to advance. The
new broadcast doesn't seem to cause any measurable slowdown, so ideas
for optimizing the common no-waiters case were abandoned for now.
It may not be possible to reach any real deadlock with existing AIO
users, but that situation could change. There's also no reason the
waiter shouldn't begin to wait via the IO method as soon as possible
even without a deadlock.
Picked up by testing a proposed IO method that has ->wait_one(), like
io_method=io_uring, and code review.
Backpatch-through: 18
Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/CA%2BhUKG%2BmZYrSdnhk-XrBYO18H829K77S9gMKUsykOiTJtqB43g%40mail.gmail.com
---
src/backend/storage/aio/aio.c | 26 ++++++++++++++++---
src/backend/storage/aio/aio_init.c | 1 +
.../utils/activity/wait_event_names.txt | 1 +
src/include/storage/aio_internal.h | 7 +++++
4 files changed, 32 insertions(+), 3 deletions(-)
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 8f7e26607b9..9aa118a7592 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -610,6 +610,23 @@ pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
elog(ERROR, "IO in wrong state: %d", state);
break;
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+
+ /*
+ * The owner hasn't submitted the IO yet. Wait for it to do
+ * so. Only then can this backend wait via the IO method if
+ * required, or otherwise ioh->cv.
+ */
+ ConditionVariablePrepareToSleep(&pgaio_ctl->backend_state[ioh->owner_procno].submit_cv);
+ while (!pgaio_io_was_recycled(ioh, ref_generation, &state) &&
+ (state == PGAIO_HS_DEFINED ||
+ state == PGAIO_HS_STAGED))
+ ConditionVariableSleep(&pgaio_ctl->backend_state[ioh->owner_procno].submit_cv,
+ WAIT_EVENT_AIO_IO_SUBMIT);
+ ConditionVariableCancelSleep();
+ continue;
+
case PGAIO_HS_SUBMITTED:
/*
@@ -624,9 +641,6 @@ pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
}
pg_fallthrough;
- /* waiting for owner to submit */
- case PGAIO_HS_DEFINED:
- case PGAIO_HS_STAGED:
/* waiting for reaper to complete */
/* fallthrough */
case PGAIO_HS_COMPLETED_IO:
@@ -1152,6 +1166,12 @@ pgaio_submit_staged(void)
pgaio_my_backend->num_staged_ios = 0;
+ /*
+ * Wake any backend that started waiting for any of these IOs before
+ * submission, so that it can call ->wait_one() if necessary.
+ */
+ ConditionVariableBroadcast(&pgaio_my_backend->submit_cv);
+
pgaio_debug(DEBUG4,
"aio: submitted %d IOs",
total_submitted);
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index de50e6a8a31..88662941ef4 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -199,6 +199,7 @@ AioShmemInit(void *arg)
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
+ ConditionVariableInit(&bs->submit_cv);
dclist_init(&bs->in_flight_ios);
/* initialize per-backend IOs */
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3d366fd1114..f8a939fafc6 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -201,6 +201,7 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
AIO_IO_COMPLETION "Waiting for another process to complete IO."
+AIO_IO_SUBMIT "Waiting for another process to submit IO."
AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
AIO_IO_URING_EXECUTION "Waiting for IO execution via io_uring."
BASEBACKUP_READ "Waiting for base backup to read from a file."
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 9ca4087aa7f..c3afde4d58f 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -216,6 +216,13 @@ typedef struct PgAioBackend
uint16 num_staged_ios;
PgAioHandle *staged_ios[PGAIO_SUBMIT_BATCH_SIZE];
+ /*
+ * Other backends sometimes need to wait for the owning backend to submit.
+ * The per-IO CV would work for that purpose, but a per-backend CV allows
+ * for just one broadcast per submitted batch.
+ */
+ ConditionVariable submit_cv;
+
/*
* List of in-flight IOs. Also contains IOs that aren't strictly speaking
* in-flight anymore, but have been waited-for and completed by another
--
2.37.1 (Apple Git-137.1)
v3-0002-aio-Fix-lost-wakeups-when-waiting-for-another-bac.patch (application/octet-stream, 2.1K) download
From 6403b444c11ceec91582e05ae412e029724aaa1a Mon Sep 17 00:00:00 2001
From: shihao zhong <zhong950419@gmail.com>
Date: Sun, 25 Oct 2026 14:24:19 -0400
Subject: [PATCH v3 2/2] aio: Fix lost wakeups when waiting for another
backend's buffer IO.
A backend that saw BM_IO_IN_PROGRESS before io_wref was set slept on
the buffer CV and was not woken once io_wref was set. With io_uring
the IO could then never be reaped. Also wake submit_cv waiters after
synchronous IO execution, which skips pgaio_submit_staged().
Reported-by: Tomas Barton <barton.tomas@gmail.com>
Backpatch-through: 18
Discussion: https://postgr.es/m/19792-9da3b35b8f4b3e60%40postgresql.org
---
src/backend/storage/aio/aio.c | 3 +++
src/backend/storage/buffer/bufmgr.c | 9 +++++++++
2 files changed, 12 insertions(+)
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 9aa118a7592..bb823f00152 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -474,6 +474,9 @@ pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
{
pgaio_io_prepare_submit(ioh);
pgaio_io_perform_synchronously(ioh);
+
+ /* this IO never goes through pgaio_submit_staged() */
+ ConditionVariableBroadcast(&pgaio_my_backend->submit_cv);
}
RESUME_INTERRUPTS();
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index c17fc3590f0..a21a3926bf0 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -8410,8 +8410,17 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state);
}
else
+ {
UnlockBufHdrExt(buf_hdr, buf_state, 0, 0, 1);
+ /*
+ * WaitIO() may have seen BM_IO_IN_PROGRESS before io_wref was
+ * set, and gone to sleep on the CV. Wake it up so it can wait
+ * via io_wref, instead of depending on us to complete the IO.
+ */
+ ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf_hdr));
+ }
+
/*
* Ensure the content lock that prevents buffer modifications while
* the buffer is being written out is not released early due to an
--
2.37.1 (Apple Git-137.1)
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6)
2026-10-23 13:46 BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6) PG Bug reporting form <noreply@postgresql.org>
2026-10-25 18:31 ` Re: BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6) shihao zhong <zhong950419@gmail.com>
@ 2026-10-28 07:30 ` Tomas Barton <barton.tomas@gmail.com>
2026-10-28 13:58 ` Re: BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6) shihao zhong <zhong950419@gmail.com>
0 siblings, 1 reply; 4+ messages in thread
From: Tomas Barton @ 2026-10-28 07:30 UTC (permalink / raw)
To: shihao zhong <zhong950419@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
Hi Shihao,
Thanks for looking into this and for the patch, looks promising! I'm happy
to test it, however it might be complicated to verify this.
We've been running workload on io_uring for 4 months without any issue. So
far the issue appeared on 2 independent clusters,
with a fairly heavy I/O workload (lots of autovacuum). After restarting the
postgresql process (and db recovery) the cluster continued
to run workload for months (with the same io_method = io_uring) without
hitting this bug.
Regards,
Tomas
On Sun, 25 Oct 2026 at 19:31, shihao zhong <zhong950419@gmail.com> wrote:
> Hi Thomas,
>
> Thanks for the report.
>
> I'm starting to learn AIO and have no Linux machine for io_uring, so I
> could not
> reproduce this with a real workload. From the code, I think it goes
> like this.
>
> t1 A: an INSERT splitting leaf 627 holds its lock.
> t2 B: autovacuum's btree scan starts a look-ahead read of 628.
> BM_IO_IN_PROGRESS is set, but io_wref is not yet.
> t3 A: gets 628 from the FSM. WaitIO() sees no io_wref, so it
> sleeps on the buffer CV.
> t4 B: stages the IO and sets io_wref, but does not wake A.
> t5 B: waits for the lock on 627 that A holds.
> t6 A and B wait on each other forever.
>
> With io_uring, only the owner or a backend that waits through io_wref
> reaps the completion, so nobody reaps it. [1] only fixes waits
> through io_wref, and A never gets there.
>
> 0001 is Thomas Munro's patch from [1]. A woken A can find the IO
> still staged, and waits on submit_cv.
> 0002 wakes the buffer CV once io_wref is set. It also wakes
> submit_cv after synchronous execution, which 0001 misses.
>
> Draft attached. Could someone check them?
>
> [1]
> https://www.postgresql.org/message-id/CA%2BhUKG%2BmZYrSdnhk-XrBYO18H829K77S9gMKUsykOiTJtqB43g%40mail...
>
> Thanks,
> Shihao
>
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6)
2026-10-23 13:46 BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6) PG Bug reporting form <noreply@postgresql.org>
2026-10-25 18:31 ` Re: BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6) shihao zhong <zhong950419@gmail.com>
2026-10-28 07:30 ` Re: BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6) Tomas Barton <barton.tomas@gmail.com>
@ 2026-10-28 13:58 ` shihao zhong <zhong950419@gmail.com>
0 siblings, 0 replies; 4+ messages in thread
From: shihao zhong @ 2026-10-28 13:58 UTC (permalink / raw)
To: Tomas Barton <barton.tomas@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
Hi Tomas,
Thanks for getting back to me, and sorry I mistyped your name in the first
email.
Let me try to build a spec for this case tonight to see if we can easily
reproduce it, also I
will get a Linux VM to do the experiment.
Thanks,
Shihao
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2026-10-28 13:58 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-23 13:46 BUG #19792: BUG: Deadlock between WaitIO and read-stream look-ahead under io_method=io_uring (18.6) PG Bug reporting form <noreply@postgresql.org>
2026-10-25 18:31 ` shihao zhong <zhong950419@gmail.com>
2026-10-28 07:30 ` Tomas Barton <barton.tomas@gmail.com>
2026-10-28 13:58 ` shihao zhong <zhong950419@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox