pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker
7+ messages / 3 participants
[nested] [flat]
* BUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker
@ 2026-09-28 03:34 PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 7+ messages in thread
From: PG Bug reporting form @ 2026-09-28 03:34 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: weijie1006jl@gmail.com
The following bug has been logged on the website:
Bug reference: 19725
Logged by: weijie JL
Email address: weijie1006jl@gmail.com
PostgreSQL version: 18.6
Operating system: RockyLinux9
Description:
Hello,
I can reproduce a read failure during pg_restore with io_method=io_uring.
The error disappears with worker and recurs after switching back to
io_uring.
Environment:
- PostgreSQL 18.6, x86_64
- VMware VM: 8 vCPUs, 32 GB RAM
- Kernel: 5.14.0-611.55.1.el9_7.x86_64
Settings:
effective_io_concurrency = 256
maintenance_io_concurrency = 16
max_parallel_maintenance_workers = 2
Restore command:
nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &
One failing statement:
ALTER TABLE ONLY dhr.obj_permission_object
ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);
Server error:
ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
Operation canceled
LOCATION: md_readv_report, md.c:2059
SQLSTATE: XX000
Observed test sequence:
1. io_method=io_uring: the read error occurred during index creation.
2. Changed io_method to worker: retried without errors.
3. Changed io_method back to io_uring: the read error was reproduced.
I have not measured the failure rate or prepared a self-contained
minimal reproducer.
One earlier run also reported "could not map dynamic shared memory
segment" from parallel workers immediately after the read error.
Those DSM errors have not recurred in subsequent failing tests.
Does this match a known PostgreSQL bug or kernel-side io_uring issue?
What additional logs or tracing would help identify the cause?
Thank you.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: BUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker
@ 2026-10-04 09:51 weijie JL <weijie1006jl@gmail.com>
parent: PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 7+ messages in thread
From: weijie JL @ 2026-10-04 09:51 UTC (permalink / raw)
To: weijie1006jl@gmail.com; pgsql-bugs@lists.postgresql.org
After setting backtrace_functions=md_readv_report, I received more log information:
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG: XX000: could not read blocks 143..158 in file "base/6406098/6407866": Operation canceled
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOCATION: md_readv_report, md.c:2059
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]BACKTRACE:
postgres: postgres test001 [local] CREATE INDEX() [0x92eafc]
postgres: postgres test001 [local] CREATE INDEX() [0x92f7ec]
postgres: postgres test001 [local] CREATE INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
postgres: postgres test001 [local] CREATE INDEX() [0x8f209c]
postgres: postgres test001 [local] CREATE INDEX() [0x8f2b63]
postgres: postgres test001 [local] CREATE INDEX() [0x8f0127]
postgres: postgres test001 [local] CREATE INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
postgres: postgres test001 [local] CREATE INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
postgres: postgres test001 [local] CREATE INDEX() [0x5874d6]
postgres: postgres test001 [local] CREATE INDEX(heap_getnext+0x88) [0x5878e8]
postgres: postgres test001 [local] CREATE INDEX() [0x58dcbd]
postgres: postgres test001 [local] CREATE INDEX() [0x5b11ae]
postgres: postgres test001 [local] CREATE INDEX(btbuild+0xb05) [0x5b7325]
postgres: postgres test001 [local] CREATE INDEX(index_build+0x19c) [0x61fa2c]
postgres: postgres test001 [local] CREATE INDEX(index_create+0x1328) [0x621608]
postgres: postgres test001 [local] CREATE INDEX(DefineIndex+0x9bb) [0x6d11cb]
postgres: postgres test001 [local] CREATE INDEX() [0xb18111]
postgres: postgres test001 [local] CREATE INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
/usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
postgres: postgres test001 [local] CREATE INDEX() [0x93cc3a]
postgres: postgres test001 [local] CREATE INDEX() [0x93cdaf]
postgres: postgres test001 [local] CREATE INDEX(PortalRun+0x291) [0x93d371]
postgres: postgres test001 [local] CREATE INDEX() [0x935ab0]
postgres: postgres test001 [local] CREATE INDEX(PostgresMain+0x1f1b) [0x937f6b]
postgres: postgres test001 [local] CREATE INDEX() [0x9389d5]
postgres: postgres test001 [local] CREATE INDEX(postmaster_child_launch+0x12d) [0x888ced]
postgres: postgres test001 [local] CREATE INDEX() [0x891cb2]
postgres: postgres test001 [local] CREATE INDEX(PostmasterMain+0xce9) [0x894309]
postgres: postgres test001 [local] CREATE INDEX(main+0x1d0) [0x531bc0]
/lib64/libc.so.6(+0x2a610) [0x7f209b82a610]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0]
postgres: postgres test001 [local] CREATE INDEX(_start+0x25) [0x5321b5]
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]STATEMENT: CREATE INDEX idx_sys_notification_channel_notification_id ON dhr.sys_notification_channel USING btree (c_notification_id);
> 2026年9月28日 11:34,PG Bug reporting form <noreply@postgresql.org> 写道:
>
> The following bug has been logged on the website:
>
> Bug reference: 19725
> Logged by: weijie JL
> Email address: weijie1006jl@gmail.com
> PostgreSQL version: 18.6
> Operating system: RockyLinux9
> Description:
>
> Hello,
>
> I can reproduce a read failure during pg_restore with io_method=io_uring.
> The error disappears with worker and recurs after switching back to
> io_uring.
>
> Environment:
> - PostgreSQL 18.6, x86_64
> - VMware VM: 8 vCPUs, 32 GB RAM
> - Kernel: 5.14.0-611.55.1.el9_7.x86_64
>
> Settings:
> effective_io_concurrency = 256
> maintenance_io_concurrency = 16
> max_parallel_maintenance_workers = 2
>
> Restore command:
> nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &
>
> One failing statement:
> ALTER TABLE ONLY dhr.obj_permission_object
> ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);
>
> Server error:
> ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
> Operation canceled
> LOCATION: md_readv_report, md.c:2059
> SQLSTATE: XX000
>
> Observed test sequence:
> 1. io_method=io_uring: the read error occurred during index creation.
> 2. Changed io_method to worker: retried without errors.
> 3. Changed io_method back to io_uring: the read error was reproduced.
>
> I have not measured the failure rate or prepared a self-contained
> minimal reproducer.
>
> One earlier run also reported "could not map dynamic shared memory
> segment" from parallel workers immediately after the read error.
> Those DSM errors have not recurred in subsequent failing tests.
>
> Does this match a known PostgreSQL bug or kernel-side io_uring issue?
> What additional logs or tracing would help identify the cause?
>
> Thank you.
>
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: BUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker
@ 2026-10-21 17:44 Dmytro Astapov <dastapov@gmail.com>
parent: weijie JL <weijie1006jl@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: Dmytro Astapov @ 2026-10-21 17:44 UTC (permalink / raw)
To: weijie JL <weijie1006jl@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
Hi!
I can reproduce this on version 18.6 + kernel 6.12.109.el8, and I think
that I have the explanation for how it happens + proposal for the fix as
well.
*What I think is happening*: postgres io_uring requests hit the kernel
codepath where new io workers are spawned up to RLIMIT_NPRPOC. Once this
limit is hit, no new workers can't be spawned, but also postgresmaster
can't fork() or system() anymore, leading to "could not read blocks X..Y in
file ...: Operation canceled"
*How it could be fixed*: postgres should either call
io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
*Detailed description of the chain of events follows:*
*1. Postgres asks io_uring to send buffered reads to worker threads*
In method_io_uring.c (line 441), once a backend already has more than 4
I/Os in flight, every further buffered read gets IOSQE_ASYNC:
if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
If `maintenance_io_concurrency` is higher that 4, then all of the
maintenance reads starting from 5th will take this path.
*2. The kernel places these reads into the pool whose only cap is
RLIMIT_NPROC*
Each task's io_uring worker pool has two halves:
- bounded (meant for regular files), capped at min(sq_entries, 4*ncpu);
- unbound, capped only at the task's RLIMIT_NPROC (io-wq.c).
Requests are classified in io_prep_async_work (
https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L480-L492):
if (req->file && (req->flags & REQ_F_ISREG)) { ... bound ... }
else if (!req->file || !S_ISBLK(...)) {
if (def->unbound_nonreg_file) // set for READ/READV
atomic_or(IO_WQ_WORK_UNBOUND, ...);
}
However, io_init_req sets req->file = NULL (at
https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L2046),
and the file is only attached later, at issue time. An IOSQE_ASYNC request
goes straight though io_queue_iowq() to io_prep_async_work() while
`req->file` is still NULL.
So a read of a regular file ends up routed to the unbound pool, whose cap
is RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also marks the work
as IO_WQ_WORK_CONCURRENT, which lets each enqueue create a worker instead
of waiting for a running one.
*3. Workers get created far faster than the actual I/O needs*
In io_wq_dec_running, when a worker blocks and it was the last running
worker while work is still queued, the kernel queues creation of a
brand-new worker immediately.
It doesn't wake one of the idle workers. On a cold buffer cache almost
every read blocks on disk, so this code path fires constantly. As a result,
the new worker handles one item and then goes idle. Only the head of the
free list gets reused, so the rest of the workers sit idle until
WORKER_IDLE_TIMEOUT = 5s expires and then they exit.
As a result, the steady-state thread count is therefore roughly "creation
rate × 5 s", not "I/Os in flight". The cap for the workers, as was
mentioned earlier, is RLIMIT_NPROC.
*Similar threads or bugs from the past*
-
Justin Pryzby on pgsql-hackers
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg208462.html;
(2025-10-17)
reported could not read blocks 122..126 ...: Operation canceled from
pg_dump under io_uring. He guessed it might be related to the io_uring
file descriptors leaking after crash-restarts, but I cannot find a
resolution.
-
Alexander Lakhin's AIO stress test
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196158.html;
(pgsql-hackers,
April 2025, before 18.0) hit could not read blocks 0..0 in file
"global/1213": Operation canceled with parallel createdb/dropdb jobs, on
kernels 6.8 and 6.11. The bugs fixed later in that thread (1
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196804.html;
, 2
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg197025.html;)
were different ones, a pg_aios crash and a PANIC from interrupt
handling. As far as I can see, the ECANCELED was never explained or fixed.
io_uring bugs that look relevant to me
-
liburing #420 <https://github.com/axboe/liburing/issues/420; (2021): a
user found IOSQE_ASYNC "spawns a thread for each request, meaning 10000+
threads". There is a commen there that bounded work is limited but
unbounded work "really [is] not". That discussion led to creation of
IORING_REGISTER_IOWQ_MAX_WORKERS.
-
liburing #349 <https://github.com/axboe/liburing/issues/349; (2021)
reported "unexpected high count of io_worker threads by using the
IOSQE_ASYNC flag".
-
Cloudflare, "Missing Manuals – io_uring worker pool"
<https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/; explains
that unbound workers default to RLIMIT_NPROC for each thread, and that
raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS can
backfire. The io_uring_register_iowq_max_workers(3)
<https://man7.org/linux/man-pages/man3/io_uring_register_iowq_max_workers.3.html;
man
page says the same.
So, in conclusion, it looks to me that the issue is that req->file is still
NULL when an IOSQE_ASYNC request is classified, and the reads end up in the
unbound pool, and the fix is either to call
io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
Best regards, Dmytro
On Sun, Oct 4, 2026 at 10:51 AM weijie JL <weijie1006jl@gmail.com> wrote:
> After setting backtrace_functions=md_readv_report, I received more log
> information:
> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG: XX000:
> could not read blocks 143..158 in file "base/6406098/6407866": Operation
> canceled
> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOCATION:
> md_readv_report, md.c:2059
> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]BACKTRACE:
> postgres: postgres test001 [local] CREATE INDEX() [0x92eafc]
> postgres: postgres test001 [local] CREATE INDEX() [0x92f7ec]
> postgres: postgres test001 [local] CREATE
> INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
> postgres: postgres test001 [local] CREATE INDEX() [0x8f209c]
> postgres: postgres test001 [local] CREATE INDEX() [0x8f2b63]
> postgres: postgres test001 [local] CREATE INDEX() [0x8f0127]
> postgres: postgres test001 [local] CREATE
> INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
> postgres: postgres test001 [local] CREATE
> INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
> postgres: postgres test001 [local] CREATE INDEX() [0x5874d6]
> postgres: postgres test001 [local] CREATE INDEX(heap_getnext+0x88)
> [0x5878e8]
> postgres: postgres test001 [local] CREATE INDEX() [0x58dcbd]
> postgres: postgres test001 [local] CREATE INDEX() [0x5b11ae]
> postgres: postgres test001 [local] CREATE INDEX(btbuild+0xb05)
> [0x5b7325]
> postgres: postgres test001 [local] CREATE INDEX(index_build+0x19c)
> [0x61fa2c]
> postgres: postgres test001 [local] CREATE
> INDEX(index_create+0x1328) [0x621608]
> postgres: postgres test001 [local] CREATE INDEX(DefineIndex+0x9bb)
> [0x6d11cb]
> postgres: postgres test001 [local] CREATE INDEX() [0xb18111]
> postgres: postgres test001 [local] CREATE
> INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
> /usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
> postgres: postgres test001 [local] CREATE INDEX() [0x93cc3a]
> postgres: postgres test001 [local] CREATE INDEX() [0x93cdaf]
> postgres: postgres test001 [local] CREATE INDEX(PortalRun+0x291)
> [0x93d371]
> postgres: postgres test001 [local] CREATE INDEX() [0x935ab0]
> postgres: postgres test001 [local] CREATE
> INDEX(PostgresMain+0x1f1b) [0x937f6b]
> postgres: postgres test001 [local] CREATE INDEX() [0x9389d5]
> postgres: postgres test001 [local] CREATE
> INDEX(postmaster_child_launch+0x12d) [0x888ced]
> postgres: postgres test001 [local] CREATE INDEX() [0x891cb2]
> postgres: postgres test001 [local] CREATE
> INDEX(PostmasterMain+0xce9) [0x894309]
> postgres: postgres test001 [local] CREATE INDEX(main+0x1d0)
> [0x531bc0]
> /lib64/libc.so.6(+0x2a610) [0x7f209b82a610]
> /lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0]
> postgres: postgres test001 [local] CREATE INDEX(_start+0x25)
> [0x5321b5]
> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]STATEMENT:
> CREATE INDEX idx_sys_notification_channel_notification_id ON
> dhr.sys_notification_channel USING btree (c_notification_id);
>
>
>
>
>
> > 2026年9月28日 11:34,PG Bug reporting form <noreply@postgresql.org> 写道:
> >
> > The following bug has been logged on the website:
> >
> > Bug reference: 19725
> > Logged by: weijie JL
> > Email address: weijie1006jl@gmail.com
> > PostgreSQL version: 18.6
> > Operating system: RockyLinux9
> > Description:
> >
> > Hello,
> >
> > I can reproduce a read failure during pg_restore with io_method=io_uring.
> > The error disappears with worker and recurs after switching back to
> > io_uring.
> >
> > Environment:
> > - PostgreSQL 18.6, x86_64
> > - VMware VM: 8 vCPUs, 32 GB RAM
> > - Kernel: 5.14.0-611.55.1.el9_7.x86_64
> >
> > Settings:
> > effective_io_concurrency = 256
> > maintenance_io_concurrency = 16
> > max_parallel_maintenance_workers = 2
> >
> > Restore command:
> > nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &
> >
> > One failing statement:
> > ALTER TABLE ONLY dhr.obj_permission_object
> > ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);
> >
> > Server error:
> > ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
> > Operation canceled
> > LOCATION: md_readv_report, md.c:2059
> > SQLSTATE: XX000
> >
> > Observed test sequence:
> > 1. io_method=io_uring: the read error occurred during index creation.
> > 2. Changed io_method to worker: retried without errors.
> > 3. Changed io_method back to io_uring: the read error was reproduced.
> >
> > I have not measured the failure rate or prepared a self-contained
> > minimal reproducer.
> >
> > One earlier run also reported "could not map dynamic shared memory
> > segment" from parallel workers immediately after the read error.
> > Those DSM errors have not recurred in subsequent failing tests.
> >
> > Does this match a known PostgreSQL bug or kernel-side io_uring issue?
> > What additional logs or tracing would help identify the cause?
> >
> > Thank you.
> >
> >
>
>
>
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: BUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker
@ 2026-10-23 07:15 weijie JL <weijie1006jl@gmail.com>
parent: Dmytro Astapov <dastapov@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: weijie JL @ 2026-10-23 07:15 UTC (permalink / raw)
To: Dmytro Astapov <dastapov@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
Hi Dmytro,
Thanks for reproducing the issue and for the detailed analysis and proposed fixes. This is very helpful.
Over the past few days, normal SQL queries in our test environment have also triggered this error. To work around the issue in the current version, I’ve switched all our production environments to the default io_method = worker setting.
> 2026年10月22日 01:44,Dmytro Astapov <dastapov@gmail.com> 写道:
>
> Hi!
>
> I can reproduce this on version 18.6 + kernel 6.12.109.el8, and I think that I have the explanation for how it happens + proposal for the fix as well.
>
> What I think is happening: postgres io_uring requests hit the kernel codepath where new io workers are spawned up to RLIMIT_NPRPOC. Once this limit is hit, no new workers can't be spawned, but also postgresmaster can't fork() or system() anymore, leading to "could not read blocks X..Y in file ...: Operation canceled"
>
> How it could be fixed: postgres should either call io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
>
> Detailed description of the chain of events follows:
>
> 1. Postgres asks io_uring to send buffered reads to worker threads
>
> In method_io_uring.c (line 441), once a backend already has more than 4 I/Os in flight, every further buffered read gets IOSQE_ASYNC:
>
> if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
> io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
>
> If `maintenance_io_concurrency` is higher that 4, then all of the maintenance reads starting from 5th will take this path.
>
> 2. The kernel places these reads into the pool whose only cap is RLIMIT_NPROC
>
> Each task's io_uring worker pool has two halves:
> - bounded (meant for regular files), capped at min(sq_entries, 4*ncpu);
> - unbound, capped only at the task's RLIMIT_NPROC (io-wq.c).
>
> Requests are classified in io_prep_async_work (https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L480-L492):
>
> if (req->file && (req->flags & REQ_F_ISREG)) { ... bound ... }
> else if (!req->file || !S_ISBLK(...)) {
> if (def->unbound_nonreg_file) // set for READ/READV
> atomic_or(IO_WQ_WORK_UNBOUND, ...);
> }
>
> However, io_init_req sets req->file = NULL (at https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L2046), and the file is only attached later, at issue time. An IOSQE_ASYNC request goes straight though io_queue_iowq() to io_prep_async_work() while `req->file` is still NULL.
>
> So a read of a regular file ends up routed to the unbound pool, whose cap is RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also marks the work as IO_WQ_WORK_CONCURRENT, which lets each enqueue create a worker instead of waiting for a running one.
>
> 3. Workers get created far faster than the actual I/O needs
>
> In io_wq_dec_running, when a worker blocks and it was the last running worker while work is still queued, the kernel queues creation of a brand-new worker immediately.
>
> It doesn't wake one of the idle workers. On a cold buffer cache almost every read blocks on disk, so this code path fires constantly. As a result, the new worker handles one item and then goes idle. Only the head of the free list gets reused, so the rest of the workers sit idle until WORKER_IDLE_TIMEOUT = 5s expires and then they exit.
>
> As a result, the steady-state thread count is therefore roughly "creation rate × 5 s", not "I/Os in flight". The cap for the workers, as was mentioned earlier, is RLIMIT_NPROC.
>
> Similar threads or bugs from the past
> Justin Pryzby on pgsql-hackers <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg208462.html; (2025-10-17) reported could not read blocks 122..126 ...: Operation canceled from pg_dump under io_uring. He guessed it might be related to the io_uring file descriptors leaking after crash-restarts, but I cannot find a resolution.
> Alexander Lakhin's AIO stress test <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196158.html; (pgsql-hackers, April 2025, before 18.0) hit could not read blocks 0..0 in file "global/1213": Operation canceled with parallel createdb/dropdb jobs, on kernels 6.8 and 6.11. The bugs fixed later in that thread (1 <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196804.html;, 2 <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg197025.html;) were different ones, a pg_aios crash and a PANIC from interrupt handling. As far as I can see, the ECANCELED was never explained or fixed.
>
> io_uring bugs that look relevant to me
> liburing #420 <https://github.com/axboe/liburing/issues/420; (2021): a user found IOSQE_ASYNC "spawns a thread for each request, meaning 10000+ threads". There is a commen there that bounded work is limited but unbounded work "really [is] not". That discussion led to creation of IORING_REGISTER_IOWQ_MAX_WORKERS.
> liburing #349 <https://github.com/axboe/liburing/issues/349; (2021) reported "unexpected high count of io_worker threads by using the IOSQE_ASYNC flag".
> Cloudflare, "Missing Manuals – io_uring worker pool" <https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/; explains that unbound workers default to RLIMIT_NPROC for each thread, and that raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS can backfire. The io_uring_register_iowq_max_workers(3) <https://man7.org/linux/man-pages/man3/io_uring_register_iowq_max_workers.3.html; man page says the same.
>
> So, in conclusion, it looks to me that the issue is that req->file is still NULL when an IOSQE_ASYNC request is classified, and the reads end up in the unbound pool, and the fix is either to call io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
>
> Best regards, Dmytro
>
>
> On Sun, Oct 4, 2026 at 10:51 AM weijie JL <weijie1006jl@gmail.com <mailto:weijie1006jl@gmail.com>> wrote:
>> After setting backtrace_functions=md_readv_report, I received more log information:
>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG: XX000: could not read blocks 143..158 in file "base/6406098/6407866": Operation canceled
>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOCATION: md_readv_report, md.c:2059
>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]BACKTRACE:
>> postgres: postgres test001 [local] CREATE INDEX() [0x92eafc]
>> postgres: postgres test001 [local] CREATE INDEX() [0x92f7ec]
>> postgres: postgres test001 [local] CREATE INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
>> postgres: postgres test001 [local] CREATE INDEX() [0x8f209c]
>> postgres: postgres test001 [local] CREATE INDEX() [0x8f2b63]
>> postgres: postgres test001 [local] CREATE INDEX() [0x8f0127]
>> postgres: postgres test001 [local] CREATE INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
>> postgres: postgres test001 [local] CREATE INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
>> postgres: postgres test001 [local] CREATE INDEX() [0x5874d6]
>> postgres: postgres test001 [local] CREATE INDEX(heap_getnext+0x88) [0x5878e8]
>> postgres: postgres test001 [local] CREATE INDEX() [0x58dcbd]
>> postgres: postgres test001 [local] CREATE INDEX() [0x5b11ae]
>> postgres: postgres test001 [local] CREATE INDEX(btbuild+0xb05) [0x5b7325]
>> postgres: postgres test001 [local] CREATE INDEX(index_build+0x19c) [0x61fa2c]
>> postgres: postgres test001 [local] CREATE INDEX(index_create+0x1328) [0x621608]
>> postgres: postgres test001 [local] CREATE INDEX(DefineIndex+0x9bb) [0x6d11cb]
>> postgres: postgres test001 [local] CREATE INDEX() [0xb18111]
>> postgres: postgres test001 [local] CREATE INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
>> /usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
>> postgres: postgres test001 [local] CREATE INDEX() [0x93cc3a]
>> postgres: postgres test001 [local] CREATE INDEX() [0x93cdaf]
>> postgres: postgres test001 [local] CREATE INDEX(PortalRun+0x291) [0x93d371]
>> postgres: postgres test001 [local] CREATE INDEX() [0x935ab0]
>> postgres: postgres test001 [local] CREATE INDEX(PostgresMain+0x1f1b) [0x937f6b]
>> postgres: postgres test001 [local] CREATE INDEX() [0x9389d5]
>> postgres: postgres test001 [local] CREATE INDEX(postmaster_child_launch+0x12d) [0x888ced]
>> postgres: postgres test001 [local] CREATE INDEX() [0x891cb2]
>> postgres: postgres test001 [local] CREATE INDEX(PostmasterMain+0xce9) [0x894309]
>> postgres: postgres test001 [local] CREATE INDEX(main+0x1d0) [0x531bc0]
>> /lib64/libc.so.6(+0x2a610) [0x7f209b82a610]
>> /lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0]
>> postgres: postgres test001 [local] CREATE INDEX(_start+0x25) [0x5321b5]
>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]STATEMENT: CREATE INDEX idx_sys_notification_channel_notification_id ON dhr.sys_notification_channel USING btree (c_notification_id);
>>
>>
>>
>>
>>
>> > 2026年9月28日 11:34,PG Bug reporting form <noreply@postgresql.org <mailto:noreply@postgresql.org>> 写道:
>> >
>> > The following bug has been logged on the website:
>> >
>> > Bug reference: 19725
>> > Logged by: weijie JL
>> > Email address: weijie1006jl@gmail.com <mailto:weijie1006jl@gmail.com>
>> > PostgreSQL version: 18.6
>> > Operating system: RockyLinux9
>> > Description:
>> >
>> > Hello,
>> >
>> > I can reproduce a read failure during pg_restore with io_method=io_uring.
>> > The error disappears with worker and recurs after switching back to
>> > io_uring.
>> >
>> > Environment:
>> > - PostgreSQL 18.6, x86_64
>> > - VMware VM: 8 vCPUs, 32 GB RAM
>> > - Kernel: 5.14.0-611.55.1.el9_7.x86_64
>> >
>> > Settings:
>> > effective_io_concurrency = 256
>> > maintenance_io_concurrency = 16
>> > max_parallel_maintenance_workers = 2
>> >
>> > Restore command:
>> > nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &
>> >
>> > One failing statement:
>> > ALTER TABLE ONLY dhr.obj_permission_object
>> > ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);
>> >
>> > Server error:
>> > ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
>> > Operation canceled
>> > LOCATION: md_readv_report, md.c:2059
>> > SQLSTATE: XX000
>> >
>> > Observed test sequence:
>> > 1. io_method=io_uring: the read error occurred during index creation.
>> > 2. Changed io_method to worker: retried without errors.
>> > 3. Changed io_method back to io_uring: the read error was reproduced.
>> >
>> > I have not measured the failure rate or prepared a self-contained
>> > minimal reproducer.
>> >
>> > One earlier run also reported "could not map dynamic shared memory
>> > segment" from parallel workers immediately after the read error.
>> > Those DSM errors have not recurred in subsequent failing tests.
>> >
>> > Does this match a known PostgreSQL bug or kernel-side io_uring issue?
>> > What additional logs or tracing would help identify the cause?
>> >
>> > Thank you.
>> >
>> >
>>
>>
>>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: BUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker
@ 2026-10-24 12:07 Dmytro Astapov <dastapov@gmail.com>
parent: weijie JL <weijie1006jl@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: Dmytro Astapov @ 2026-10-24 12:07 UTC (permalink / raw)
To: weijie JL <weijie1006jl@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
Hi,
A follow-up to my earlier message. On further examination, *this is a
kernel bug, not a Postgres bug*. The changes I suggested for Postgres
(calling io_uring_register_iowq_max_workers(), or not setting IOSQE_ASYNC)
would only be workarounds, and my suggestion was therefore wrong.
As a quick recap, Postgres assumes that a buffered read of a regular file,
submitted with IOSQE_ASYNC, goes to io_uring's bounded worker pool. This
assumption is valid. That pool is limited to min(SQ ring size, 4 × CPUs),
which for Postgres is about io_max_concurrency workers per backend. This is
how io_uring is meant to work: reads and writes of regular files and block
devices are bounded work.
The kernel instead sends these requests to the unbound pool, whose only
limit is RLIMIT_NPROC. It decides the pool in io_prep_async_work(), but for
an IOSQE_ASYNC request the file has not been looked up yet at that point (
req->file is NULL), so the read is treated as "not a regular file". With
enough backends and a cold cache, the worker threads use up RLIMIT_NPROC.
After that, fork() fails and queued reads complete with -ECANCELED.
This bug was introduced in Linux 5.18 by commit 6bf9c47a3989 ("io_uring:
defer file assignment"), which was also marked for 5.15+ stable kernels:
https://github.com/torvalds/linux/commit/6bf9c47a398911e0ab920e362115153596c80432
A fix was made on 2025-10-15, and is queued for 7.4 in Jens Axboe's tree,
as commit 98bc044d5a53 ("io_uring: look up the file before queueing to
io-wq"):
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/commit/?id=98bc044d5a5374f8d3605fe55...
It is not (yet?) tagged for stable releases, so older kernels might only
get it if someone backports it. Until then, io_method = worker is the safe
choice, as you have already done.
I have a small C program that reproduces the issue (no Postgres needed),
which could be used to check the if host is susceptible to this problem.
Let me know if anyone wants it.
Best regards, Dmytro
On Fri, Oct 23, 2026 at 8:16 AM weijie JL <weijie1006jl@gmail.com> wrote:
> Hi Dmytro,
>
> Thanks for reproducing the issue and for the detailed analysis and
> proposed fixes. This is very helpful.
>
> Over the past few days, normal SQL queries in our test environment have
> also triggered this error. To work around the issue in the current version,
> I’ve switched all our production environments to the default io_method =
> worker setting.
>
> 2026年10月22日 01:44,Dmytro Astapov <dastapov@gmail.com> 写道:
>
> Hi!
>
> I can reproduce this on version 18.6 + kernel 6.12.109.el8, and I think
> that I have the explanation for how it happens + proposal for the fix as
> well.
>
> *What I think is happening*: postgres io_uring requests hit the kernel
> codepath where new io workers are spawned up to RLIMIT_NPRPOC. Once this
> limit is hit, no new workers can't be spawned, but also postgresmaster
> can't fork() or system() anymore, leading to "could not read blocks X..Y
> in file ...: Operation canceled"
>
> *How it could be fixed*: postgres should either call
> io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
>
> *Detailed description of the chain of events follows:*
>
>
> *1. Postgres asks io_uring to send buffered reads to worker threads*
> In method_io_uring.c (line 441), once a backend already has more than 4
> I/Os in flight, every further buffered read gets IOSQE_ASYNC:
>
> if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
> io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
>
> If `maintenance_io_concurrency` is higher that 4, then all of the
> maintenance reads starting from 5th will take this path.
>
>
> *2. The kernel places these reads into the pool whose only cap is
> RLIMIT_NPROC*
> Each task's io_uring worker pool has two halves:
> - bounded (meant for regular files), capped at min(sq_entries, 4*ncpu);
> - unbound, capped only at the task's RLIMIT_NPROC (io-wq.c).
>
> Requests are classified in io_prep_async_work (
> https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L480-L492
> ):
>
> if (req->file && (req->flags & REQ_F_ISREG)) { ... bound ... }
> else if (!req->file || !S_ISBLK(...)) {
> if (def->unbound_nonreg_file) // set for READ/READV
> atomic_or(IO_WQ_WORK_UNBOUND, ...);
> }
>
> However, io_init_req sets req->file = NULL (at
> https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L2046),
> and the file is only attached later, at issue time. An IOSQE_ASYNC request
> goes straight though io_queue_iowq() to io_prep_async_work() while
> `req->file` is still NULL.
>
> So a read of a regular file ends up routed to the unbound pool, whose cap
> is RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also marks the work
> as IO_WQ_WORK_CONCURRENT, which lets each enqueue create a worker instead
> of waiting for a running one.
>
> *3. Workers get created far faster than the actual I/O needs*
>
> In io_wq_dec_running, when a worker blocks and it was the last running
> worker while work is still queued, the kernel queues creation of a
> brand-new worker immediately.
>
> It doesn't wake one of the idle workers. On a cold buffer cache almost
> every read blocks on disk, so this code path fires constantly. As a result,
> the new worker handles one item and then goes idle. Only the head of the
> free list gets reused, so the rest of the workers sit idle until
> WORKER_IDLE_TIMEOUT = 5s expires and then they exit.
>
> As a result, the steady-state thread count is therefore roughly "creation
> rate × 5 s", not "I/Os in flight". The cap for the workers, as was
> mentioned earlier, is RLIMIT_NPROC.
>
> *Similar threads or bugs from the past*
>
> - Justin Pryzby on pgsql-hackers
> <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg208462.html; (2025-10-17)
> reported could not read blocks 122..126 ...: Operation canceled from
> pg_dump under io_uring. He guessed it might be related to the io_uring
> file descriptors leaking after crash-restarts, but I cannot find a
> resolution.
> - Alexander Lakhin's AIO stress test
> <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196158.html; (pgsql-hackers,
> April 2025, before 18.0) hit could not read blocks 0..0 in file
> "global/1213": Operation canceled with parallel createdb/dropdb jobs,
> on kernels 6.8 and 6.11. The bugs fixed later in that thread (1
> <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196804.html;
> , 2
> <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg197025.html;)
> were different ones, a pg_aios crash and a PANIC from interrupt
> handling. As far as I can see, the ECANCELED was never explained or fixed.
>
>
> io_uring bugs that look relevant to me
>
> - liburing #420 <https://github.com/axboe/liburing/issues/420; (2021):
> a user found IOSQE_ASYNC "spawns a thread for each request, meaning
> 10000+ threads". There is a commen there that bounded work is limited but
> unbounded work "really [is] not". That discussion led to creation of
> IORING_REGISTER_IOWQ_MAX_WORKERS.
> - liburing #349 <https://github.com/axboe/liburing/issues/349; (2021)
> reported "unexpected high count of io_worker threads by using the
> IOSQE_ASYNC flag".
> - Cloudflare, "Missing Manuals – io_uring worker pool"
> <https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/; explains
> that unbound workers default to RLIMIT_NPROC for each thread, and that
> raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS can
> backfire. The io_uring_register_iowq_max_workers(3)
> <https://man7.org/linux/man-pages/man3/io_uring_register_iowq_max_workers.3.html; man
> page says the same.
>
>
> So, in conclusion, it looks to me that the issue is that req->file is
> still NULL when an IOSQE_ASYNC request is classified, and the reads end
> up in the unbound pool, and the fix is either to call
> io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
>
> Best regards, Dmytro
>
>
> On Sun, Oct 4, 2026 at 10:51 AM weijie JL <weijie1006jl@gmail.com> wrote:
>
>> After setting backtrace_functions=md_readv_report, I received more log
>> information:
>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG: XX000:
>> could not read blocks 143..158 in file "base/6406098/6407866": Operation
>> canceled
>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOCATION:
>> md_readv_report, md.c:2059
>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]BACKTRACE:
>> postgres: postgres test001 [local] CREATE INDEX() [0x92eafc]
>> postgres: postgres test001 [local] CREATE INDEX() [0x92f7ec]
>> postgres: postgres test001 [local] CREATE
>> INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
>> postgres: postgres test001 [local] CREATE INDEX() [0x8f209c]
>> postgres: postgres test001 [local] CREATE INDEX() [0x8f2b63]
>> postgres: postgres test001 [local] CREATE INDEX() [0x8f0127]
>> postgres: postgres test001 [local] CREATE
>> INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
>> postgres: postgres test001 [local] CREATE
>> INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
>> postgres: postgres test001 [local] CREATE INDEX() [0x5874d6]
>> postgres: postgres test001 [local] CREATE
>> INDEX(heap_getnext+0x88) [0x5878e8]
>> postgres: postgres test001 [local] CREATE INDEX() [0x58dcbd]
>> postgres: postgres test001 [local] CREATE INDEX() [0x5b11ae]
>> postgres: postgres test001 [local] CREATE INDEX(btbuild+0xb05)
>> [0x5b7325]
>> postgres: postgres test001 [local] CREATE
>> INDEX(index_build+0x19c) [0x61fa2c]
>> postgres: postgres test001 [local] CREATE
>> INDEX(index_create+0x1328) [0x621608]
>> postgres: postgres test001 [local] CREATE
>> INDEX(DefineIndex+0x9bb) [0x6d11cb]
>> postgres: postgres test001 [local] CREATE INDEX() [0xb18111]
>> postgres: postgres test001 [local] CREATE
>> INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
>> /usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
>> postgres: postgres test001 [local] CREATE INDEX() [0x93cc3a]
>> postgres: postgres test001 [local] CREATE INDEX() [0x93cdaf]
>> postgres: postgres test001 [local] CREATE INDEX(PortalRun+0x291)
>> [0x93d371]
>> postgres: postgres test001 [local] CREATE INDEX() [0x935ab0]
>> postgres: postgres test001 [local] CREATE
>> INDEX(PostgresMain+0x1f1b) [0x937f6b]
>> postgres: postgres test001 [local] CREATE INDEX() [0x9389d5]
>> postgres: postgres test001 [local] CREATE
>> INDEX(postmaster_child_launch+0x12d) [0x888ced]
>> postgres: postgres test001 [local] CREATE INDEX() [0x891cb2]
>> postgres: postgres test001 [local] CREATE
>> INDEX(PostmasterMain+0xce9) [0x894309]
>> postgres: postgres test001 [local] CREATE INDEX(main+0x1d0)
>> [0x531bc0]
>> /lib64/libc.so.6(+0x2a610) [0x7f209b82a610]
>> /lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0]
>> postgres: postgres test001 [local] CREATE INDEX(_start+0x25)
>> [0x5321b5]
>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]STATEMENT:
>> CREATE INDEX idx_sys_notification_channel_notification_id ON
>> dhr.sys_notification_channel USING btree (c_notification_id);
>>
>>
>>
>>
>>
>> > 2026年9月28日 11:34,PG Bug reporting form <noreply@postgresql.org> 写道:
>> >
>> > The following bug has been logged on the website:
>> >
>> > Bug reference: 19725
>> > Logged by: weijie JL
>> > Email address: weijie1006jl@gmail.com
>> > PostgreSQL version: 18.6
>> > Operating system: RockyLinux9
>> > Description:
>> >
>> > Hello,
>> >
>> > I can reproduce a read failure during pg_restore with
>> io_method=io_uring.
>> > The error disappears with worker and recurs after switching back to
>> > io_uring.
>> >
>> > Environment:
>> > - PostgreSQL 18.6, x86_64
>> > - VMware VM: 8 vCPUs, 32 GB RAM
>> > - Kernel: 5.14.0-611.55.1.el9_7.x86_64
>> >
>> > Settings:
>> > effective_io_concurrency = 256
>> > maintenance_io_concurrency = 16
>> > max_parallel_maintenance_workers = 2
>> >
>> > Restore command:
>> > nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &
>> >
>> > One failing statement:
>> > ALTER TABLE ONLY dhr.obj_permission_object
>> > ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);
>> >
>> > Server error:
>> > ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
>> > Operation canceled
>> > LOCATION: md_readv_report, md.c:2059
>> > SQLSTATE: XX000
>> >
>> > Observed test sequence:
>> > 1. io_method=io_uring: the read error occurred during index creation.
>> > 2. Changed io_method to worker: retried without errors.
>> > 3. Changed io_method back to io_uring: the read error was reproduced.
>> >
>> > I have not measured the failure rate or prepared a self-contained
>> > minimal reproducer.
>> >
>> > One earlier run also reported "could not map dynamic shared memory
>> > segment" from parallel workers immediately after the read error.
>> > Those DSM errors have not recurred in subsequent failing tests.
>> >
>> > Does this match a known PostgreSQL bug or kernel-side io_uring issue?
>> > What additional logs or tracing would help identify the cause?
>> >
>> > Thank you.
>> >
>> >
>>
>>
>>
>>
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: BUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker
@ 2026-11-03 01:08 Dmytro Astapov <dastapov@gmail.com>
parent: Dmytro Astapov <dastapov@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: Dmytro Astapov @ 2026-11-03 01:08 UTC (permalink / raw)
To: weijie JL <weijie1006jl@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
JFYI, the kernel fix for this was merged into Linux 7.4 this Sat
(2026-10-31), and the fix was included in all stable kernel versions as
well.
Best regards, Dmytro
On Sat, Oct 24, 2026 at 1:07 PM Dmytro Astapov <dastapov@gmail.com> wrote:
> Hi,
>
> A follow-up to my earlier message. On further examination, *this is a
> kernel bug, not a Postgres bug*. The changes I suggested for Postgres
> (calling io_uring_register_iowq_max_workers(), or not setting IOSQE_ASYNC)
> would only be workarounds, and my suggestion was therefore wrong.
>
> As a quick recap, Postgres assumes that a buffered read of a regular file,
> submitted with IOSQE_ASYNC, goes to io_uring's bounded worker pool. This
> assumption is valid. That pool is limited to min(SQ ring size, 4 × CPUs),
> which for Postgres is about io_max_concurrency workers per backend. This
> is how io_uring is meant to work: reads and writes of regular files and
> block devices are bounded work.
>
> The kernel instead sends these requests to the unbound pool, whose only
> limit is RLIMIT_NPROC. It decides the pool in io_prep_async_work(), but
> for an IOSQE_ASYNC request the file has not been looked up yet at that
> point (req->file is NULL), so the read is treated as "not a regular
> file". With enough backends and a cold cache, the worker threads use up
> RLIMIT_NPROC. After that, fork() fails and queued reads complete with
> -ECANCELED.
>
> This bug was introduced in Linux 5.18 by commit 6bf9c47a3989 ("io_uring:
> defer file assignment"), which was also marked for 5.15+ stable kernels:
> https://github.com/torvalds/linux/commit/6bf9c47a398911e0ab920e362115153596c80432
>
> A fix was made on 2025-10-15, and is queued for 7.4 in Jens Axboe's tree,
> as commit 98bc044d5a53 ("io_uring: look up the file before queueing to
> io-wq"):
> https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/commit/?id=98bc044d5a5374f8d3605fe55...
>
> It is not (yet?) tagged for stable releases, so older kernels might only
> get it if someone backports it. Until then, io_method = worker is the
> safe choice, as you have already done.
>
> I have a small C program that reproduces the issue (no Postgres needed),
> which could be used to check the if host is susceptible to this problem.
> Let me know if anyone wants it.
>
> Best regards, Dmytro
>
>
> On Fri, Oct 23, 2026 at 8:16 AM weijie JL <weijie1006jl@gmail.com> wrote:
>
>> Hi Dmytro,
>>
>> Thanks for reproducing the issue and for the detailed analysis and
>> proposed fixes. This is very helpful.
>>
>> Over the past few days, normal SQL queries in our test environment have
>> also triggered this error. To work around the issue in the current version,
>> I’ve switched all our production environments to the default io_method =
>> worker setting.
>>
>> 2026年10月22日 01:44,Dmytro Astapov <dastapov@gmail.com> 写道:
>>
>> Hi!
>>
>> I can reproduce this on version 18.6 + kernel 6.12.109.el8, and I think
>> that I have the explanation for how it happens + proposal for the fix as
>> well.
>>
>> *What I think is happening*: postgres io_uring requests hit the kernel
>> codepath where new io workers are spawned up to RLIMIT_NPRPOC. Once this
>> limit is hit, no new workers can't be spawned, but also postgresmaster
>> can't fork() or system() anymore, leading to "could not read blocks X..Y
>> in file ...: Operation canceled"
>>
>> *How it could be fixed*: postgres should either call
>> io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
>>
>> *Detailed description of the chain of events follows:*
>>
>>
>> *1. Postgres asks io_uring to send buffered reads to worker threads*
>> In method_io_uring.c (line 441), once a backend already has more than 4
>> I/Os in flight, every further buffered read gets IOSQE_ASYNC:
>>
>> if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
>> io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
>>
>> If `maintenance_io_concurrency` is higher that 4, then all of the
>> maintenance reads starting from 5th will take this path.
>>
>>
>> *2. The kernel places these reads into the pool whose only cap is
>> RLIMIT_NPROC*
>> Each task's io_uring worker pool has two halves:
>> - bounded (meant for regular files), capped at min(sq_entries, 4*ncpu);
>> - unbound, capped only at the task's RLIMIT_NPROC (io-wq.c).
>>
>> Requests are classified in io_prep_async_work (
>> https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L480-L492
>> ):
>>
>> if (req->file && (req->flags & REQ_F_ISREG)) { ... bound ... }
>> else if (!req->file || !S_ISBLK(...)) {
>> if (def->unbound_nonreg_file) // set for READ/READV
>> atomic_or(IO_WQ_WORK_UNBOUND, ...);
>> }
>>
>> However, io_init_req sets req->file = NULL (at
>> https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L2046),
>> and the file is only attached later, at issue time. An IOSQE_ASYNC request
>> goes straight though io_queue_iowq() to io_prep_async_work() while
>> `req->file` is still NULL.
>>
>> So a read of a regular file ends up routed to the unbound pool, whose cap
>> is RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also marks the work
>> as IO_WQ_WORK_CONCURRENT, which lets each enqueue create a worker instead
>> of waiting for a running one.
>>
>> *3. Workers get created far faster than the actual I/O needs*
>>
>> In io_wq_dec_running, when a worker blocks and it was the last running
>> worker while work is still queued, the kernel queues creation of a
>> brand-new worker immediately.
>>
>> It doesn't wake one of the idle workers. On a cold buffer cache almost
>> every read blocks on disk, so this code path fires constantly. As a result,
>> the new worker handles one item and then goes idle. Only the head of the
>> free list gets reused, so the rest of the workers sit idle until
>> WORKER_IDLE_TIMEOUT = 5s expires and then they exit.
>>
>> As a result, the steady-state thread count is therefore roughly "creation
>> rate × 5 s", not "I/Os in flight". The cap for the workers, as was
>> mentioned earlier, is RLIMIT_NPROC.
>>
>> *Similar threads or bugs from the past*
>>
>> - Justin Pryzby on pgsql-hackers
>> <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg208462.html; (2025-10-17)
>> reported could not read blocks 122..126 ...: Operation canceled from
>> pg_dump under io_uring. He guessed it might be related to the
>> io_uring file descriptors leaking after crash-restarts, but I cannot find a
>> resolution.
>> - Alexander Lakhin's AIO stress test
>> <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196158.html; (pgsql-hackers,
>> April 2025, before 18.0) hit could not read blocks 0..0 in file
>> "global/1213": Operation canceled with parallel createdb/dropdb jobs,
>> on kernels 6.8 and 6.11. The bugs fixed later in that thread (1
>> <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196804.html;
>> , 2
>> <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg197025.html;)
>> were different ones, a pg_aios crash and a PANIC from interrupt
>> handling. As far as I can see, the ECANCELED was never explained or fixed.
>>
>>
>> io_uring bugs that look relevant to me
>>
>> - liburing #420 <https://github.com/axboe/liburing/issues/420; (2021):
>> a user found IOSQE_ASYNC "spawns a thread for each request, meaning
>> 10000+ threads". There is a commen there that bounded work is limited but
>> unbounded work "really [is] not". That discussion led to creation of
>> IORING_REGISTER_IOWQ_MAX_WORKERS.
>> - liburing #349 <https://github.com/axboe/liburing/issues/349; (2021)
>> reported "unexpected high count of io_worker threads by using the
>> IOSQE_ASYNC flag".
>> - Cloudflare, "Missing Manuals – io_uring worker pool"
>> <https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/; explains
>> that unbound workers default to RLIMIT_NPROC for each thread, and
>> that raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS can
>> backfire. The io_uring_register_iowq_max_workers(3)
>> <https://man7.org/linux/man-pages/man3/io_uring_register_iowq_max_workers.3.html; man
>> page says the same.
>>
>>
>> So, in conclusion, it looks to me that the issue is that req->file is
>> still NULL when an IOSQE_ASYNC request is classified, and the reads end
>> up in the unbound pool, and the fix is either to call
>> io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
>>
>> Best regards, Dmytro
>>
>>
>> On Sun, Oct 4, 2026 at 10:51 AM weijie JL <weijie1006jl@gmail.com> wrote:
>>
>>> After setting backtrace_functions=md_readv_report, I received more log
>>> information:
>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG:
>>> XX000: could not read blocks 143..158 in file "base/6406098/6407866":
>>> Operation canceled
>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOCATION:
>>> md_readv_report, md.c:2059
>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]BACKTRACE:
>>> postgres: postgres test001 [local] CREATE INDEX() [0x92eafc]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x92f7ec]
>>> postgres: postgres test001 [local] CREATE
>>> INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x8f209c]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x8f2b63]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x8f0127]
>>> postgres: postgres test001 [local] CREATE
>>> INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
>>> postgres: postgres test001 [local] CREATE
>>> INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x5874d6]
>>> postgres: postgres test001 [local] CREATE
>>> INDEX(heap_getnext+0x88) [0x5878e8]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x58dcbd]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x5b11ae]
>>> postgres: postgres test001 [local] CREATE INDEX(btbuild+0xb05)
>>> [0x5b7325]
>>> postgres: postgres test001 [local] CREATE
>>> INDEX(index_build+0x19c) [0x61fa2c]
>>> postgres: postgres test001 [local] CREATE
>>> INDEX(index_create+0x1328) [0x621608]
>>> postgres: postgres test001 [local] CREATE
>>> INDEX(DefineIndex+0x9bb) [0x6d11cb]
>>> postgres: postgres test001 [local] CREATE INDEX() [0xb18111]
>>> postgres: postgres test001 [local] CREATE
>>> INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
>>> /usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x93cc3a]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x93cdaf]
>>> postgres: postgres test001 [local] CREATE INDEX(PortalRun+0x291)
>>> [0x93d371]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x935ab0]
>>> postgres: postgres test001 [local] CREATE
>>> INDEX(PostgresMain+0x1f1b) [0x937f6b]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x9389d5]
>>> postgres: postgres test001 [local] CREATE
>>> INDEX(postmaster_child_launch+0x12d) [0x888ced]
>>> postgres: postgres test001 [local] CREATE INDEX() [0x891cb2]
>>> postgres: postgres test001 [local] CREATE
>>> INDEX(PostmasterMain+0xce9) [0x894309]
>>> postgres: postgres test001 [local] CREATE INDEX(main+0x1d0)
>>> [0x531bc0]
>>> /lib64/libc.so.6(+0x2a610) [0x7f209b82a610]
>>> /lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0]
>>> postgres: postgres test001 [local] CREATE INDEX(_start+0x25)
>>> [0x5321b5]
>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]STATEMENT:
>>> CREATE INDEX idx_sys_notification_channel_notification_id ON
>>> dhr.sys_notification_channel USING btree (c_notification_id);
>>>
>>>
>>>
>>>
>>>
>>> > 2026年9月28日 11:34,PG Bug reporting form <noreply@postgresql.org> 写道:
>>> >
>>> > The following bug has been logged on the website:
>>> >
>>> > Bug reference: 19725
>>> > Logged by: weijie JL
>>> > Email address: weijie1006jl@gmail.com
>>> > PostgreSQL version: 18.6
>>> > Operating system: RockyLinux9
>>> > Description:
>>> >
>>> > Hello,
>>> >
>>> > I can reproduce a read failure during pg_restore with
>>> io_method=io_uring.
>>> > The error disappears with worker and recurs after switching back to
>>> > io_uring.
>>> >
>>> > Environment:
>>> > - PostgreSQL 18.6, x86_64
>>> > - VMware VM: 8 vCPUs, 32 GB RAM
>>> > - Kernel: 5.14.0-611.55.1.el9_7.x86_64
>>> >
>>> > Settings:
>>> > effective_io_concurrency = 256
>>> > maintenance_io_concurrency = 16
>>> > max_parallel_maintenance_workers = 2
>>> >
>>> > Restore command:
>>> > nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &
>>> >
>>> > One failing statement:
>>> > ALTER TABLE ONLY dhr.obj_permission_object
>>> > ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);
>>> >
>>> > Server error:
>>> > ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
>>> > Operation canceled
>>> > LOCATION: md_readv_report, md.c:2059
>>> > SQLSTATE: XX000
>>> >
>>> > Observed test sequence:
>>> > 1. io_method=io_uring: the read error occurred during index creation.
>>> > 2. Changed io_method to worker: retried without errors.
>>> > 3. Changed io_method back to io_uring: the read error was reproduced.
>>> >
>>> > I have not measured the failure rate or prepared a self-contained
>>> > minimal reproducer.
>>> >
>>> > One earlier run also reported "could not map dynamic shared memory
>>> > segment" from parallel workers immediately after the read error.
>>> > Those DSM errors have not recurred in subsequent failing tests.
>>> >
>>> > Does this match a known PostgreSQL bug or kernel-side io_uring issue?
>>> > What additional logs or tracing would help identify the cause?
>>> >
>>> > Thank you.
>>> >
>>> >
>>>
>>>
>>>
>>>
>>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: BUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker
@ 2026-11-03 01:22 weijie JL <weijie1006jl@gmail.com>
parent: Dmytro Astapov <dastapov@gmail.com>
0 siblings, 0 replies; 7+ messages in thread
From: weijie JL @ 2026-11-03 01:22 UTC (permalink / raw)
To: Dmytro Astapov <dastapov@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
Thanks for the update and for your detailed investigation. It’s great to hear that the fix has been merged and backported to the stable kernel branches.
I really appreciate your help in tracking down the issue and following up on the fix.
Best regards,
> 2026年11月3日 09:08,Dmytro Astapov <dastapov@gmail.com> 写道:
>
> JFYI, the kernel fix for this was merged into Linux 7.4 this Sat (2026-10-31), and the fix was included in all stable kernel versions as well.
>
> Best regards, Dmytro
>
>
> On Sat, Oct 24, 2026 at 1:07 PM Dmytro Astapov <dastapov@gmail.com <mailto:dastapov@gmail.com>> wrote:
>> Hi,
>> A follow-up to my earlier message. On further examination, this is a kernel bug, not a Postgres bug. The changes I suggested for Postgres (calling io_uring_register_iowq_max_workers(), or not setting IOSQE_ASYNC) would only be workarounds, and my suggestion was therefore wrong.
>> As a quick recap, Postgres assumes that a buffered read of a regular file, submitted with IOSQE_ASYNC, goes to io_uring's bounded worker pool. This assumption is valid. That pool is limited to min(SQ ring size, 4 × CPUs), which for Postgres is about io_max_concurrency workers per backend. This is how io_uring is meant to work: reads and writes of regular files and block devices are bounded work.
>> The kernel instead sends these requests to the unbound pool, whose only limit is RLIMIT_NPROC. It decides the pool in io_prep_async_work(), but for an IOSQE_ASYNC request the file has not been looked up yet at that point (req->file is NULL), so the read is treated as "not a regular file". With enough backends and a cold cache, the worker threads use up RLIMIT_NPROC. After that, fork() fails and queued reads complete with -ECANCELED.
>> This bug was introduced in Linux 5.18 by commit 6bf9c47a3989 ("io_uring: defer file assignment"), which was also marked for 5.15+ stable kernels: https://github.com/torvalds/linux/commit/6bf9c47a398911e0ab920e362115153596c80432
>> A fix was made on 2025-10-15, and is queued for 7.4 in Jens Axboe's tree, as commit 98bc044d5a53 ("io_uring: look up the file before queueing to io-wq"): https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/commit/?id=98bc044d5a5374f8d3605fe55...
>> It is not (yet?) tagged for stable releases, so older kernels might only get it if someone backports it. Until then, io_method = worker is the safe choice, as you have already done.
>> I have a small C program that reproduces the issue (no Postgres needed), which could be used to check the if host is susceptible to this problem. Let me know if anyone wants it.
>>
>> Best regards, Dmytro
>>
>>
>> On Fri, Oct 23, 2026 at 8:16 AM weijie JL <weijie1006jl@gmail.com <mailto:weijie1006jl@gmail.com>> wrote:
>>> Hi Dmytro,
>>>
>>> Thanks for reproducing the issue and for the detailed analysis and proposed fixes. This is very helpful.
>>>
>>> Over the past few days, normal SQL queries in our test environment have also triggered this error. To work around the issue in the current version, I’ve switched all our production environments to the default io_method = worker setting.
>>>
>>>
>>>> 2026年10月22日 01:44,Dmytro Astapov <dastapov@gmail.com <mailto:dastapov@gmail.com>> 写道:
>>>>
>>>> Hi!
>>>>
>>>> I can reproduce this on version 18.6 + kernel 6.12.109.el8, and I think that I have the explanation for how it happens + proposal for the fix as well.
>>>>
>>>> What I think is happening: postgres io_uring requests hit the kernel codepath where new io workers are spawned up to RLIMIT_NPRPOC. Once this limit is hit, no new workers can't be spawned, but also postgresmaster can't fork() or system() anymore, leading to "could not read blocks X..Y in file ...: Operation canceled"
>>>>
>>>> How it could be fixed: postgres should either call io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
>>>>
>>>> Detailed description of the chain of events follows:
>>>>
>>>> 1. Postgres asks io_uring to send buffered reads to worker threads
>>>>
>>>> In method_io_uring.c (line 441), once a backend already has more than 4 I/Os in flight, every further buffered read gets IOSQE_ASYNC:
>>>>
>>>> if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
>>>> io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
>>>>
>>>> If `maintenance_io_concurrency` is higher that 4, then all of the maintenance reads starting from 5th will take this path.
>>>>
>>>> 2. The kernel places these reads into the pool whose only cap is RLIMIT_NPROC
>>>>
>>>> Each task's io_uring worker pool has two halves:
>>>> - bounded (meant for regular files), capped at min(sq_entries, 4*ncpu);
>>>> - unbound, capped only at the task's RLIMIT_NPROC (io-wq.c).
>>>>
>>>> Requests are classified in io_prep_async_work (https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L480-L492):
>>>>
>>>> if (req->file && (req->flags & REQ_F_ISREG)) { ... bound ... }
>>>> else if (!req->file || !S_ISBLK(...)) {
>>>> if (def->unbound_nonreg_file) // set for READ/READV
>>>> atomic_or(IO_WQ_WORK_UNBOUND, ...);
>>>> }
>>>>
>>>> However, io_init_req sets req->file = NULL (at https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L2046), and the file is only attached later, at issue time. An IOSQE_ASYNC request goes straight though io_queue_iowq() to io_prep_async_work() while `req->file` is still NULL.
>>>>
>>>> So a read of a regular file ends up routed to the unbound pool, whose cap is RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also marks the work as IO_WQ_WORK_CONCURRENT, which lets each enqueue create a worker instead of waiting for a running one.
>>>>
>>>> 3. Workers get created far faster than the actual I/O needs
>>>>
>>>> In io_wq_dec_running, when a worker blocks and it was the last running worker while work is still queued, the kernel queues creation of a brand-new worker immediately.
>>>>
>>>> It doesn't wake one of the idle workers. On a cold buffer cache almost every read blocks on disk, so this code path fires constantly. As a result, the new worker handles one item and then goes idle. Only the head of the free list gets reused, so the rest of the workers sit idle until WORKER_IDLE_TIMEOUT = 5s expires and then they exit.
>>>>
>>>> As a result, the steady-state thread count is therefore roughly "creation rate × 5 s", not "I/Os in flight". The cap for the workers, as was mentioned earlier, is RLIMIT_NPROC.
>>>>
>>>> Similar threads or bugs from the past
>>>> Justin Pryzby on pgsql-hackers <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg208462.html; (2025-10-17) reported could not read blocks 122..126 ...: Operation canceled from pg_dump under io_uring. He guessed it might be related to the io_uring file descriptors leaking after crash-restarts, but I cannot find a resolution.
>>>> Alexander Lakhin's AIO stress test <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196158.html; (pgsql-hackers, April 2025, before 18.0) hit could not read blocks 0..0 in file "global/1213": Operation canceled with parallel createdb/dropdb jobs, on kernels 6.8 and 6.11. The bugs fixed later in that thread (1 <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196804.html;, 2 <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg197025.html;) were different ones, a pg_aios crash and a PANIC from interrupt handling. As far as I can see, the ECANCELED was never explained or fixed.
>>>>
>>>> io_uring bugs that look relevant to me
>>>> liburing #420 <https://github.com/axboe/liburing/issues/420; (2021): a user found IOSQE_ASYNC "spawns a thread for each request, meaning 10000+ threads". There is a commen there that bounded work is limited but unbounded work "really [is] not". That discussion led to creation of IORING_REGISTER_IOWQ_MAX_WORKERS.
>>>> liburing #349 <https://github.com/axboe/liburing/issues/349; (2021) reported "unexpected high count of io_worker threads by using the IOSQE_ASYNC flag".
>>>> Cloudflare, "Missing Manuals – io_uring worker pool" <https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/; explains that unbound workers default to RLIMIT_NPROC for each thread, and that raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS can backfire. The io_uring_register_iowq_max_workers(3) <https://man7.org/linux/man-pages/man3/io_uring_register_iowq_max_workers.3.html; man page says the same.
>>>>
>>>> So, in conclusion, it looks to me that the issue is that req->file is still NULL when an IOSQE_ASYNC request is classified, and the reads end up in the unbound pool, and the fix is either to call io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
>>>>
>>>> Best regards, Dmytro
>>>>
>>>>
>>>> On Sun, Oct 4, 2026 at 10:51 AM weijie JL <weijie1006jl@gmail.com <mailto:weijie1006jl@gmail.com>> wrote:
>>>>> After setting backtrace_functions=md_readv_report, I received more log information:
>>>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG: XX000: could not read blocks 143..158 in file "base/6406098/6407866": Operation canceled
>>>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOCATION: md_readv_report, md.c:2059
>>>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]BACKTRACE:
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x92eafc]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x92f7ec]
>>>>> postgres: postgres test001 [local] CREATE INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x8f209c]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x8f2b63]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x8f0127]
>>>>> postgres: postgres test001 [local] CREATE INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
>>>>> postgres: postgres test001 [local] CREATE INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x5874d6]
>>>>> postgres: postgres test001 [local] CREATE INDEX(heap_getnext+0x88) [0x5878e8]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x58dcbd]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x5b11ae]
>>>>> postgres: postgres test001 [local] CREATE INDEX(btbuild+0xb05) [0x5b7325]
>>>>> postgres: postgres test001 [local] CREATE INDEX(index_build+0x19c) [0x61fa2c]
>>>>> postgres: postgres test001 [local] CREATE INDEX(index_create+0x1328) [0x621608]
>>>>> postgres: postgres test001 [local] CREATE INDEX(DefineIndex+0x9bb) [0x6d11cb]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0xb18111]
>>>>> postgres: postgres test001 [local] CREATE INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
>>>>> /usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x93cc3a]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x93cdaf]
>>>>> postgres: postgres test001 [local] CREATE INDEX(PortalRun+0x291) [0x93d371]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x935ab0]
>>>>> postgres: postgres test001 [local] CREATE INDEX(PostgresMain+0x1f1b) [0x937f6b]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x9389d5]
>>>>> postgres: postgres test001 [local] CREATE INDEX(postmaster_child_launch+0x12d) [0x888ced]
>>>>> postgres: postgres test001 [local] CREATE INDEX() [0x891cb2]
>>>>> postgres: postgres test001 [local] CREATE INDEX(PostmasterMain+0xce9) [0x894309]
>>>>> postgres: postgres test001 [local] CREATE INDEX(main+0x1d0) [0x531bc0]
>>>>> /lib64/libc.so.6(+0x2a610) [0x7f209b82a610]
>>>>> /lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0]
>>>>> postgres: postgres test001 [local] CREATE INDEX(_start+0x25) [0x5321b5]
>>>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]STATEMENT: CREATE INDEX idx_sys_notification_channel_notification_id ON dhr.sys_notification_channel USING btree (c_notification_id);
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> > 2026年9月28日 11:34,PG Bug reporting form <noreply@postgresql.org <mailto:noreply@postgresql.org>> 写道:
>>>>> >
>>>>> > The following bug has been logged on the website:
>>>>> >
>>>>> > Bug reference: 19725
>>>>> > Logged by: weijie JL
>>>>> > Email address: weijie1006jl@gmail.com <mailto:weijie1006jl@gmail.com>
>>>>> > PostgreSQL version: 18.6
>>>>> > Operating system: RockyLinux9
>>>>> > Description:
>>>>> >
>>>>> > Hello,
>>>>> >
>>>>> > I can reproduce a read failure during pg_restore with io_method=io_uring.
>>>>> > The error disappears with worker and recurs after switching back to
>>>>> > io_uring.
>>>>> >
>>>>> > Environment:
>>>>> > - PostgreSQL 18.6, x86_64
>>>>> > - VMware VM: 8 vCPUs, 32 GB RAM
>>>>> > - Kernel: 5.14.0-611.55.1.el9_7.x86_64
>>>>> >
>>>>> > Settings:
>>>>> > effective_io_concurrency = 256
>>>>> > maintenance_io_concurrency = 16
>>>>> > max_parallel_maintenance_workers = 2
>>>>> >
>>>>> > Restore command:
>>>>> > nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &
>>>>> >
>>>>> > One failing statement:
>>>>> > ALTER TABLE ONLY dhr.obj_permission_object
>>>>> > ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);
>>>>> >
>>>>> > Server error:
>>>>> > ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
>>>>> > Operation canceled
>>>>> > LOCATION: md_readv_report, md.c:2059
>>>>> > SQLSTATE: XX000
>>>>> >
>>>>> > Observed test sequence:
>>>>> > 1. io_method=io_uring: the read error occurred during index creation.
>>>>> > 2. Changed io_method to worker: retried without errors.
>>>>> > 3. Changed io_method back to io_uring: the read error was reproduced.
>>>>> >
>>>>> > I have not measured the failure rate or prepared a self-contained
>>>>> > minimal reproducer.
>>>>> >
>>>>> > One earlier run also reported "could not map dynamic shared memory
>>>>> > segment" from parallel workers immediately after the read error.
>>>>> > Those DSM errors have not recurred in subsequent failing tests.
>>>>> >
>>>>> > Does this match a known PostgreSQL bug or kernel-side io_uring issue?
>>>>> > What additional logs or tracing would help identify the cause?
>>>>> >
>>>>> > Thank you.
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>>
>>>
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2026-11-03 01:22 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-09-28 03:34 BUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker PG Bug reporting form <noreply@postgresql.org>
2026-10-04 09:51 ` weijie JL <weijie1006jl@gmail.com>
2026-10-21 17:44 ` Dmytro Astapov <dastapov@gmail.com>
2026-10-23 07:15 ` weijie JL <weijie1006jl@gmail.com>
2026-10-24 12:07 ` Dmytro Astapov <dastapov@gmail.com>
2026-11-03 01:08 ` Dmytro Astapov <dastapov@gmail.com>
2026-11-03 01:22 ` weijie JL <weijie1006jl@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox