Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xK9VW-00000000fol-3fTa for pgsql-bugs@arkaria.postgresql.org; Fri, 23 Oct 2026 07:16:07 +0000 Received: from localhost ([127.0.0.1]:45538 helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xK9VV-0000000EhgE-2lZb for pgsql-bugs@arkaria.postgresql.org; Fri, 23 Oct 2026 07:16:05 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]:37422) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xK9VV-0000000Ehg5-1I60 for pgsql-bugs@lists.postgresql.org; Fri, 23 Oct 2026 07:16:05 +0000 Received: from mail-dy1-x1329.google.com ([2607:f8b0:4864:20::1329]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.98.2) (envelope-from ) id 1xK9VS-00000000Xcq-3Rgi for pgsql-bugs@lists.postgresql.org; Fri, 23 Oct 2026 07:16:04 +0000 Received: by mail-dy1-x1329.google.com with SMTP id 5a478bee46e88-33bf40b0c8bso154254eec.0 for ; Fri, 23 Oct 2026 00:16:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1792739760; x=1793344560; darn=lists.postgresql.org; h=references:to:cc:in-reply-to:date:subject:mime-version:content-type :message-id:from:from:to:cc:subject:date:message-id:reply-to :content-type; bh=BifiOtiNE1Jk7DThQFqanzO+MEwiC0YOiPXU1yJCA04=; b=d/KXa51byESSaIiwyo+gspwr6dWXz5m5epfywT51FvAVKsDL0MUq7Vom2ZAN/rmfkr W0hdgJZ7lD4uHVqotBE72Hu9B0aDs05qgeUU1Kh+d3ue5NkRLII4Lu5x3MB/t2vI6cem yazAygYPsQii1Zs25irdAsZGAjyRSNnoTYJPKFNH0SmZF24oe09gQWkYAif9TlwZ0IvF UQsP3o0stgdjSnZ4ZFfCgxRQLMgNUwSUNp/7WA6WtQ1pBYAmHg/ZKvHBL+eRgCpOHaPJ 757QKj/c9unk+oqlTD2i7KpNWvAWGD6TuqoH+3aTwlEXu5FcJZabRiiOUTRoORkXViZJ k2+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792739760; x=1793344560; h=references:to:cc:in-reply-to:date:subject:mime-version:content-type :message-id:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=BifiOtiNE1Jk7DThQFqanzO+MEwiC0YOiPXU1yJCA04=; b=yQH5pjP2VNwvX0RRSriZDbGIrR1dqPa8dJT9rt/T7a4kjChzfJuXpLiM9a/caDi4Z6 9CoCLC02jDFlIT5eaMe33yCHwwMm911919EZBGVyHWD6pQkJuKns1XTydpi7jP0pdmmw qxbteNmVVGvqC/z6ZzsGcfTIxMG/hpPjD8anpV755cRSS0DsTK7mkhrK2JzcOzoUENNy ryL6AVs+fQk2bDffjyv30gdbmMabLtRGmHwefO/YPJfJQ7lt5VqdSe5BWYeyDcoElb1W jDbwm2S0J7XHBTiwAgDhsNyQK6vTqSZOYy4t9OVOf2KkIQQCtQnTpZXWKVyyePkwPpDV n3cQ== X-Gm-Message-State: AFq9FYJxJ9YmG3+w7D6SpliHhM8ZZORQtXiY1lXPQdPBIBaBvJjV1sK6 lJqHEvF87WuGWt024wNWSEayJqc3KfDe5a/5jLoglDFdV2Jd3NzG9gbiyg6Rcq/of18= X-Gm-Gg: AYBFou35oLVBwxWhSKkePfg75QC0NlBeeeAWEz5ZUF4DMCetK6Xnxh20DxHF1Ps5ZzM SUVpa5+GS9qaAySyfSOz7p57sgsElMdq3stlNUUPmBM/LrICQPPN9BpfKNt+ERefKLdthQZk94N +4y5Ewg41b8Y1UUoz7eT02U7ynDAwjEpAGMDNH2cV+0XCn+bltyt+BW5t/iHAv6ZtcwVhrc7xJy bs9WKa2ZRT6GGZzV7Q/jGoRTKQH+/X8qOwxVxFIZHqEeBpz05dqDxdJ6vvbFQXxO68g9573zwX8 jgTvLN9E+dSj158onVmoA7INtKn+etcTaeEPPLCyvaX30cLd29RdZhCtdZHLpAhLL8h/EK/SvFV pO1+r+7NtCWfYiftqSRaHAO2QLvJULufq0xjgIRfuyRs5l5FcBTaVKj89MKKZy2tyrAhYnmG6Zv aJuUp0M+kL+GXWGz74Tfkn8+MgM6D/k5BotXtgzBm8f5kYAwYcW0RsLk6bkoY+l0XGAiL6alzoZ aBrj9udNn6VzwBMKFkB+VxekcngYg/mHB+g9VEGdKhlLgoZVZzT7aivxW/1oYw5BwLS7UMN6wUd qol9WIf717O0VQwEc6tb8Q== X-Received: by 2002:a05:693c:4152:10b0:369:9af5:7bc7 with SMTP id 5a478bee46e88-369d971fe22mr2230184eec.12.1792739759155; Fri, 23 Oct 2026 00:15:59 -0700 (PDT) Received: from smtpclient.apple ([92.223.30.25]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-369d9e1a085sm4541897eec.29.2026.10.23.00.15.57 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Oct 2026 00:15:58 -0700 (PDT) From: weijie JL Message-Id: <9ABA0472-4ECF-4DA7-A55F-BB08D7B19495@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_35D5764D-0C26-45EB-AB63-1FD9AFD9847F" Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3901.100.1.1.12\)) Subject: Re: BUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker Date: Fri, 23 Oct 2026 15:15:43 +0800 In-Reply-To: Cc: pgsql-bugs@lists.postgresql.org To: Dmytro Astapov References: <19725-a209093fc33dfb3e@postgresql.org> <2C1F5CFC-B46F-4578-B930-1C413B5851FB@gmail.com> X-Mailer: Apple Mail (2.3901.100.1.1.12) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --Apple-Mail=_35D5764D-0C26-45EB-AB63-1FD9AFD9847F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Dmytro, Thanks for reproducing the issue and for the detailed analysis and = proposed fixes. This is very helpful. Over the past few days, normal SQL queries in our test environment have = also triggered this error. To work around the issue in the current = version, I=E2=80=99ve switched all our production environments to the = default io_method =3D worker setting. > 2026=E5=B9=B410=E6=9C=8822=E6=97=A5 01:44=EF=BC=8CDmytro Astapov = =E5=86=99=E9=81=93=EF=BC=9A >=20 > Hi! >=20 > I can reproduce this on version 18.6 + kernel 6.12.109.el8, and I = think that I have the explanation for how it happens + proposal for the = fix as well. >=20 > What I think is happening: postgres io_uring requests hit the kernel = codepath where new io workers are spawned up to RLIMIT_NPRPOC. Once this = limit is hit, no new workers can't be spawned, but also postgresmaster = can't fork() or system() anymore, leading to "could not read blocks X..Y = in file ...: Operation canceled" >=20 > How it could be fixed: postgres should either call = io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads. >=20 > Detailed description of the chain of events follows: >=20 > 1. Postgres asks io_uring to send buffered reads to worker threads >=20 > In method_io_uring.c (line 441), once a backend already has more than = 4 I/Os in flight, every further buffered read gets IOSQE_ASYNC: >=20 > if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED)) > io_uring_sqe_set_flags(sqe, IOSQE_ASYNC); >=20 > If `maintenance_io_concurrency` is higher that 4, then all of the = maintenance reads starting from 5th will take this path.=20 >=20 > 2. The kernel places these reads into the pool whose only cap is = RLIMIT_NPROC >=20 > Each task's io_uring worker pool has two halves: > - bounded (meant for regular files), capped at min(sq_entries, = 4*ncpu); > - unbound, capped only at the task's RLIMIT_NPROC (io-wq.c). >=20 > Requests are classified in io_prep_async_work = (https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L480-L49= 2): >=20 > if (req->file && (req->flags & REQ_F_ISREG)) { ... bound ... } > else if (!req->file || !S_ISBLK(...)) { > if (def->unbound_nonreg_file) // set for READ/READV > atomic_or(IO_WQ_WORK_UNBOUND, ...); > } >=20 > However, io_init_req sets req->file =3D NULL (at = https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L2046), = and the file is only attached later, at issue time. An IOSQE_ASYNC = request goes straight though io_queue_iowq() to io_prep_async_work() = while `req->file` is still NULL.=20 >=20 > So a read of a regular file ends up routed to the unbound pool, whose = cap is RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also marks = the work as IO_WQ_WORK_CONCURRENT, which lets each enqueue create a = worker instead of waiting for a running one. >=20 > 3. Workers get created far faster than the actual I/O needs >=20 > In io_wq_dec_running, when a worker blocks and it was the last running = worker while work is still queued, the kernel queues creation of a = brand-new worker immediately.=20 >=20 > It doesn't wake one of the idle workers. On a cold buffer cache almost = every read blocks on disk, so this code path fires constantly. As a = result, the new worker handles one item and then goes idle. Only the = head of the free list gets reused, so the rest of the workers sit idle = until WORKER_IDLE_TIMEOUT =3D 5s expires and then they exit. >=20 > As a result, the steady-state thread count is therefore roughly = "creation rate =C3=97 5 s", not "I/Os in flight". The cap for the = workers, as was mentioned earlier, is RLIMIT_NPROC. >=20 > Similar threads or bugs from the past > Justin Pryzby on pgsql-hackers = (2025-10-17) reported could not read blocks 122..126 ...: = Operation canceled from pg_dump under io_uring. He guessed it might be = related to the io_uring file descriptors leaking after crash-restarts, = but I cannot find a resolution. > Alexander Lakhin's AIO stress test = (pgsql-hackers, April 2025, before 18.0) hit could not read = blocks 0..0 in file "global/1213": Operation canceled with parallel = createdb/dropdb jobs, on kernels 6.8 and 6.11. The bugs fixed later in = that thread (1 = , 2 = ) were different ones, a pg_aios crash and a PANIC from interrupt = handling. As far as I can see, the ECANCELED was never explained or = fixed. >=20 > io_uring bugs that look relevant to me > liburing #420 (2021): a = user found IOSQE_ASYNC "spawns a thread for each request, meaning 10000+ = threads". There is a commen there that bounded work is limited but = unbounded work "really [is] not". That discussion led to creation of = IORING_REGISTER_IOWQ_MAX_WORKERS. > liburing #349 (2021) = reported "unexpected high count of io_worker threads by using the = IOSQE_ASYNC flag". > Cloudflare, "Missing Manuals =E2=80=93 io_uring worker pool" = = explains that unbound workers default to RLIMIT_NPROC for each thread, = and that raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS = can backfire. The io_uring_register_iowq_max_workers(3) = man page says the same. >=20 > So, in conclusion, it looks to me that the issue is that req->file is = still NULL when an IOSQE_ASYNC request is classified, and the reads end = up in the unbound pool, and the fix is either to call = io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads. >=20 > Best regards, Dmytro >=20 >=20 > On Sun, Oct 4, 2026 at 10:51=E2=80=AFAM weijie JL = > wrote: >> After setting backtrace_functions=3Dmd_readv_report, I received more = log information: >> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG: = XX000: could not read blocks 143..158 in file "base/6406098/6407866": = Operation canceled >> 2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]LOCATION: md_readv_report, md.c:2059 >> 2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]BACKTRACE: =20 >> postgres: postgres test001 [local] CREATE INDEX() [0x92eafc] >> postgres: postgres test001 [local] CREATE INDEX() [0x92f7ec] >> postgres: postgres test001 [local] CREATE = INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea] >> postgres: postgres test001 [local] CREATE INDEX() [0x8f209c] >> postgres: postgres test001 [local] CREATE INDEX() [0x8f2b63] >> postgres: postgres test001 [local] CREATE INDEX() [0x8f0127] >> postgres: postgres test001 [local] CREATE = INDEX(WaitReadBuffers+0x1f7) [0x8f5137] >> postgres: postgres test001 [local] CREATE = INDEX(read_stream_next_buffer+0x211) [0x8f6ac1] >> postgres: postgres test001 [local] CREATE INDEX() [0x5874d6] >> postgres: postgres test001 [local] CREATE = INDEX(heap_getnext+0x88) [0x5878e8] >> postgres: postgres test001 [local] CREATE INDEX() [0x58dcbd] >> postgres: postgres test001 [local] CREATE INDEX() [0x5b11ae] >> postgres: postgres test001 [local] CREATE = INDEX(btbuild+0xb05) [0x5b7325] >> postgres: postgres test001 [local] CREATE = INDEX(index_build+0x19c) [0x61fa2c] >> postgres: postgres test001 [local] CREATE = INDEX(index_create+0x1328) [0x621608] >> postgres: postgres test001 [local] CREATE = INDEX(DefineIndex+0x9bb) [0x6d11cb] >> postgres: postgres test001 [local] CREATE INDEX() [0xb18111] >> postgres: postgres test001 [local] CREATE = INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8] >> /usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) = [0x7f209b366375] >> postgres: postgres test001 [local] CREATE INDEX() [0x93cc3a] >> postgres: postgres test001 [local] CREATE INDEX() [0x93cdaf] >> postgres: postgres test001 [local] CREATE = INDEX(PortalRun+0x291) [0x93d371] >> postgres: postgres test001 [local] CREATE INDEX() [0x935ab0] >> postgres: postgres test001 [local] CREATE = INDEX(PostgresMain+0x1f1b) [0x937f6b] >> postgres: postgres test001 [local] CREATE INDEX() [0x9389d5] >> postgres: postgres test001 [local] CREATE = INDEX(postmaster_child_launch+0x12d) [0x888ced] >> postgres: postgres test001 [local] CREATE INDEX() [0x891cb2] >> postgres: postgres test001 [local] CREATE = INDEX(PostmasterMain+0xce9) [0x894309] >> postgres: postgres test001 [local] CREATE INDEX(main+0x1d0) = [0x531bc0] >> /lib64/libc.so.6(+0x2a610) [0x7f209b82a610] >> /lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0] >> postgres: postgres test001 [local] CREATE INDEX(_start+0x25) = [0x5321b5] >> 2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]STATEMENT: CREATE INDEX = idx_sys_notification_channel_notification_id ON = dhr.sys_notification_channel USING btree (c_notification_id); >>=20 >>=20 >>=20 >>=20 >>=20 >> > 2026=E5=B9=B49=E6=9C=8828=E6=97=A5 11:34=EF=BC=8CPG Bug reporting = form > =E5=86=99=E9= =81=93=EF=BC=9A >> >=20 >> > The following bug has been logged on the website: >> >=20 >> > Bug reference: 19725 >> > Logged by: weijie JL >> > Email address: weijie1006jl@gmail.com = >> > PostgreSQL version: 18.6 >> > Operating system: RockyLinux9 >> > Description: =20 >> >=20 >> > Hello, >> >=20 >> > I can reproduce a read failure during pg_restore with = io_method=3Dio_uring. >> > The error disappears with worker and recurs after switching back to >> > io_uring. >> >=20 >> > Environment: >> > - PostgreSQL 18.6, x86_64 >> > - VMware VM: 8 vCPUs, 32 GB RAM >> > - Kernel: 5.14.0-611.55.1.el9_7.x86_64 >> >=20 >> > Settings: >> > effective_io_concurrency =3D 256 >> > maintenance_io_concurrency =3D 16 >> > max_parallel_maintenance_workers =3D 2 >> >=20 >> > Restore command: >> > nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 & >> >=20 >> > One failing statement: >> > ALTER TABLE ONLY dhr.obj_permission_object >> > ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id); >> >=20 >> > Server error: >> > ERROR: could not read blocks 143..158 in file = "base/1684908/1686127": >> > Operation canceled >> > LOCATION: md_readv_report, md.c:2059 >> > SQLSTATE: XX000 >> >=20 >> > Observed test sequence: >> > 1. io_method=3Dio_uring: the read error occurred during index = creation. >> > 2. Changed io_method to worker: retried without errors. >> > 3. Changed io_method back to io_uring: the read error was = reproduced. >> >=20 >> > I have not measured the failure rate or prepared a self-contained >> > minimal reproducer. >> >=20 >> > One earlier run also reported "could not map dynamic shared memory >> > segment" from parallel workers immediately after the read error. >> > Those DSM errors have not recurred in subsequent failing tests. >> >=20 >> > Does this match a known PostgreSQL bug or kernel-side io_uring = issue? >> > What additional logs or tracing would help identify the cause? >> >=20 >> > Thank you. >> >=20 >> >=20 >>=20 >>=20 >>=20 --Apple-Mail=_35D5764D-0C26-45EB-AB63-1FD9AFD9847F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

Hi = Dmytro,

Thanks for reproducing the = issue and for the detailed analysis and proposed fixes. This is very = helpful.

Over the past few days, = normal SQL queries in our test environment have also triggered this = error. To work around the issue in the current version, I=E2=80=99ve = switched all our production environments to the = default io_method =3D = worker setting.


2026=E5=B9=B410=E6=9C=8822=E6=97=A5 01:44=EF=BC=8CDmytr= o Astapov <dastapov@gmail.com> =E5=86=99=E9=81=93=EF=BC=9A

Hi!

I can = reproduce this = on version 18.6 + kernel 6.12.109.el8, and I think that I have the = explanation for how it happens + = proposal for the fix as well.

What = I think is happening: postgres io_uring requests hit the kernel = codepath where new io workers are spawned up to RLIMIT_NPRPOC. Once this = limit is hit, no new workers can't be = spawned, but also = postgresmaster can't fork() or system() anymore, leading to = "could not read blocks X..Y in file ...: Operation = canceled"

How = it could be fixed: postgres should either = call io_uring_register_iowq_max_workers= , or don't set IOSQE_ASYNC=  for reads.

Detailed description of the chain of events = follows:

1. = Postgres asks io_uring to send buffered reads to worker = threads

In = method_io_uring.c (line 441), once a backend already has more than 4 = I/Os in flight, every further buffered read gets IOSQE_ASYNC:

if = (in_flight_before > 4 && (ioh->flags & = PGAIO_HF_BUFFERED))
    io_uring_sqe_set_flags(sqe, = IOSQE_ASYNC);

If  = `maintenance_io_concurrency` is higher that 4, then all of the = maintenance reads starting from 5th will take this = path. 

2. = The kernel places these reads into the pool whose only cap is = RLIMIT_NPROC

Each = task's io_uring worker pool has two halves:
- bounded (meant for = regular files), capped at min(sq_entries, 4*ncpu);
- unbound, capped = only at the task's RLIMIT_NPROC (io-wq.c).

Requests are = classified in io_prep_async_work (https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c= #L480-L492):

if (req->file && (req->flags & = REQ_F_ISREG)) { ... bound ... }
else if (!req->file || = !S_ISBLK(...)) {
    if (def->unbound_nonreg_file) =        // set for READ/READV
      =   atomic_or(IO_WQ_WORK_UNBOUND, = ...);
}

However, = io_init_req sets req->file =3D NULL (at https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L20= 46), and the file is only attached later, at issue time. An = IOSQE_ASYNC request goes straight though io_queue_iowq() to = io_prep_async_work() while `req->file` is still = NULL. 

So a = read of a regular file ends up routed to the unbound pool, whose cap is = RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also marks the work = as IO_WQ_WORK_CONCURRENT, which lets each enqueue create a worker = instead of waiting for a running one.

3. Workers get created = far faster than the actual I/O needs

In io_wq_dec_running, when a worker blocks and it was = the last running worker while work is still queued, the kernel queues = creation of a brand-new worker immediately. 

It = doesn't wake one of the idle workers. On a cold buffer cache almost = every read blocks on disk, so this code path fires constantly. As a = result, the new worker handles one item and then goes idle. Only the = head of the free list gets reused, so the rest of the workers sit idle = until WORKER_IDLE_TIMEOUT =3D 5s expires and then they exit.

As a = result, the steady-state thread count is therefore roughly "creation = rate =C3=97 5 s", not "I/Os in flight". The cap for the workers, as was = mentioned earlier, is RLIMIT_NPROC.

Similar threads or bugs = from the past
  • Justin Pryzby = on pgsql-hackers (2025-10-17) reported could not read blocks 122..126 ...: Operation = canceled from pg_dump under io_uring. He guessed it might be related to = the io_uring file descriptors leaking after crash-restarts, but I cannot = find a resolution.
  • Alexander = Lakhin's AIO stress test (pgsql-hackers, April 2025, = before 18.0) hit could not read blocks = 0..0 in file "global/1213": Operation canceled wi= th parallel createdb/dropdb jobs, on kernels 6.8 and 6.11. The bugs = fixed later in that thread (1, 2) were different ones, = a pg_aios crash and a PANIC from interrupt handling. As far = as I can see, the ECANCELED was never explained or = fixed.

io_uring bugs that look = relevant to me
  • liburing = #420 (2021): a user found IOSQE_ASYNC "spawns a thread for each request, meaning = 10000+ threads". There is a commen there that bounded work is limited = but unbounded work "really [is] not". That discussion led to creation = of IORING_REGISTER_IOWQ_MAX_WORKERS.
  • liburing = #349 (2021) reported "unexpected high count of io_worker = threads by using the IOSQE_ASYNC flag".
  • Cloudflare, = "Missing Manuals =E2=80=93 io_uring worker = pool" explains that unbound workers default = to RLIMIT_NPROC for each thread, and that raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS can backfire. = The io_uring_register_iowq_max_worker= s(3) man page says the same.

So, in conclusion, it looks to me that the = issue is that=  req->file=  is still NULL when an IOSQE_ASYNC=  request is classified, and the reads end up in the unbound pool, = and the fix is either = to call io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.

Best = regards, Dmytro


On Sun, Oct 4, 2026 at 10:51=E2=80=AFAM weijie JL = <weijie1006jl@gmail.com> = wrote:
After = setting backtrace_functions=3Dmd_readv_report, I received more log = information:
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG:  = XX000: could not read blocks 143..158 in file "base/6406098/6407866": = Operation canceled
2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]LOCATION:  md_readv_report, md.c:2059
2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]BACKTRACE: 
        postgres: postgres test001 [local] CREATE = INDEX() [0x92eafc]
        postgres: postgres test001 [local] CREATE = INDEX() [0x92f7ec]
        postgres: postgres test001 [local] CREATE = INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
        postgres: postgres test001 [local] CREATE = INDEX() [0x8f209c]
        postgres: postgres test001 [local] CREATE = INDEX() [0x8f2b63]
        postgres: postgres test001 [local] CREATE = INDEX() [0x8f0127]
        postgres: postgres test001 [local] CREATE = INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
        postgres: postgres test001 [local] CREATE = INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
        postgres: postgres test001 [local] CREATE = INDEX() [0x5874d6]
        postgres: postgres test001 [local] CREATE = INDEX(heap_getnext+0x88) [0x5878e8]
        postgres: postgres test001 [local] CREATE = INDEX() [0x58dcbd]
        postgres: postgres test001 [local] CREATE = INDEX() [0x5b11ae]
        postgres: postgres test001 [local] CREATE = INDEX(btbuild+0xb05) [0x5b7325]
        postgres: postgres test001 [local] CREATE = INDEX(index_build+0x19c) [0x61fa2c]
        postgres: postgres test001 [local] CREATE = INDEX(index_create+0x1328) [0x621608]
        postgres: postgres test001 [local] CREATE = INDEX(DefineIndex+0x9bb) [0x6d11cb]
        postgres: postgres test001 [local] CREATE = INDEX() [0xb18111]
        postgres: postgres test001 [local] CREATE = INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
        = /usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
        postgres: postgres test001 [local] CREATE = INDEX() [0x93cc3a]
        postgres: postgres test001 [local] CREATE = INDEX() [0x93cdaf]
        postgres: postgres test001 [local] CREATE = INDEX(PortalRun+0x291) [0x93d371]
        postgres: postgres test001 [local] CREATE = INDEX() [0x935ab0]
        postgres: postgres test001 [local] CREATE = INDEX(PostgresMain+0x1f1b) [0x937f6b]
        postgres: postgres test001 [local] CREATE = INDEX() [0x9389d5]
        postgres: postgres test001 [local] CREATE = INDEX(postmaster_child_launch+0x12d) [0x888ced]
        postgres: postgres test001 [local] CREATE = INDEX() [0x891cb2]
        postgres: postgres test001 [local] CREATE = INDEX(PostmasterMain+0xce9) [0x894309]
        postgres: postgres test001 [local] CREATE = INDEX(main+0x1d0) [0x531bc0]
        /lib64/libc.so.6(+0x2a610) = [0x7f209b82a610]
        /lib64/libc.so.6(__libc_start_main+0x80) = [0x7f209b82a6c0]
        postgres: postgres test001 [local] CREATE = INDEX(_start+0x25) [0x5321b5]
2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]STATEMENT:  CREATE INDEX = idx_sys_notification_channel_notification_id ON = dhr.sys_notification_channel USING btree (c_notification_id);





> 2026=E5=B9=B49=E6=9C=8828=E6=97=A5 11:34=EF=BC=8CPG Bug reporting = form <noreply@postgresql.org> =E5=86=99=E9=81=93=EF=BC=9A=
>
> The following bug has been logged on the website:
>
> Bug reference:      19725
> Logged by:          weijie JL
> Email address:      weijie1006jl@gmail.com
> PostgreSQL version: 18.6
> Operating system:   RockyLinux9
> Description:       
>
> Hello,
>
> I can reproduce a read failure during pg_restore with = io_method=3Dio_uring.
> The error disappears with worker and recurs after switching back = to
> io_uring.
>
> Environment:
> - PostgreSQL 18.6, x86_64
> - VMware VM: 8 vCPUs, 32 GB RAM
> - Kernel: 5.14.0-611.55.1.el9_7.x86_64
>
> Settings:
> effective_io_concurrency =3D 256
> maintenance_io_concurrency =3D 16
> max_parallel_maintenance_workers =3D 2
>
> Restore command:
> nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log = 2>&1 &
>
> One failing statement:
> ALTER TABLE ONLY dhr.obj_permission_object
>    ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY = (id);
>
> Server error:
> ERROR: could not read blocks 143..158 in file = "base/1684908/1686127":
> Operation canceled
> LOCATION: md_readv_report, md.c:2059
> SQLSTATE: XX000
>
> Observed test sequence:
> 1. io_method=3Dio_uring: the read error occurred during index = creation.
> 2. Changed io_method to worker: retried without errors.
> 3. Changed io_method back to io_uring: the read error was = reproduced.
>
> I have not measured the failure rate or prepared a = self-contained
> minimal reproducer.
>
> One earlier run also reported "could not map dynamic shared = memory
> segment" from parallel workers immediately after the read = error.
> Those DSM errors have not recurred in subsequent failing tests.
>
> Does this match a known PostgreSQL bug or kernel-side io_uring = issue?
> What additional logs or tracing would help identify the cause?
>
> Thank you.
>
>




= --Apple-Mail=_35D5764D-0C26-45EB-AB63-1FD9AFD9847F--