Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xO3F6-00000002tVs-0hT2 for pgsql-bugs@arkaria.postgresql.org; Tue, 03 Nov 2026 01:23:17 +0000 Received: from localhost ([127.0.0.1]:45598 helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xO3F4-00000006jxu-2PaB for pgsql-bugs@arkaria.postgresql.org; Tue, 03 Nov 2026 01:23:14 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]:58584) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xO3F3-00000006jxl-3dvg for pgsql-bugs@lists.postgresql.org; Tue, 03 Nov 2026 01:23:14 +0000 Received: from mail-dy1-x132a.google.com ([2607:f8b0:4864:20::132a]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.98.2) (envelope-from ) id 1xO3F0-00000002FyX-3bYF for pgsql-bugs@lists.postgresql.org; Tue, 03 Nov 2026 01:23:13 +0000 Received: by mail-dy1-x132a.google.com with SMTP id 5a478bee46e88-33fb4680717so5883784eec.1 for ; Mon, 02 Nov 2026 17:23:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1793668990; x=1794273790; darn=lists.postgresql.org; h=references:to:cc:in-reply-to:date:subject:mime-version:content-type :message-id:from:from:to:cc:subject:date:message-id:reply-to :content-type; bh=+t59qSJtPDhHezwlEsR7LKovCKtq6y01NMBuWKRVaJA=; b=PWtdMiBgk92T9X5URn6IHKK6O2d54LKitBymM7oytcFzGJJI3lpJT3SFz6sL3Z3ccA 7AwYqUsnc9hqS3ttPx81WzrXlxyg1o5i63wrHZ7KhVztQJKwf4407Ri7PxZ4EZDd3RfO YHnrrLmTHI4H/JlBCvAoDLgTbIDyanlrxSS0xuw7ydasX3b28GclCcxhvhq4R5YtXvVt ZZUI9mhPffuFdUljAI4LnDM+o0MAb0BnST2TTGiosP0+mDuRojspumrNaIBhW9aZ3u6Z +L2kUAz8Gi7Ta3oEfyhgrOa6JHiW3ylj13R/Gqq7aqJza5RxPSLCjwk3PJdgSgaPFowQ bIIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793668990; x=1794273790; h=references:to:cc:in-reply-to:date:subject:mime-version:content-type :message-id:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+t59qSJtPDhHezwlEsR7LKovCKtq6y01NMBuWKRVaJA=; b=rootR4W2ai+5LFXoskDYHbuJA9oeGZ1wLyv8kcCKITIafx2G//EvLQ/B1ThxcEWcpA 51ua00P99V5QuVbvFSoUm7guNoXMOzWjWhxUkhglP1oEO+W7EYq+lvUiJzx2ce7QJtdZ m4eAYH8zbYpjlz5d9iSO2uKbISSLxXWCiSKxVfYczVZH1EPqVlSqwL2qRLZ8rOhBzDs8 uiBCOs4anf8ZRXzO6sNVJxOJPfSa6hM8rGA1yzRFmJpi5soPX3LBnkIWN2K9i3GbiQKV 7GoGkE2cXCGB65G8aY5c0LNvNl8Tr1PqksSuHfU/jwVB4JS7F9h6tlsSUluwuNzStwgx Gh6w== X-Gm-Message-State: APuU4Z8oeqjxEJHSn6l1xUrWxToY61q+nixPMzKiMYl/LbkNR4n5j16k anKWMMqX7s2mezu2jiQYmqdSdc+26XwKKLcawarLDoixmHaCiE2UcO6NRoqqKbNdEV0= X-Gm-Gg: AYBFou3kl56mSabY2gkiHRs2y3AAqtXIVF4PtT4REbUkRBMq0Kdjax+toeC+i0wgQ6j 0AOkQ5jb4Tvb5YEHd+qJeNje2emf0TlZ/jxIC2nS4HOtUFVwnBaLcvxpRAth0faX9GE1qA9HX/P GWR5C3uK3Bi6tO/7kpyslHGSmijeWxoVanxy9ToFa5f0G1m9YdP3xzxndcGUHvUAwKtkAoZBEMN Fb6u7mxmuMcq6fVMM72ySnTy8tJt1MpCAD1zRqc80BK344n7g4yzPEaw5oyZtnhjpeGZUhTRtep xqciotPvEhoElLZkzi+hXWhjoVi6AgAoaPFlbDlD2vfRJQj/ToXAz2TA1XFfDSifnNNdOibcbP8 Gxjh5rBaigu2rRIU0rmYG+h7GdWmmQs5B2GznITS2DDD6FDTmorUc86wm0Wstnn3k6uApGZJoc6 BzikgpZRzD0mV5ARSUm+Ed5wLDs8R1FeFDOsNjeYAaqMD4fDL4fYjB7UY7K42IaaaLFgcrQtTTD 0fkRdGItnM6vt4nU8g8H4aEktW14GE/2fjALLpQt7VUh3GjIz7bmnggYHICIrfo5c9Ms9p/ajFE y7mTiR0WwS9n76KIFJwA187t8QipUdK6yDHMrbCdI1XFGqvl X-Received: by 2002:a05:693c:880e:10b0:369:779c:6e0b with SMTP id 5a478bee46e88-36a4241bcc0mr19582850eec.36.1793668989419; Mon, 02 Nov 2026 17:23:09 -0800 (PST) Received: from smtpclient.apple ([92.223.30.25]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-36a42a504c6sm42191104eec.21.2026.11.02.17.23.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Nov 2026 17:23:08 -0800 (PST) From: weijie JL Message-Id: <247F2D6E-6C50-4261-AF29-1CB31A17C317@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_7CCAD5E6-D26E-4ED5-B5D7-D941D822FD8D" Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3901.100.1.1.12\)) Subject: Re: BUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker Date: Tue, 3 Nov 2026 09:22:55 +0800 In-Reply-To: Cc: pgsql-bugs@lists.postgresql.org To: Dmytro Astapov References: <19725-a209093fc33dfb3e@postgresql.org> <2C1F5CFC-B46F-4578-B930-1C413B5851FB@gmail.com> <9ABA0472-4ECF-4DA7-A55F-BB08D7B19495@gmail.com> X-Mailer: Apple Mail (2.3901.100.1.1.12) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --Apple-Mail=_7CCAD5E6-D26E-4ED5-B5D7-D941D822FD8D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Thanks for the update and for your detailed investigation. It=E2=80=99s = great to hear that the fix has been merged and backported to the stable = kernel branches. I really appreciate your help in tracking down the issue and following = up on the fix. Best regards, > 2026=E5=B9=B411=E6=9C=883=E6=97=A5 09:08=EF=BC=8CDmytro Astapov = =E5=86=99=E9=81=93=EF=BC=9A >=20 > JFYI, the kernel fix for this was merged into Linux 7.4 this Sat = (2026-10-31), and the fix was included in all stable kernel versions as = well. >=20 > Best regards, Dmytro >=20 >=20 > On Sat, Oct 24, 2026 at 1:07=E2=80=AFPM Dmytro Astapov = > wrote: >> Hi, >> A follow-up to my earlier message. On further examination, this is a = kernel bug, not a Postgres bug. The changes I suggested for Postgres = (calling io_uring_register_iowq_max_workers(), or not setting = IOSQE_ASYNC) would only be workarounds, and my suggestion was therefore = wrong. >> As a quick recap, Postgres assumes that a buffered read of a regular = file, submitted with IOSQE_ASYNC, goes to io_uring's bounded worker = pool. This assumption is valid. That pool is limited to min(SQ ring = size, 4 =C3=97 CPUs), which for Postgres is about io_max_concurrency = workers per backend. This is how io_uring is meant to work: reads and = writes of regular files and block devices are bounded work. >> The kernel instead sends these requests to the unbound pool, whose = only limit is RLIMIT_NPROC. It decides the pool in io_prep_async_work(), = but for an IOSQE_ASYNC request the file has not been looked up yet at = that point (req->file is NULL), so the read is treated as "not a regular = file". With enough backends and a cold cache, the worker threads use up = RLIMIT_NPROC. After that, fork() fails and queued reads complete with = -ECANCELED. >> This bug was introduced in Linux 5.18 by commit 6bf9c47a3989 = ("io_uring: defer file assignment"), which was also marked for 5.15+ = stable kernels: = https://github.com/torvalds/linux/commit/6bf9c47a398911e0ab920e36211515359= 6c80432 >> A fix was made on 2025-10-15, and is queued for 7.4 in Jens Axboe's = tree, as commit 98bc044d5a53 ("io_uring: look up the file before = queueing to io-wq"): = https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/commit/?id= =3D98bc044d5a5374f8d3605fe55a8e2df6d6fc3f82 >> It is not (yet?) tagged for stable releases, so older kernels might = only get it if someone backports it. Until then, io_method =3D worker is = the safe choice, as you have already done. >> I have a small C program that reproduces the issue (no Postgres = needed), which could be used to check the if host is susceptible to this = problem. Let me know if anyone wants it. >>=20 >> Best regards, Dmytro >>=20 >>=20 >> On Fri, Oct 23, 2026 at 8:16=E2=80=AFAM weijie JL = > wrote: >>> Hi Dmytro, >>>=20 >>> Thanks for reproducing the issue and for the detailed analysis and = proposed fixes. This is very helpful. >>>=20 >>> Over the past few days, normal SQL queries in our test environment = have also triggered this error. To work around the issue in the current = version, I=E2=80=99ve switched all our production environments to the = default io_method =3D worker setting. >>>=20 >>>=20 >>>> 2026=E5=B9=B410=E6=9C=8822=E6=97=A5 01:44=EF=BC=8CDmytro Astapov = > =E5=86=99=E9=81=93=EF=BC=9A= >>>>=20 >>>> Hi! >>>>=20 >>>> I can reproduce this on version 18.6 + kernel 6.12.109.el8, and I = think that I have the explanation for how it happens + proposal for the = fix as well. >>>>=20 >>>> What I think is happening: postgres io_uring requests hit the = kernel codepath where new io workers are spawned up to RLIMIT_NPRPOC. = Once this limit is hit, no new workers can't be spawned, but also = postgresmaster can't fork() or system() anymore, leading to "could not = read blocks X..Y in file ...: Operation canceled" >>>>=20 >>>> How it could be fixed: postgres should either call = io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads. >>>>=20 >>>> Detailed description of the chain of events follows: >>>>=20 >>>> 1. Postgres asks io_uring to send buffered reads to worker threads >>>>=20 >>>> In method_io_uring.c (line 441), once a backend already has more = than 4 I/Os in flight, every further buffered read gets IOSQE_ASYNC: >>>>=20 >>>> if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED)) >>>> io_uring_sqe_set_flags(sqe, IOSQE_ASYNC); >>>>=20 >>>> If `maintenance_io_concurrency` is higher that 4, then all of the = maintenance reads starting from 5th will take this path.=20 >>>>=20 >>>> 2. The kernel places these reads into the pool whose only cap is = RLIMIT_NPROC >>>>=20 >>>> Each task's io_uring worker pool has two halves: >>>> - bounded (meant for regular files), capped at min(sq_entries, = 4*ncpu); >>>> - unbound, capped only at the task's RLIMIT_NPROC (io-wq.c). >>>>=20 >>>> Requests are classified in io_prep_async_work = (https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L480-L49= 2): >>>>=20 >>>> if (req->file && (req->flags & REQ_F_ISREG)) { ... bound ... } >>>> else if (!req->file || !S_ISBLK(...)) { >>>> if (def->unbound_nonreg_file) // set for READ/READV >>>> atomic_or(IO_WQ_WORK_UNBOUND, ...); >>>> } >>>>=20 >>>> However, io_init_req sets req->file =3D NULL (at = https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L2046), = and the file is only attached later, at issue time. An IOSQE_ASYNC = request goes straight though io_queue_iowq() to io_prep_async_work() = while `req->file` is still NULL.=20 >>>>=20 >>>> So a read of a regular file ends up routed to the unbound pool, = whose cap is RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also = marks the work as IO_WQ_WORK_CONCURRENT, which lets each enqueue create = a worker instead of waiting for a running one. >>>>=20 >>>> 3. Workers get created far faster than the actual I/O needs >>>>=20 >>>> In io_wq_dec_running, when a worker blocks and it was the last = running worker while work is still queued, the kernel queues creation of = a brand-new worker immediately.=20 >>>>=20 >>>> It doesn't wake one of the idle workers. On a cold buffer cache = almost every read blocks on disk, so this code path fires constantly. As = a result, the new worker handles one item and then goes idle. Only the = head of the free list gets reused, so the rest of the workers sit idle = until WORKER_IDLE_TIMEOUT =3D 5s expires and then they exit. >>>>=20 >>>> As a result, the steady-state thread count is therefore roughly = "creation rate =C3=97 5 s", not "I/Os in flight". The cap for the = workers, as was mentioned earlier, is RLIMIT_NPROC. >>>>=20 >>>> Similar threads or bugs from the past >>>> Justin Pryzby on pgsql-hackers = (2025-10-17) reported could not read blocks 122..126 ...: = Operation canceled from pg_dump under io_uring. He guessed it might be = related to the io_uring file descriptors leaking after crash-restarts, = but I cannot find a resolution. >>>> Alexander Lakhin's AIO stress test = (pgsql-hackers, April 2025, before 18.0) hit could not read = blocks 0..0 in file "global/1213": Operation canceled with parallel = createdb/dropdb jobs, on kernels 6.8 and 6.11. The bugs fixed later in = that thread (1 = , 2 = ) were different ones, a pg_aios crash and a PANIC from interrupt = handling. As far as I can see, the ECANCELED was never explained or = fixed. >>>>=20 >>>> io_uring bugs that look relevant to me >>>> liburing #420 = (2021): a user found IOSQE_ASYNC "spawns a thread for each request, = meaning 10000+ threads". There is a commen there that bounded work is = limited but unbounded work "really [is] not". That discussion led to = creation of IORING_REGISTER_IOWQ_MAX_WORKERS. >>>> liburing #349 (2021) = reported "unexpected high count of io_worker threads by using the = IOSQE_ASYNC flag". >>>> Cloudflare, "Missing Manuals =E2=80=93 io_uring worker pool" = = explains that unbound workers default to RLIMIT_NPROC for each thread, = and that raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS = can backfire. The io_uring_register_iowq_max_workers(3) = man page says the same. >>>>=20 >>>> So, in conclusion, it looks to me that the issue is that req->file = is still NULL when an IOSQE_ASYNC request is classified, and the reads = end up in the unbound pool, and the fix is either to call = io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads. >>>>=20 >>>> Best regards, Dmytro >>>>=20 >>>>=20 >>>> On Sun, Oct 4, 2026 at 10:51=E2=80=AFAM weijie JL = > wrote: >>>>> After setting backtrace_functions=3Dmd_readv_report, I received = more log information: >>>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG: = XX000: could not read blocks 143..158 in file "base/6406098/6407866": = Operation canceled >>>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]LOCATION: md_readv_report, md.c:2059 >>>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]BACKTRACE: =20 >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x92eafc] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x92f7ec] >>>>> postgres: postgres test001 [local] CREATE = INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x8f209c] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x8f2b63] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x8f0127] >>>>> postgres: postgres test001 [local] CREATE = INDEX(WaitReadBuffers+0x1f7) [0x8f5137] >>>>> postgres: postgres test001 [local] CREATE = INDEX(read_stream_next_buffer+0x211) [0x8f6ac1] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x5874d6] >>>>> postgres: postgres test001 [local] CREATE = INDEX(heap_getnext+0x88) [0x5878e8] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x58dcbd] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x5b11ae] >>>>> postgres: postgres test001 [local] CREATE = INDEX(btbuild+0xb05) [0x5b7325] >>>>> postgres: postgres test001 [local] CREATE = INDEX(index_build+0x19c) [0x61fa2c] >>>>> postgres: postgres test001 [local] CREATE = INDEX(index_create+0x1328) [0x621608] >>>>> postgres: postgres test001 [local] CREATE = INDEX(DefineIndex+0x9bb) [0x6d11cb] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0xb18111] >>>>> postgres: postgres test001 [local] CREATE = INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8] >>>>> /usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) = [0x7f209b366375] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x93cc3a] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x93cdaf] >>>>> postgres: postgres test001 [local] CREATE = INDEX(PortalRun+0x291) [0x93d371] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x935ab0] >>>>> postgres: postgres test001 [local] CREATE = INDEX(PostgresMain+0x1f1b) [0x937f6b] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x9389d5] >>>>> postgres: postgres test001 [local] CREATE = INDEX(postmaster_child_launch+0x12d) [0x888ced] >>>>> postgres: postgres test001 [local] CREATE INDEX() = [0x891cb2] >>>>> postgres: postgres test001 [local] CREATE = INDEX(PostmasterMain+0xce9) [0x894309] >>>>> postgres: postgres test001 [local] CREATE = INDEX(main+0x1d0) [0x531bc0] >>>>> /lib64/libc.so.6(+0x2a610) [0x7f209b82a610] >>>>> /lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0] >>>>> postgres: postgres test001 [local] CREATE = INDEX(_start+0x25) [0x5321b5] >>>>> 2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]STATEMENT: CREATE INDEX = idx_sys_notification_channel_notification_id ON = dhr.sys_notification_channel USING btree (c_notification_id); >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> > 2026=E5=B9=B49=E6=9C=8828=E6=97=A5 11:34=EF=BC=8CPG Bug = reporting form > = =E5=86=99=E9=81=93=EF=BC=9A >>>>> >=20 >>>>> > The following bug has been logged on the website: >>>>> >=20 >>>>> > Bug reference: 19725 >>>>> > Logged by: weijie JL >>>>> > Email address: weijie1006jl@gmail.com = >>>>> > PostgreSQL version: 18.6 >>>>> > Operating system: RockyLinux9 >>>>> > Description: =20 >>>>> >=20 >>>>> > Hello, >>>>> >=20 >>>>> > I can reproduce a read failure during pg_restore with = io_method=3Dio_uring. >>>>> > The error disappears with worker and recurs after switching back = to >>>>> > io_uring. >>>>> >=20 >>>>> > Environment: >>>>> > - PostgreSQL 18.6, x86_64 >>>>> > - VMware VM: 8 vCPUs, 32 GB RAM >>>>> > - Kernel: 5.14.0-611.55.1.el9_7.x86_64 >>>>> >=20 >>>>> > Settings: >>>>> > effective_io_concurrency =3D 256 >>>>> > maintenance_io_concurrency =3D 16 >>>>> > max_parallel_maintenance_workers =3D 2 >>>>> >=20 >>>>> > Restore command: >>>>> > nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 & >>>>> >=20 >>>>> > One failing statement: >>>>> > ALTER TABLE ONLY dhr.obj_permission_object >>>>> > ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id); >>>>> >=20 >>>>> > Server error: >>>>> > ERROR: could not read blocks 143..158 in file = "base/1684908/1686127": >>>>> > Operation canceled >>>>> > LOCATION: md_readv_report, md.c:2059 >>>>> > SQLSTATE: XX000 >>>>> >=20 >>>>> > Observed test sequence: >>>>> > 1. io_method=3Dio_uring: the read error occurred during index = creation. >>>>> > 2. Changed io_method to worker: retried without errors. >>>>> > 3. Changed io_method back to io_uring: the read error was = reproduced. >>>>> >=20 >>>>> > I have not measured the failure rate or prepared a = self-contained >>>>> > minimal reproducer. >>>>> >=20 >>>>> > One earlier run also reported "could not map dynamic shared = memory >>>>> > segment" from parallel workers immediately after the read error. >>>>> > Those DSM errors have not recurred in subsequent failing tests. >>>>> >=20 >>>>> > Does this match a known PostgreSQL bug or kernel-side io_uring = issue? >>>>> > What additional logs or tracing would help identify the cause? >>>>> >=20 >>>>> > Thank you. >>>>> >=20 >>>>> >=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>=20 --Apple-Mail=_7CCAD5E6-D26E-4ED5-B5D7-D941D822FD8D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

Thanks for the update and for your detailed = investigation. It=E2=80=99s great to hear that the fix has been merged = and backported to the stable kernel branches.

I really appreciate your help in tracking down the issue and = following up on the fix.

Best = regards,


2026=E5=B9=B411=E6=9C=88= 3=E6=97=A5 09:08=EF=BC=8CDmytro Astapov <dastapov@gmail.com> = =E5=86=99=E9=81=93=EF=BC=9A

JFYI, the = kernel fix for this was merged into Linux 7.4 this Sat (2026-10-31), and = the fix was included in all stable kernel versions as = well.

Best regards, = Dmytro


On Sat, Oct = 24, 2026 at 1:07=E2=80=AFPM Dmytro Astapov <dastapov@gmail.com> = wrote:

Hi,

A follow-up to my earlier = message. On further examination, this is a kernel bug, not a Postgres = bug. The changes I suggested for Postgres (calling io_uring_register_iowq_max_workers(), or not setting IOSQE_ASYNC) would only be workarounds, and my suggestion was = therefore wrong.

As a quick recap, Postgres = assumes that a buffered read of a regular file, submitted = with IOSQE_ASYNC, goes to io_uring's bounded worker pool. This = assumption is valid. That pool is limited to min(SQ ring size, 4 =C3=97 = CPUs), which for Postgres is about io_max_concurrency workers per backend. This is how = io_uring is meant to work: reads and writes of regular files and block = devices are bounded work.

The kernel instead sends these = requests to the unbound pool, whose only limit is RLIMIT_NPROC. It decides the pool in io_prep_async_work(), but for an IOSQE_ASYNC request the file has not been looked up yet at = that point (req->file is NULL), so the read is treated as "not a = regular file". With enough backends and a cold cache, the worker threads = use up RLIMIT_NPROC. After that, fork() fails and queued reads complete = with -ECANCELED.

This bug was introduced in Linux = 5.18 by commit 6bf9c47a3989 ("io_uring: defer file assignment"), which = was also marked for 5.15+ stable kernels: https://github.com/torvalds/linux/commit/6bf9c47a398911e= 0ab920e362115153596c80432

A fix was made on 2025-10-15, and = is queued for 7.4 in Jens Axboe's tree, as commit 98bc044d5a53 = ("io_uring: look up the file before queueing to io-wq"): https://git.kernel.org/pub/scm/linux/kernel/git/axboe/li= nux.git/commit/?id=3D98bc044d5a5374f8d3605fe55a8e2df6d6fc3f82

It is not (yet?) tagged for = stable releases, so older kernels might only get it if someone backports = it. Until then, io_method =3D worker is the safe choice, as you have = already done.

I have a small C program that = reproduces the issue (no Postgres needed), which could be used to check the if host is = susceptible to this problem. Let me know if anyone wants = it.


Best regards, = Dmytro


On Fri, Oct = 23, 2026 at 8:16=E2=80=AFAM weijie JL <weijie1006jl@gmail.com> = wrote:

Hi = Dmytro,

Thanks for reproducing the = issue and for the detailed analysis and proposed fixes. This is very = helpful.

Over the past few days, normal = SQL queries in our test environment have also triggered this error. To = work around the issue in the current version, I=E2=80=99ve switched all = our production environments to the default io_method =3D = worker setting.


2026=E5=B9=B410=E6=9C=8822=E6=97=A5 01:44=EF=BC=8CDmytr= o Astapov <dastapov@gmail.com> = =E5=86=99=E9=81=93=EF=BC=9A

Hi!

I can = reproduce this = on version 18.6 + kernel 6.12.109.el8, and I think that I have the = explanation for how it happens + = proposal for the fix as well.

What = I think is happening: postgres io_uring requests hit the kernel = codepath where new io workers are spawned up to RLIMIT_NPRPOC. Once this = limit is hit, no new workers can't be = spawned, but also = postgresmaster can't fork() or system() anymore, leading to = "could not read blocks X..Y in file ...: Operation = canceled"

How = it could be fixed: postgres should either = call io_uring_register_iowq_max_workers= , or don't set IOSQE_ASYNC=  for reads.

Detailed description of the chain of events = follows:

1. = Postgres asks io_uring to send buffered reads to worker = threads

In = method_io_uring.c (line 441), once a backend already has more than 4 = I/Os in flight, every further buffered read gets IOSQE_ASYNC:

if = (in_flight_before > 4 && (ioh->flags & = PGAIO_HF_BUFFERED))
    io_uring_sqe_set_flags(sqe, = IOSQE_ASYNC);

If  = `maintenance_io_concurrency` is higher that 4, then all of the = maintenance reads starting from 5th will take this = path. 

2. = The kernel places these reads into the pool whose only cap is = RLIMIT_NPROC

Each = task's io_uring worker pool has two halves:
- bounded (meant for = regular files), capped at min(sq_entries, 4*ncpu);
- unbound, capped = only at the task's RLIMIT_NPROC (io-wq.c).

Requests are = classified in io_prep_async_work (https://github.com/torvalds/linux/blob/v6.12/io_uring/io= _uring.c#L480-L492):

if (req->file && = (req->flags & REQ_F_ISREG)) { ... bound ... }
else if = (!req->file || !S_ISBLK(...)) {
    if = (def->unbound_nonreg_file)        // set for = READ/READV
        atomic_or(IO_WQ_WORK_UNBOUND, = ...);
}

However, = io_init_req sets req->file =3D NULL (at https://github.com/torvalds/linux/blob/v6.12/io_uring/io= _uring.c#L2046), and the file is only attached later, at issue time. = An IOSQE_ASYNC request goes straight though io_queue_iowq() to = io_prep_async_work() while `req->file` is still = NULL. 

So a = read of a regular file ends up routed to the unbound pool, whose cap is = RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also marks the work = as IO_WQ_WORK_CONCURRENT, which lets each enqueue create a worker = instead of waiting for a running one.

3. Workers get created = far faster than the actual I/O needs

In io_wq_dec_running, when a worker blocks and it was = the last running worker while work is still queued, the kernel queues = creation of a brand-new worker immediately. 

It = doesn't wake one of the idle workers. On a cold buffer cache almost = every read blocks on disk, so this code path fires constantly. As a = result, the new worker handles one item and then goes idle. Only the = head of the free list gets reused, so the rest of the workers sit idle = until WORKER_IDLE_TIMEOUT =3D 5s expires and then they exit.

As a = result, the steady-state thread count is therefore roughly "creation = rate =C3=97 5 s", not "I/Os in flight". The cap for the workers, as was = mentioned earlier, is RLIMIT_NPROC.

Similar threads or bugs = from the past
  • Justin Pryzby on = pgsql-hackers (2025-10-17) reported could not read blocks 122..126 ...: Operation = canceled from pg_dump under io_uring. He guessed it might be related to = the io_uring file descriptors leaking after crash-restarts, but I cannot = find a resolution.
  • Alexander Lakhin's AIO stress = test (pgsql-hackers, April 2025, before 18.0) = hit could not read = blocks 0..0 in file "global/1213": Operation canceled wi= th parallel createdb/dropdb jobs, on kernels 6.8 and 6.11. The bugs = fixed later in that thread (1, 2) were = different ones, a pg_aios crash and a PANIC from interrupt handling. As far = as I can see, the ECANCELED was never explained or = fixed.

io_uring bugs that look = relevant to me
  • liburing #420 (2021): a user = found IOSQE_ASYNC "spawns a thread for each request, meaning = 10000+ threads". There is a commen there that bounded work is limited = but unbounded work "really [is] not". That discussion led to creation = of IORING_REGISTER_IOWQ_MAX_WORKERS.
  • liburing #349 (2021) reported = "unexpected high count of io_worker threads by using the IOSQE_ASYNC = flag".
  • Cloudflare, "Missing Manuals =E2=80=93 io_uring worker = pool" explains that unbound workers default = to RLIMIT_NPROC for each thread, and that raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS can backfire. The io_uring_register_iowq_max_workers(3) man = page says the same.

So, in conclusion, it looks to me that the = issue is that=  req->file=  is still NULL when an IOSQE_ASYNC=  request is classified, and the reads end up in the unbound pool, = and the fix is either = to call io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.

Best = regards, Dmytro


On Sun, Oct 4, 2026 at = 10:51=E2=80=AFAM weijie JL <weijie1006jl@gmail.com> = wrote:
After = setting backtrace_functions=3Dmd_readv_report, I received more log = information:
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG:  = XX000: could not read blocks 143..158 in file "base/6406098/6407866": = Operation canceled
2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]LOCATION:  md_readv_report, md.c:2059
2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]BACKTRACE: 
        postgres: postgres test001 [local] CREATE = INDEX() [0x92eafc]
        postgres: postgres test001 [local] CREATE = INDEX() [0x92f7ec]
        postgres: postgres test001 [local] CREATE = INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
        postgres: postgres test001 [local] CREATE = INDEX() [0x8f209c]
        postgres: postgres test001 [local] CREATE = INDEX() [0x8f2b63]
        postgres: postgres test001 [local] CREATE = INDEX() [0x8f0127]
        postgres: postgres test001 [local] CREATE = INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
        postgres: postgres test001 [local] CREATE = INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
        postgres: postgres test001 [local] CREATE = INDEX() [0x5874d6]
        postgres: postgres test001 [local] CREATE = INDEX(heap_getnext+0x88) [0x5878e8]
        postgres: postgres test001 [local] CREATE = INDEX() [0x58dcbd]
        postgres: postgres test001 [local] CREATE = INDEX() [0x5b11ae]
        postgres: postgres test001 [local] CREATE = INDEX(btbuild+0xb05) [0x5b7325]
        postgres: postgres test001 [local] CREATE = INDEX(index_build+0x19c) [0x61fa2c]
        postgres: postgres test001 [local] CREATE = INDEX(index_create+0x1328) [0x621608]
        postgres: postgres test001 [local] CREATE = INDEX(DefineIndex+0x9bb) [0x6d11cb]
        postgres: postgres test001 [local] CREATE = INDEX() [0xb18111]
        postgres: postgres test001 [local] CREATE = INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
        = /usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
        postgres: postgres test001 [local] CREATE = INDEX() [0x93cc3a]
        postgres: postgres test001 [local] CREATE = INDEX() [0x93cdaf]
        postgres: postgres test001 [local] CREATE = INDEX(PortalRun+0x291) [0x93d371]
        postgres: postgres test001 [local] CREATE = INDEX() [0x935ab0]
        postgres: postgres test001 [local] CREATE = INDEX(PostgresMain+0x1f1b) [0x937f6b]
        postgres: postgres test001 [local] CREATE = INDEX() [0x9389d5]
        postgres: postgres test001 [local] CREATE = INDEX(postmaster_child_launch+0x12d) [0x888ced]
        postgres: postgres test001 [local] CREATE = INDEX() [0x891cb2]
        postgres: postgres test001 [local] CREATE = INDEX(PostmasterMain+0xce9) [0x894309]
        postgres: postgres test001 [local] CREATE = INDEX(main+0x1d0) [0x531bc0]
        /lib64/libc.so.6(+0x2a610) = [0x7f209b82a610]
        /lib64/libc.so.6(__libc_start_main+0x80) = [0x7f209b82a6c0]
        postgres: postgres test001 [local] CREATE = INDEX(_start+0x25) [0x5321b5]
2026-10-04 17:46:08.322 CST [local] test001 postgres = [154450]STATEMENT:  CREATE INDEX = idx_sys_notification_channel_notification_id ON = dhr.sys_notification_channel USING btree (c_notification_id);





> 2026=E5=B9=B49=E6=9C=8828=E6=97=A5 11:34=EF=BC=8CPG Bug reporting = form <noreply@postgresql.org> =E5=86=99=E9=81=93=EF=BC=9A=
>
> The following bug has been logged on the website:
>
> Bug reference:      19725
> Logged by:          weijie JL
> Email address:      weijie1006jl@gmail.com
> PostgreSQL version: 18.6
> Operating system:   RockyLinux9
> Description:       
>
> Hello,
>
> I can reproduce a read failure during pg_restore with = io_method=3Dio_uring.
> The error disappears with worker and recurs after switching back = to
> io_uring.
>
> Environment:
> - PostgreSQL 18.6, x86_64
> - VMware VM: 8 vCPUs, 32 GB RAM
> - Kernel: 5.14.0-611.55.1.el9_7.x86_64
>
> Settings:
> effective_io_concurrency =3D 256
> maintenance_io_concurrency =3D 16
> max_parallel_maintenance_workers =3D 2
>
> Restore command:
> nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log = 2>&1 &
>
> One failing statement:
> ALTER TABLE ONLY dhr.obj_permission_object
>    ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY = (id);
>
> Server error:
> ERROR: could not read blocks 143..158 in file = "base/1684908/1686127":
> Operation canceled
> LOCATION: md_readv_report, md.c:2059
> SQLSTATE: XX000
>
> Observed test sequence:
> 1. io_method=3Dio_uring: the read error occurred during index = creation.
> 2. Changed io_method to worker: retried without errors.
> 3. Changed io_method back to io_uring: the read error was = reproduced.
>
> I have not measured the failure rate or prepared a = self-contained
> minimal reproducer.
>
> One earlier run also reported "could not map dynamic shared = memory
> segment" from parallel workers immediately after the read = error.
> Those DSM errors have not recurred in subsequent failing tests.
>
> Does this match a known PostgreSQL bug or kernel-side io_uring = issue?
> What additional logs or tracing would help identify the cause?
>
> Thank you.
>
>





= --Apple-Mail=_7CCAD5E6-D26E-4ED5-B5D7-D941D822FD8D--