MIME-Version: 1.0
From: SATYANARAYANA NARLAPURAM <satyanarlapuram@gmail.com>
Date: Mon, 13 Apr 2026 18:55:28 -0700
Message-ID: 
 <CAHg+QDdH+uBc0deP7uPE5rBkn0EjdkVC2_8ykK3KV90V4z6_5Q@mail.gmail.com>
Subject: Fix checkpointer PANIC due to missing fsync cancel in mdunlinkfork()
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
 Andres Freund <andres@anarazel.de>,
	Thomas Munro <thomas.munro@gmail.com>
Content-Type: multipart/mixed; boundary="0000000000006b14ee064f61e5e0"
Archived-At: 
 <https://www.postgresql.org/message-id/CAHg%2BQDdH%2BuBc0deP7uPE5rBkn0EjdkVC2_8ykK3KV90V4z6_5Q%40mail.gmail.com>
Precedence: bulk

--0000000000006b14ee064f61e5e0
Content-Type: multipart/alternative; boundary="0000000000006b14ec064f61e5de"

--0000000000006b14ec064f61e5de
Content-Type: text/plain; charset="UTF-8"

Hi hackers,

I think I found a bug in mdunlinkfork() that causes the checkpointer
to PANIC with "could not fsync file ... No such file or directory" when
a relation is moved between tablespaces and the old tablespace is dropped.

mdunlinkfork() has two code paths for unlinking a relation segment file:
(1) The 'if' branch (isRedo, binary upgrade, non-MAIN forknum, or temp
relations) correctly calls register_forget_request() to cancel any pending
fsync before unlinking. (2) The 'else' branch that handles normal
MAIN_FORKNUM unlink is missing register_forget_request().
This leaves a stale SYNC_REQUEST entry in the checkpointer's
pendingOps hash table.

The comment in ProcessSyncRequests() states below:

/*
* The fsync table could contain requests to fsync segments that
* have been deleted (unlinked) by the time we get to them. Rather
* than just hoping an ENOENT (or EACCES on Windows) error can be
* ignored, what we do on error is absorb pending requests and
* then retry. Since mdunlink() queues a "cancel" message before
* actually unlinking, the fsync request is guaranteed to be
* marked canceled after the absorb if it really was this case.
* DROP DATABASE likewise has to tell us to forget fsync requests
* before it starts deletions.
*/

but this guarantee is not provided by the else branch.

Repro:
Reproducing this is not easy as it is a race but the test below
when repeated sufficient number of times you can see ocasionally
checkpointer PANIC. Though I used TABLESPACE for somewhat
consistent repro, this race exists for operations like DROP table,
TRUNCATE table as well I believe.

CREATE TABLESPACE ts LOCATION '/tmp/ts';
CREATE TABLE t (id int, pad text) TABLESPACE ts;
INSERT INTO t SELECT g, repeat('x', 500) FROM generate_series(1,50000) g;

-- Concurrent updates to dirty shared buffers (background sessions)
UPDATE t SET pad = repeat('y', 500) WHERE id <= 25000;

-- Move table away; FlushRelationBuffers sends SYNC_REQUESTs for old path
ALTER TABLE t SET TABLESPACE pg_default;

-- Drop tablespace forces checkpoint + removes directory
DROP TABLESPACE ts;

-- Next checkpoint hits stale entry -> PANIC
CHECKPOINT;

PANIC:  could not fsync file "pg_tblspc/41343/PG_19_202604061/5/41348": No
such file or directory
LOG:  checkpointer process (PID 166749) was terminated by signal 6: Aborted
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing

The fix is to add register_forget_request() in the else branch of
mdunlinkfork(), before register_unlink_segment(), matching what the
'if' branch already does.

Thoughts?

Thanks,
Satya

--0000000000006b14ec064f61e5de
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi hackers,<div><br></div><div>I think I found a bug in md=
unlinkfork() that causes the checkpointer=C2=A0</div><div>to PANIC with &qu=
ot;could not fsync file ... No such file or directory&quot; when=C2=A0</div=
><div>a relation is moved between tablespaces and the old tablespace is dro=
pped.</div><div><br></div><div>mdunlinkfork() has two code paths for unlink=
ing a relation segment file:<br>(1) The &#39;if&#39; branch (isRedo, binary=
 upgrade, non-MAIN forknum, or temp<br>relations) correctly calls register_=
forget_request() to cancel any pending=C2=A0</div><div>fsync before unlinki=
ng. (2) The &#39;else&#39; branch that handles=C2=A0normal=C2=A0</div><div>=
MAIN_FORKNUM unlink is missing=C2=A0register_forget_request().=C2=A0</div><=
div>This leaves a stale SYNC_REQUEST entry in the checkpointer&#39;s=C2=A0<=
/div><div>pendingOps hash table.</div><div><br>The comment in ProcessSyncRe=
quests() states below:</div><div><br></div><div>/*</div><div>* The fsync ta=
ble could contain requests to fsync segments that<br>			 * have been delete=
d (unlinked) by the time we get to them. Rather<br>			 * than just hoping a=
n ENOENT (or EACCES on Windows) error can be<br>			 * ignored, what we do o=
n error is absorb pending requests and<br>			 * then retry. Since mdunlink(=
) queues a &quot;cancel&quot; message before<br>			 * actually unlinking, t=
he fsync request is guaranteed to be<br>			 * marked canceled after the abs=
orb if it really was this case.<br>			 * DROP DATABASE likewise has to tell=
 us to forget fsync requests<br>			 * before it starts deletions.<br>			 */=
</div><div><br></div><div>but this guarantee is not provided by the else br=
anch.<br><div><div><br></div><div>Repro:</div><div>Reproducing this is not =
easy as it is a race but the test below</div><div>when repeated sufficient =
number of times you can see ocasionally</div><div>checkpointer PANIC. Thoug=
h I used TABLESPACE for somewhat</div><div>consistent repro, this race exis=
ts for operations like DROP table,</div><div>TRUNCATE table as well I belie=
ve.=C2=A0</div><div><br></div><div>CREATE TABLESPACE ts LOCATION &#39;/tmp/=
ts&#39;;<br>CREATE TABLE t (id int, pad text) TABLESPACE ts;<br>INSERT INTO=
 t SELECT g, repeat(&#39;x&#39;, 500) FROM generate_series(1,50000) g;<br><=
br>-- Concurrent updates to dirty shared buffers (background sessions)<br>U=
PDATE t SET pad =3D repeat(&#39;y&#39;, 500) WHERE id &lt;=3D 25000;<br><br=
>-- Move table away; FlushRelationBuffers sends SYNC_REQUESTs for old path<=
br>ALTER TABLE t SET TABLESPACE pg_default;<br><br>-- Drop tablespace force=
s checkpoint + removes directory<br>DROP TABLESPACE ts;<br><br>-- Next chec=
kpoint hits stale entry -&gt; PANIC<br>CHECKPOINT;</div><div><br></div><div=
>PANIC: =C2=A0could not fsync file &quot;pg_tblspc/41343/PG_19_202604061/5/=
41348&quot;: No such file or directory<br>LOG: =C2=A0checkpointer process (=
PID 166749) was terminated by signal 6: Aborted<br>LOG: =C2=A0terminating a=
ny other active server processes<br>LOG: =C2=A0all server processes termina=
ted; reinitializing</div></div></div><div><br></div><div>The fix is to add =
register_forget_request() in the else branch of<br>mdunlinkfork(), before r=
egister_unlink_segment(), matching what the<br>&#39;if&#39; branch already =
does.</div><div><br></div><div>Thoughts?</div><div><br></div><div>Thanks,</=
div><div>Satya</div></div>

--0000000000006b14ec064f61e5de--

--0000000000006b14ee064f61e5e0
Content-Type: application/octet-stream;
	name="0001-Fix-checkpointer-PANIC-due-to-missing-fsync-cancel-i.patch"
Content-Disposition: attachment;
	filename="0001-Fix-checkpointer-PANIC-due-to-missing-fsync-cancel-i.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_mnxyxooa0>
X-Attachment-Id: f_mnxyxooa0

RnJvbSA0ODA4OWU2NzhiNDI1MjkxNDQyZjRkYzE0ZDg3Y2EyNWM1MjRmNjVmIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBTYXR5YW5hcmF5YW5hIE5hcmxhcHVyYW0gPHNhdHlhbmFybGFw
dXJhbUBnbWFpbC5jb20+CkRhdGU6IFR1ZSwgMTQgQXByIDIwMjYgMDA6MDk6NDQgKzAwMDAKU3Vi
amVjdDogW1BBVENIXSBGaXggY2hlY2twb2ludGVyIFBBTklDIGR1ZSB0byBtaXNzaW5nIGZzeW5j
IGNhbmNlbCBpbgogbWR1bmxpbmtmb3JrKCkKCkFkZCBtaXNzaW5nIHJlZ2lzdGVyX2ZvcmdldF9y
ZXF1ZXN0KCkgaW4gdGhlICdlbHNlJyBicmFuY2ggb2YKbWR1bmxpbmtmb3JrKCksIGJlZm9yZSBy
ZWdpc3Rlcl91bmxpbmtfc2VnbWVudCgpLCBtYXRjaGluZyB3aGF0IHRoZQonaWYnIGJyYW5jaCBh
bHJlYWR5IGRvZXMuCi0tLQogc3JjL2JhY2tlbmQvc3RvcmFnZS9zbWdyL21kLmMgfCA1ICsrKysr
CiAxIGZpbGUgY2hhbmdlZCwgNSBpbnNlcnRpb25zKCspCgpkaWZmIC0tZ2l0IGEvc3JjL2JhY2tl
bmQvc3RvcmFnZS9zbWdyL21kLmMgYi9zcmMvYmFja2VuZC9zdG9yYWdlL3NtZ3IvbWQuYwppbmRl
eCBkZWUyOTAzLi5kNzE1MzA3IDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9zdG9yYWdlL3NtZ3Iv
bWQuYworKysgYi9zcmMvYmFja2VuZC9zdG9yYWdlL3NtZ3IvbWQuYwpAQCAtNDE5LDYgKzQxOSwx
MSBAQCBtZHVubGlua2ZvcmsoUmVsRmlsZUxvY2F0b3JCYWNrZW5kIHJsb2NhdG9yLCBGb3JrTnVt
YmVyIGZvcmtudW0sIGJvb2wgaXNSZWRvKQogCQkvKiBQcmV2ZW50IG90aGVyIGJhY2tlbmRzJyBm
ZHMgZnJvbSBob2xkaW5nIG9uIHRvIHRoZSBkaXNrIHNwYWNlICovCiAJCXJldCA9IGRvX3RydW5j
YXRlKHBhdGguc3RyKTsKIAorCQkvKiBGb3JnZXQgYW55IHBlbmRpbmcgc3luYyByZXF1ZXN0cyBm
b3IgdGhlIGZpcnN0IHNlZ21lbnQgKi8KKwkJc2F2ZV9lcnJubyA9IGVycm5vOworCQlyZWdpc3Rl
cl9mb3JnZXRfcmVxdWVzdChybG9jYXRvciwgZm9ya251bSwgMCAvKiBmaXJzdCBzZWcgKi8gKTsK
KwkJZXJybm8gPSBzYXZlX2Vycm5vOworCiAJCS8qIFJlZ2lzdGVyIHJlcXVlc3QgdG8gdW5saW5r
IGZpcnN0IHNlZ21lbnQgbGF0ZXIgKi8KIAkJc2F2ZV9lcnJubyA9IGVycm5vOwogCQlyZWdpc3Rl
cl91bmxpbmtfc2VnbWVudChybG9jYXRvciwgZm9ya251bSwgMCAvKiBmaXJzdCBzZWcgKi8gKTsK
LS0gCjIuNDMuMAoK
--0000000000006b14ee064f61e5e0--