MIME-Version: 1.0
References: <2631a3c3-5e60-4a1a-9e20-377024322602@gmail.com>
 <CAO6_Xqq73TPa3M6nQ7RqRhKkcphy1JX7aNGTYy-x_Sn+6a8Z_Q@mail.gmail.com>
 <CAHGQGwGvnpN=2bo+F7H90YLFcx9=SazwLkcx+0gEcrbQy5NVZg@mail.gmail.com>
 <CAHGQGwECpyJtMqkCEvyqgZDiwAeMj3RKobui7jONrDd35W0x3Q@mail.gmail.com>
 <vzguaguldbcyfbyuq76qj7hx5qdr5kmh67gqkncyb2yhsygrdt@dfhcpteqifux>
 <CAO6_Xqq=-KnxrFQvmyF++XH4ngtXVhcDB979rEn6SPtUhSmNYg@mail.gmail.com>
 <CAHGQGwGpfEsBJBLQDou7xccnb0tj1T_NQv2gZfuQfgzn_b=RsQ@mail.gmail.com>
 <CAHGQGwFsuPcC7Aov73FTqMC5LEOdZ6uoWno5BR1HX2LnZRNKBw@mail.gmail.com>
 <CAO6_XqqvqMqUgZ0swSkB2VNUFos6b6Vbt9BpOEGCHz4nCQEi+g@mail.gmail.com>
 <CAHGQGwFxoU0O6BDPp1_7D9aYh--sp9csC2WtiUkMR36oLXRCuA@mail.gmail.com>
 <abiRmwKizJsfa1_P@paquier.xyz>
In-Reply-To: <abiRmwKizJsfa1_P@paquier.xyz>
From: Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>
Date: Tue, 17 Mar 2026 17:51:40 +0100
Message-ID: 
 <CAO6_XqqaaejpuoQ5y04n6tX6ZdsWLtEyvyV3fEm2A9tSaWL5qw@mail.gmail.com>
Subject: Re: Shutdown indefinitely stuck due to unflushed FPI_FOR_HINT record
To: Michael Paquier <michael@paquier.xyz>
Cc: Fujii Masao <masao.fujii@gmail.com>, Andres Freund <andres@anarazel.de>,
	Alexander Lakhin <exclusion@gmail.com>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Content-Type: multipart/mixed; boundary="00000000000004d7fd064d3b27c0"
Archived-At: 
 <https://www.postgresql.org/message-id/CAO6_XqqaaejpuoQ5y04n6tX6ZdsWLtEyvyV3fEm2A9tSaWL5qw%40mail.gmail.com>
Precedence: bulk

--00000000000004d7fd064d3b27c0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, Mar 17, 2026 at 12:26=E2=80=AFAM Michael Paquier <michael@paquier.x=
yz> wrote:
> This stuff seems sensible enough that I think we should at least have
> a test, no?  It does not have to be absolutely perfect in terms of
> reproducibility, just good enough to be able to detect it across the
> buildfarm.  We already do various things with page boundaries in WAL
> during recovery, and a shutdown could be perhaps timed to increase the
> reproducibility rate of the issues discussed?

I initially thought that there was no easy way to trigger this issue
reliably in a test: the script I've been using won't work as soon as
there are changes in the record sizes. Then I remembered that
pg_logical_emit_message existed and could be used to write a WAL
record of a specific size, without allocating a xid and without
flushing the record.

With this, the test can be simplified to:
SELECT pg_switch_wal();
BEGIN;
SELECT pg_logical_emit_message(false, '', repeat('a', 16265), false);
ROLLBACK;

Any change in WAL short header, long header or xl_logical_message
struct will "break" the test since the record won't be at the exact
end of the page boundary. This also assumes that we have an 8 byte
alignment. 32 bits machine will have the WAL record ends at 3FF0, so
not exactly the end, but that should be fine to test different
conditions.

A word of caution about this test: While running it on my machine,
I've managed to trigger some weird WAL corruption. The new segment
after the switch had 1 or 2 excessive bytes at the start of the
segment just before the xlog page magic, shifting the whole file. The
first time it happened, I thought I'd messed something up and added
the bytes myself while looking at the WAL with imhex. The second time,
I've only run the script, and the new segment had a 1.1MB size shortly
after, so I'm pretty sure I didn't do anything that could have
introduced those excessive bytes.

I'm still trying to understand the trigger conditions (some race
condition between the switch and the walwriter?), but if this test is
merged, it may trigger this WAL corruption issue on the buildfarm.

Regards,
Anthonin Bonnefoy

--00000000000004d7fd064d3b27c0
Content-Type: application/octet-stream;
	name="v1-0001-Add-test-shutting-down-walsender-with-unflushed-r.patch"
Content-Disposition: attachment;
	filename="v1-0001-Add-test-shutting-down-walsender-with-unflushed-r.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_mmuujg9l0>
X-Attachment-Id: f_mmuujg9l0

RnJvbSA5N2I5YmE0Mzc0YjU0MjNmNDBiMzg2OGQ4M2I1MTVkOTQ5NjlkZmE2IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBBbnRob25pbiBCb25uZWZveSA8YW50aG9uaW4uYm9ubmVmb3lA
ZGF0YWRvZ2hxLmNvbT4KRGF0ZTogVHVlLCAxNyBNYXIgMjAyNiAxNTowNzo1NSArMDEwMApTdWJq
ZWN0OiBBZGQgdGVzdCBzaHV0dGluZyBkb3duIHdhbHNlbmRlciB3aXRoIHVuZmx1c2hlZCByZWNv
cmQKCjZlZWRiMmE1ZmQ4IGZpeGVkIGFuIGlzc3VlIHdoZXJlIHRoZSB3YWxzZW5kZXIgd2FzIHN0
dWNrIGluIGEgYnVzeSBsb29wLAp0cnlpbmcgdG8gcmVhZCBhbiB1bmZsdXNlZCByZWNvcmQuCgpk
OTI3YjRiZDk3IGZpeGVkIGFub3RoZXIgaXNzdWUgaW50cm9kdWNlZCBieSB0aGUgcHJldmlvdXMg
Y29tbWl0LCB3aGVyZQpYTG9nRmx1c2ggd291bGQgYmUgY2FsbGVkIHBhc3QgdGhlIGVuZCBvZiB0
aGUgZ2VuZXJhdGVkIFdBTCwgZ2VuZXJhdGluZwphbiBlcnJvciBsb2cuCgpUaGlzIGNvbW1pdCBh
ZGRzIGEgdGVzdCB0byBjb3ZlciB0aG9zZSB0d28gZml4ZXMuIEJ5IHdyaXRpbmcgYSBsb2dpY2Fs
Cm1lc3NhZ2Ugb2YgYSBzcGVjaWZpYyBzaXplLCB3ZSBjYW4gcmVhY2ggYSBzdGF0ZSB3aGVyZSB0
aGUgbGFzdCByZWNvcmQKaXMgY3Jvc3NpbmcgdGhlIHBhZ2UgYm91bmRhcnksIGFuZCBlbmRzIGF0
IHRoZSBlbmQgb2YgdGhlIG5leHQgcGFnZSwKcmVjcmVhdGluZyB0aGUgY29uZGl0aW9ucyBmb3Ig
dGhlIGFmb3JlbWVudGlvbmVkIGlzc3Vlcy4KLS0tCiBzcmMvdGVzdC9yZWNvdmVyeS90LzAwNl9s
b2dpY2FsX2RlY29kaW5nLnBsIHwgNTAgKysrKysrKysrKysrKysrKysrKystCiAxIGZpbGUgY2hh
bmdlZCwgNDkgaW5zZXJ0aW9ucygrKSwgMSBkZWxldGlvbigtKQoKZGlmZiAtLWdpdCBhL3NyYy90
ZXN0L3JlY292ZXJ5L3QvMDA2X2xvZ2ljYWxfZGVjb2RpbmcucGwgYi9zcmMvdGVzdC9yZWNvdmVy
eS90LzAwNl9sb2dpY2FsX2RlY29kaW5nLnBsCmluZGV4IDk3ZDExZjk4YjU5Li4xZWZiNmY3YjUy
NyAxMDA2NDQKLS0tIGEvc3JjL3Rlc3QvcmVjb3ZlcnkvdC8wMDZfbG9naWNhbF9kZWNvZGluZy5w
bAorKysgYi9zcmMvdGVzdC9yZWNvdmVyeS90LzAwNl9sb2dpY2FsX2RlY29kaW5nLnBsCkBAIC0y
NzUsNyArMjc1LDU1IEBAIGlzKCAkbm9kZV9wcmltYXJ5LT5zYWZlX3BzcWwoCiAJcXEoQ2hlY2sg
dGhhdCByZXNldCB0aW1lc3RhbXAgaXMgbGF0ZXIgYWZ0ZXIgcmVzZXR0aW5nIHN0YXRzIGZvciBz
bG90ICckc3RhdHNfdGVzdF9zbG90MScgYWdhaW4uKQogKTsKIAotIyBkb25lIHdpdGggdGhlIG5v
ZGUKK1NLSVA6Cit7CisKKwkjIHNvbWUgV2luZG93cyBQZXJscyBhdCBsZWFzdCBkb24ndCBsaWtl
IElQQzo6UnVuJ3Mgc3RhcnQva2lsbF9raWxsIHJlZ2ltZS4KKwlza2lwICJUZXN0IGZhaWxzIG9u
IFdpbmRvd3MgcGVybCIsIDIgaWYgJENvbmZpZ3tvc25hbWV9IGVxICdNU1dpbjMyJzsKKworCSMg
VGVzdCBzdG9wcGluZyB0aGUgcHJpbWFyeSB3aXRoIGFuIGFjdGl2ZSB3YWxzZW5kZXIgYW5kIGFu
IHVuZmx1c2hlZCByZWNvcmQKKwkjIHdoaWNoIGNyb3NzZXMgdGhlIHBhZ2UgYm91bmRhcnkgYW5k
IGVuZHMgYXQgdGhlIGVuZCBvZiB0aGUgbmV4dCBwYWdlLgorCSMKKwkjIEZpcnN0LCBzdGFydCBw
Z19yZWN2bG9naWNhbCB0byBoYXZlIGFuIGFjdGl2ZSB3YWxzZW5kZXIKKwlteSAkcGdfcmVjdmxv
Z2ljYWwgPSBJUEM6OlJ1bjo6c3RhcnQoCisJCVsKKwkJCSdwZ19yZWN2bG9naWNhbCcsCisJCQkn
LS1kYm5hbWUnID0+ICRub2RlX3ByaW1hcnktPmNvbm5zdHIoJ3Bvc3RncmVzJyksCisJCQknLS1z
bG90JyA9PiAndGVzdF9zbG90JywKKwkJCSctLWZpbGUnID0+ICctJywKKwkJCSctLXN0YXJ0Jwor
CQldKTsKKworCSMgVGhlbiwgd2Ugd3JpdGUgYSBsb2dpY2FsIG1lc3NhZ2UgV0FMIHJlY29yZCB3
aGljaCBmaW5pc2hlcyBhdCB0aGUgZW5kIG9mIGEKKwkjIFdBTCBwYWdlLCB1c2luZyBhIHJvbGxi
YWNrIHNvIHRoZSBXQUwgcmVjb3JkIGlzbid0IGZsdXNoZWQuCisJIworCSMgVGhlIHNpemUgb2Yg
YSBXQUwgbG9naWNhbCBtZXNzYWdlIHJlY29yZCBpcyA1NSBieXRlcyArIG1lc3NhZ2UgbGVuZ3Ro
CisJIyBTdGFydGluZyBmcm9tIGEgZnJlc2ggV0FMIHNlZ21lbnQsIHdlIGhhdmU6CisJIyAgIC0g
ODE1MiBieXRlcyBhdmFpbGFibGUgaW4gdGhlIGZpcnN0IHBhZ2UgKGxvbmcgaGVhZGVyKQorCSMg
ICAtIDgxNjggYnl0ZXMgYXZhaWxhYmxlIGluIHRoZSBzZWNvbmQgcGFnZSAoc2hvcnQgaGVhZGVy
KQorCSMgV2UgbmVlZCB0byB3cml0ZSAxNjMyMCBieXRlcyBvZiBsb2dpY2FsIG1lc3NhZ2UgV0FM
IHJlY29yZCwgd2hpY2ggY2FuIGJlIGRvbmUKKwkjIHVzaW5nIGEgMTYyNjUgYnl0ZXMgbG9uZyBt
ZXNzYWdlLgorCSRub2RlX3ByaW1hcnktPnNhZmVfcHNxbCgncG9zdGdyZXMnLAorCQlxcVsKKwkJ
U0VMRUNUIHBnX3N3aXRjaF93YWwoKTsKKwkJQkVHSU47CisJCVNFTEVDVCBwZ19sb2dpY2FsX2Vt
aXRfbWVzc2FnZShmYWxzZSwgJycsIHJlcGVhdCgnYScsIDE2MjY1KSwgZmFsc2UpOworCQlST0xM
QkFDSzsKKwkJXQorCSk7CisKKwkjIHRyeSB0byByZXN0YXJ0CisJJG5vZGVfcHJpbWFyeS0+cmVz
dGFydDsKKwkkcGdfcmVjdmxvZ2ljYWwtPmtpbGxfa2lsbDsKKworCW15ICRsb2dmaWxlID0gc2x1
cnBfZmlsZSgkbm9kZV9wcmltYXJ5LT5sb2dmaWxlKCkpOworCXVubGlrZSgKKwkJJGxvZ2ZpbGUs
CisJCXFyL3JlcXVlc3QgdG8gZmx1c2ggcGFzdCBlbmQgb2YgZ2VuZXJhdGVkIFdBTC8sCisJCSJU
aGVyZSdzIG5vIGZsdXNoIHJlcXVlc3QgcGFzdCBlbmQgb2YgZ2VuZXJhdGVkIFdBTCIpOworfQor
CisjIHN0b3AgdGhlIG5vZGUKICRub2RlX3ByaW1hcnktPnN0b3A7CiAKIGRvbmVfdGVzdGluZygp
OwotLSAKMi41My4wCgo=
--00000000000004d7fd064d3b27c0--