MIME-Version: 1.0
References: <CAO6_Xqo3co3BuUVEVzkaBVw9LidBgeeQ_2hfxeLMQcXwovB3GQ@mail.gmail.com>
In-Reply-To: <CAO6_Xqo3co3BuUVEVzkaBVw9LidBgeeQ_2hfxeLMQcXwovB3GQ@mail.gmail.com>
From: Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>
Date: Tue, 24 Feb 2026 10:46:51 +0100
Message-ID: <CAO6_XqrZEREa5d+dyjahX6bteBhoN=8Jid-3a4f6Q35sWrv9eg@mail.gmail.com>
Subject: Re: Shutdown indefinitely stuck due to unflushed FPI_FOR_HINT record
To: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Content-Type: multipart/mixed; boundary="000000000000075e64064b8ec5bc"
Archived-At: <https://www.postgresql.org/message-id/CAO6_XqrZEREa5d%2BdyjahX6bteBhoN%3D8Jid-3a4f6Q35sWrv9eg%40mail.gmail.com>
Precedence: bulk

--000000000000075e64064b8ec5bc
Content-Type: text/plain; charset="UTF-8"

Some additional informations:

XLogSendLogical is stuck in the following infinite loop:
- It attempt to read the next record with XLogReadAhead + XLogDecodeNextRecord
- The page with the record header is read
- It has the record header, it goes back to XLogDecodeNextRecord
- tot_len > len, the record needs to be reassembled
- The next page containing the rest of the record is read with
ReadPageInternal. It fails since this page was never written.
- It jumps to the err label, XLogReaderInvalReadState(state) is called
and reset the reader state
- It goes back to the start of WalSndLoop's loop

There are some attempts done by the walsender to flush the WAL using
XLogBackgroundFlush:
  /*
   * If we're shutting down, trigger pending WAL to be written out,
   * otherwise we'd possibly end up waiting for WAL that never gets
   * written, because walwriter has shut down already.
   */
  if (got_STOPPING)
    XLogBackgroundFlush();

However, XLogBackgroundFlush only writes completed blocks or the
latest async xact known. With the issue triggered, I have the
following state:

XLogCtl->LogwrtRqst: (Write = 39128056, Flush = 39124992)
LogwrtResult: (Write = 39124992, Flush = 39124992)
XLogCtl->asyncXactLSN: 39119776

There are 3064 bytes (39128056 - 39124992) that contain the next page
with the rest of the cont record that still needs to be written.
However, XLogBackgroundFlush backs off to the previous page boundary:
  /* back off to last completed page boundary */
  WriteRqst.Write -= WriteRqst.Write % XLOG_BLCKSZ;
Meaning WriteRqst.Write will be 39124992, which is already written and
flushed and asyncXactLSN is behind both write and flush.

So, it looks like the root issue is more that the async LSN isn't
updated when a transaction without xid is rollbacked.
When going through CommitTransaction, such a transaction would still
go through XLogSetAsyncXactLSN.

I've updated the patch with this new approach: XLogSetAsyncXactLSN is
now called in RecordTransactionAbort even when a xid wasn't assigned.
With this, the logical walsender is able to force the flush of the
last partial page using XLogBackgroundFlush.

--000000000000075e64064b8ec5bc
Content-Type: application/octet-stream; 
	name="v2-0001-Fix-stuck-shutdown-due-to-unflushed-records.patch"
Content-Disposition: attachment; 
	filename="v2-0001-Fix-stuck-shutdown-due-to-unflushed-records.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_mm0efy7d0>
X-Attachment-Id: f_mm0efy7d0

RnJvbSAyOGY1NTJiOWNjYjA3NjYyN2QyNTc3YjdkYWViMjNhOTNhMWU1MGVmIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBBbnRob25pbiBCb25uZWZveSA8YW50aG9uaW4uYm9ubmVmb3lA
ZGF0YWRvZ2hxLmNvbT4KRGF0ZTogVHVlLCAyNCBGZWIgMjAyNiAwOToyNDo0OCArMDEwMApTdWJq
ZWN0OiBGaXggc3R1Y2sgc2h1dGRvd24gZHVlIHRvIHVuZmx1c2hlZCByZWNvcmRzCgpTaHV0ZG93
biBzZXF1ZW5jZSBtYXkgYmUgc3R1Y2sgaW5kZWZpbml0ZWx5IHVuZGVyIHRoZSBmb2xsb3dpbmcK
Y2lyY3Vtc3RhbmNlczoKLSBEYXRhIGNoZWNrc3VtcyBpcyBlbmFibGVkCi0gQSBsb2dpY2FsIHJl
cGxpY2F0aW9uIHdhbHNlbmRlciBpcyBydW5uaW5nCi0gQSBzZWxlY3QgaW4gYW4gZXhwbGljaXQg
dHJhbnNhY3Rpb24gdHJpZXMgdG8gcHJ1bmUgYSBmdWxsIGhlYXAgcGFnZSwKICB3cm90ZSBhIEZQ
SV9GT1JfSElOVCByZWNvcmQgd2hpY2ggY3Jvc3NlcyB0aGUgcGFnZSBib3VuZGFyeQotIFRoZSBz
ZWxlY3QgaXMgcm9sbGJhY2tlZCAob3Iga2lsbGVkKQotICdwZ19jdGwgc3RvcCcgaXMgc2VudAoK
VGhlIEZQSV9GT1JfSElOVCByZWNvcmQgaXMgbGlrZWx5IGdvaW5nIHRvIGJlIGEgY29udHJlY29y
ZCBhbmQgc3RhcnRzIGEKbmV3IHBhZ2UuIEhvd2V2ZXIsIGFzIHRoZSBzZWxlY3QgaXMgcm9sbGJh
Y2tlZCwgWExvZ1NldEFzeW5jWGFjdExTTgppc24ndCBjYWxsZWQgdG8gYWR2YW5jZSB0aGUgTFNO
IHRvIGluY2x1ZGUgdGhpcyByZWNvcmQuCgpXaGVuIHRoZSBjaGVja3BvaW50ZXIgc3RhcnRzIFNo
dXRkb3duWExPRygpLCBhbGwgd2Fsc2VuZGVycyB3aWxsIGJlCm5vdGlmaWVkIHRvIHN0b3AuIEhv
d2V2ZXIsIHRoZSBsb2dpY2FsIHJlcGxpY2F0aW9uIHdhbHNlbmRlciB3aWxsIGJlCnN0dWNrIGlu
IHRoZSBmb2xsb3dpbmcgaW5maW5pdGUgbG9vcDoKLSBUcmllcyB0byByZWFkIHRoZSBsYXN0IEZQ
SV9GT1JfSElOVCByZWNvcmQKLSBUaGUgcGFnZSB3aXRoIHRoZSByZWNvcmQgaGVhZGVyIGlzIHJl
YWQKLSB0b3RfbGVuID4gbGVuLCB0aGUgcmVjb3JkIG5lZWRzIHRvIGJlIHJlYXNzZW1ibGVkCi0g
VHJpZXMgdG8gcmVhZCB0aGUgbmV4dCBwYWdlIGNvbnRhaW5pbmcgdGhlIHJlc3Qgb2YgdGhlIHJl
Y29yZC4gSXQgZmFpbHMgc2luY2UgdGhpcyBwYWdlIHdhcyBuZXZlciB3cml0dGVuLgotIHhsb2cg
cmVhZGVyIHN0YXRlIGlzIHJlc2V0IHdpdGggWExvZ1JlYWRlckludmFsUmVhZFN0YXRlCi0gSXQg
Z29lcyBiYWNrIHRvIHRoZSBzdGFydCBvZiBXYWxTbmRMb29wJ3MgbG9vcAoKVGhlcmUgYXJlIHNv
bWUgYXR0ZW1wdHMgZG9uZSBieSB0aGUgd2Fsc2VuZGVyIHRvIGZsdXNoIHRoZSBXQUwgdXNpbmcK
WExvZ0JhY2tncm91bmRGbHVzaC4gSG93ZXZlciwgWExvZ0JhY2tncm91bmRGbHVzaCBvbmx5IHdy
aXRlcyBjb21wbGV0ZWQKYmxvY2tzLCBvciB1cCB0byB0aGUgbGF0ZXN0IGtub3duIGFzeW5jIGxz
bi4KClNpbmNlIHRoZSBzZWxlY3Qgd2FzIHJvbGxiYWNrZWQsIFhMb2dCYWNrZ3JvdW5kRmx1c2gg
ZG9lc24ndCBmbHVzaCB0aGUKbmV4dCBwYXJ0aWFsIHBhZ2UuCgpUaGlzIHBhdGNoIGZpeGVzIHRo
ZSBpc3N1ZSBieSBhZHZhbmNpbmcgdGhlIGFzeW5jIExTTiwgZXZlbiB3aGVuIHRoZQp0cmFuc2Fj
dGlvbiBkb2Vzbid0IGhhdmUgYW4gYXNzaWduZWQgeGlkLiBUaGlzIGFsbG93cwpYTG9nQmFja2dy
b3VuZEZsdXNoIHRvIHdyaXRlIHRoZSBuZWNlc3NhcnkgcGFydGlhbCBwYWdlIHdoZW4gY2FsbGVk
IGJ5CnRoZSB3YWxzZW5kZXIuCi0tLQogc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGFjdC5j
IHwgMTEgKysrKysrKysrKysKIDEgZmlsZSBjaGFuZ2VkLCAxMSBpbnNlcnRpb25zKCspCgpkaWZm
IC0tZ2l0IGEvc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGFjdC5jIGIvc3JjL2JhY2tlbmQv
YWNjZXNzL3RyYW5zYW0veGFjdC5jCmluZGV4IGViYTRmMDYzMTY4Li4wYTQ5ZDViNjAzYyAxMDA2
NDQKLS0tIGEvc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGFjdC5jCisrKyBiL3NyYy9iYWNr
ZW5kL2FjY2Vzcy90cmFuc2FtL3hhY3QuYwpAQCAtMTc4Nyw3ICsxNzg3LDE4IEBAIFJlY29yZFRy
YW5zYWN0aW9uQWJvcnQoYm9vbCBpc1N1YlhhY3QpCiAJewogCQkvKiBSZXNldCBYYWN0TGFzdFJl
Y0VuZCB1bnRpbCB0aGUgbmV4dCB0cmFuc2FjdGlvbiB3cml0ZXMgc29tZXRoaW5nICovCiAJCWlm
ICghaXNTdWJYYWN0KQorCQl7CisJCQkvKgorCQkJICogRXZlbiBpZiBubyB4aWQgd2FzIGFzc2ln
bmVkLCBzb21lIHJlY29yZHMgbWF5IGhhdmUgYmVlbiB3cml0dGVuCisJCQkgKiBpbiB0aGUgV0FM
LiBSZXBvcnQgdGhlIGxhdGVzdCBhc3luYyBMU04sIHNvIHRoYXQgdGhlIFdBTCB3cml0ZXIKKwkJ
CSAqIGtub3dzIHRvIGZsdXNoIHRob3NlIHJlY29yZHMuIFRoaXMgaXMgaW1wb3J0YW50IHdoZW4g
c2h1dHRpbmcKKwkJCSAqIGRvd24sIHdhbHNlbmRlciBtYXkgdXNlIFhMb2dCYWNrZ3JvdW5kRmx1
c2ggdG8gdHJpZ2dlciBwZW5kaW5nCisJCQkgKiBXQUwgdG8gYmUgd3JpdHRlbiBvdXQuIElmIHRo
ZXkncmUgbm90IHRyYWNrZWQgYnkgYXN5bmMgeGFjdAorCQkJICogbHNuLCB0aGV5IHdvbid0IGJl
IHdyaXR0ZW4gYnkgWExvZ0JhY2tncm91bmRGbHVzaC4KKwkJCSAqLworCQkJWExvZ1NldEFzeW5j
WGFjdExTTihYYWN0TGFzdFJlY0VuZCk7CiAJCQlYYWN0TGFzdFJlY0VuZCA9IDA7CisJCX0KIAkJ
cmV0dXJuIEludmFsaWRUcmFuc2FjdGlvbklkOwogCX0KIAotLSAKMi41Mi4wCgo=
--000000000000075e64064b8ec5bc--