public inbox for [email protected]
help / color / mirror / Atom feedFrom: Anthonin Bonnefoy <[email protected]>
To: Fujii Masao <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: Shutdown indefinitely stuck due to unflushed FPI_FOR_HINT record
Date: Mon, 2 Mar 2026 11:38:06 +0100
Message-ID: <CAO6_Xqp+ADb6KZVWLMALu3xmwVUEO8S1EiCnp38mG6BrHrEnuA@mail.gmail.com> (raw)
In-Reply-To: <CAHGQGwHFKF+x4E+SqedMCnmLCitxjTUUtSyL_+mMeuq-GbEt6w@mail.gmail.com>
References: <CAO6_Xqo3co3BuUVEVzkaBVw9LidBgeeQ_2hfxeLMQcXwovB3GQ@mail.gmail.com>
<CAO6_XqrZEREa5d+dyjahX6bteBhoN=8Jid-3a4f6Q35sWrv9eg@mail.gmail.com>
<CAHGQGwHFKF+x4E+SqedMCnmLCitxjTUUtSyL_+mMeuq-GbEt6w@mail.gmail.com>
On Thu, Feb 26, 2026 at 5:20 PM Fujii Masao <[email protected]> wrote:
> Thanks for reporting this issue and for the patch!
>
> I was able to reproduce the problem on master.
Thanks for the confirmation!
> I'm a bit concerned that this approach could cause walwriter to attempt
> WAL writes more aggressively in some cases, adding overhead during
> normal processing. If the goal is only to prevent shutdown from getting stuck,
> instead, would it be sufficient for ShutdownXLOG() to call
> XLogSetAsyncXactLSN() just before WalSndWaitStopping()?
I did something similar with the first version using
XLogFlush(WriteRqstPtr), and it indeed fixes the shutdown issue.
Though I was under the impression that calling XLogSetAsyncXactLSN()
in RecordTransactionAbort was a better approach, as it is similar to
reporting the latest async abort. You may have a large backlog of
records, like selects pruning a lot of pages and timing out, which the
next commit will have to flush. Notifying the walwriter allows it to
flush those records it can.
Maybe for the context of a backport, patching ShutdownXLOG has the
benefit of minimising the amount of changes and risks?
I've updated the RecordTransactionAbort approach for now, with a small
optimization. XactLastRecEnd may be 0 if nothing was written, and we
can skip the unnecessary spinlocks in this case.
Regards,
Anthonin Bonnefoy
Attachments:
[application/octet-stream] v3-0001-Fix-stuck-shutdown-due-to-unflushed-records.patch (2.7K, 2-v3-0001-Fix-stuck-shutdown-due-to-unflushed-records.patch)
download | inline diff:
From 319254f19f9ddf1b7c93a58d9b0bab40b0a00d85 Mon Sep 17 00:00:00 2001
From: Anthonin Bonnefoy <[email protected]>
Date: Tue, 24 Feb 2026 09:24:48 +0100
Subject: Fix stuck shutdown due to unflushed records
Shutdown sequence may be stuck indefinitely under the following
circumstances:
- Data checksums is enabled
- A logical replication walsender is running
- A select in an explicit transaction tries to prune a full heap page,
wrote a FPI_FOR_HINT record which crosses the page boundary
- The select is rollbacked (or killed)
- 'pg_ctl stop' is sent
The FPI_FOR_HINT record is likely going to be a contrecord and starts a
new page. However, as the select is rollbacked, XLogSetAsyncXactLSN
isn't called to advance the LSN to include this record.
When the checkpointer starts ShutdownXLOG(), all walsenders will be
notified to stop. However, the logical replication walsender will be
stuck in the following infinite loop:
- Tries to read the last FPI_FOR_HINT record
- The page with the record header is read
- tot_len > len, the record needs to be reassembled
- Tries to read the next page containing the rest of the record. It fails since this page was never written.
- xlog reader state is reset with XLogReaderInvalReadState
- It goes back to the start of WalSndLoop's loop
There are some attempts done by the walsender to flush the WAL using
XLogBackgroundFlush. However, XLogBackgroundFlush only writes completed
blocks, or up to the latest known async lsn.
Since the select was rollbacked, XLogBackgroundFlush doesn't flush the
next partial page.
This patch fixes the issue by advancing the async LSN, even when the
transaction doesn't have an assigned xid. This allows
XLogBackgroundFlush to write the necessary partial page when called by
the walsender.
---
src/backend/access/transam/xact.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index eba4f063168..15729d9e2da 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -1787,7 +1787,19 @@ RecordTransactionAbort(bool isSubXact)
{
/* Reset XactLastRecEnd until the next transaction writes something */
if (!isSubXact)
+ {
+ /*
+ * Even if no xid was assigned, some records may have been written
+ * in the WAL. Report the latest async LSN, so that the WAL writer
+ * knows to flush those records. This is important when shutting
+ * down, walsender may use XLogBackgroundFlush to trigger pending
+ * WAL to be written out. If they're not tracked by async xact
+ * lsn, they won't be written by XLogBackgroundFlush.
+ */
+ if (XactLastRecEnd != 0)
+ XLogSetAsyncXactLSN(XactLastRecEnd);
XactLastRecEnd = 0;
+ }
return InvalidTransactionId;
}
--
2.52.0
view thread (17+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected]
Subject: Re: Shutdown indefinitely stuck due to unflushed FPI_FOR_HINT record
In-Reply-To: <CAO6_Xqp+ADb6KZVWLMALu3xmwVUEO8S1EiCnp38mG6BrHrEnuA@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox