Subject: Re: 001_rep_changes.pl fails due to publisher stuck on shutdown

From: Amit Kapila <[email-masked]>
Date: Tue, 11 Jun 2024 14:27:28 +0530

On Tue, Jun 11, 2024 at 12:34 PM Kyotaro Horiguchi
<[email-masked]> wrote:
>
> At Tue, 11 Jun 2024 11:32:12 +0530, Amit Kapila <[email-masked]> wrote in
> > Sorry, it is not clear to me why we failed to flush the last
> > continuation record in logical walsender? I see that we try to flush
> > the WAL after receiving got_STOPPING in WalSndWaitForWal(), why is
> > that not sufficient?
>
> It seems that, it uses XLogBackgroundFlush(), which does not guarantee
> flushing WAL until the end.
>

What would it take to ensure the same? I am trying to explore this
path because currently logical WALSender sends any outstanding logs up
to the shutdown checkpoint record (i.e., the latest record) and waits
for them to be replicated to the standby before exit. Please take a
look at the comments where we call WalSndDone(). The fix you are
proposing will break that guarantee.

-- 
With Regards,
Amit Kapila.

---

From: Kyotaro Horiguchi <[email-masked]>
Date: Wed, 12 Jun 2024 10:13:27 +0900 (JST)

At Tue, 11 Jun 2024 14:27:28 +0530, Amit Kapila <[email-masked]> wrote in 
> proposing will break that guarantee.

Shutdown checkpoint is performed after the walsender completed
termination since 086221cf6b, aiming to prevent walsenders from
generating competing WAL (by, for example, CREATE_REPLICATION_SLOT)
records with the shutdown checkpoint. Thus, it seems that the
walsender cannot see the shutdown record, and a certain amount of
bytes before it, as the walsender appears to have relied on the
checkpoint flushing its record, rather than on XLogBackgroundFlush().

If we approve of the walsender being terminated before the shutdown
checkpoint, we need to "fix" the comment, then provide a function to
ensure the synchronization of WAL records.

I'll consider this direction for a while.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

---

From: Amit Kapila <[email-masked]>
Date: Thu, 13 Jun 2024 09:29:03 +0530

On Wed, Jun 12, 2024 at 6:43 AM Kyotaro Horiguchi
<[email-masked]> wrote:
> termination since 086221cf6b,
>

Yeah, but the commit you quoted later reverted by commit 703f148e98
and committed again as c6c3334364.

> aiming to prevent walsenders from
> walsender cannot see the shutdown record,
>

This is true of logical walsender. The physical walsender do send
shutdown checkpoint record before getting terminated.

> and a certain amount of

Which comment do you want to fix?

> I'll consider this direction for a while.

Okay, thanks.

-- 
With Regards,
Amit Kapila.

---

From: Kyotaro Horiguchi <[email-masked]>
Date: Tue, 18 Jun 2024 17:07:14 +0900 (JST)

At Thu, 13 Jun 2024 09:29:03 +0530, Amit Kapila <[email-masked]> wrote in 
> and committed again as c6c3334364.

Yeah, right..

> shutdown checkpoint record before getting terminated.

Yes, I know. They differ in their blocking mechanisms.

> Which comment do you want to fix?

Yeah. The part you seem to think I was trying to fix is actually
fine. Instead, I have revised the comment on the modified section to
make its intention clearer.

> > I'll consider this direction for a while.

The attached patch is it. It's only for the master.

I decided not to create a new function because the simple code has
only one caller. I haven't seen the test script fail with this fix.

regards.

>From 663bdeaf8d4d2f5a192dd3ecb6d2817d9b1311f1 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <[email-masked]>
Date: Tue, 18 Jun 2024 16:36:43 +0900
Subject: [PATCH] Ensure WAL is written out when terminating a logical
 walsender

In commit c6c3334364, XLogBackgroundFlush() was assumed to flush all
written WAL to the end, but this function may not flush the last
incomplete page in a single call. In such cases, if the last written
page ends with a continuation record, the latter part will not be
flushed, causing shutdown to wait indefinitely for that part. This
commit ensures that the written records are fully flushed to the end,
guaranteeing expected behavior.
---
 src/backend/replication/walsender.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c623b07cf0..5aa0f58238 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1858,12 +1858,13 @@ WalSndWaitForWal(XLogRecPtr loc)
 		ProcessRepliesIfAny();
 
 		/*
-		 * If we're shutting down, trigger pending WAL to be written out,
-		 * otherwise we'd possibly end up waiting for WAL that never gets
-		 * written, because walwriter has shut down already.
+		 * If we're shutting down, all WAL-writing processes are gone, leaving
+		 * only checkpointer to perform the shutdown checkpoint. Ensure that
+		 * any pending WAL is written out here; otherwise, we'd possibly end up
+		 * waiting for WAL that never gets written.
 		 */
-		if (got_STOPPING)
-			XLogBackgroundFlush();
+		if (got_STOPPING && !RecoveryInProgress())
+			XLogFlush(GetInsertRecPtr());
 
 		/*
 		 * To avoid the scenario where standbys need to catch up to a newer
-- 
2.43.0

---

From: "Hayato Kuroda (Fujitsu)" <[email-masked]>
Date: Wed, 19 Jun 2024 05:14:50 +0000

Dear Horiguchi-san,

Thanks for sharing the patch! I agree this approach (ensure WAL records are flushed)
Is more proper than others.

I have an unclear point. According to the comment atop GetInsertRecPtr(), it just
returns the approximated value - the position of the last full WAL page [1].
If there is a continuation WAL record which across a page, will it return the
halfway point of the WAL record (end of the first WAL page)? If so, the proposed
fix seems not sufficient. We have to point out the exact the end of the record.

[1]:
/*
 * GetInsertRecPtr -- Returns the current insert position.
 *
 * NOTE: The value *actually* returned is the position of the last full
 * xlog page. It lags behind the real insert position by at most 1 page.
 * For that, we don't need to scan through WAL insertion locks, and an
 * approximation is enough for the current usage of this function.
 */
XLogRecPtr
GetInsertRecPtr(void)

Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/

---

From: Peter Smith <[email-masked]>
Date: Wed, 19 Jun 2024 17:58:50 +1000

FYI - I applied this latest patch and re-ran the original failing test
script 10 times (e.g. 10 x 100 test iterations; it took 4+ hours).

There were zero failures observed in my environment.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: Michael Paquier <[email-masked]>
Date: Fri, 21 Jun 2024 10:48:38 +0900

> fix seems not sufficient. We have to point out the exact the end of the record.

Yeah, that a thing of the patch I am confused with.  How are we sure
that this is the correct LSN to rely on?  If that it the case, the
patch does not offer an explanation about why it is better.

WalSndWaitForWal() is called only in the context of page callback for a
logical WAL sender.  Shouldn't we make the flush conditional on what's
stored in XLogReaderState.missingContrecPtr?  Aka, if we know that
we're in the middle of the decoding of a continuation record, we
should wait until we've dealt with it, no?

In short, I would imagine that WalSndWaitForWal() should know more
about XLogReaderState is doing.  But perhaps I'm missing something.
--
Michael

---

From: Amit Kapila <[email-masked]>
Date: Fri, 21 Jun 2024 11:48:22 +0530

> fix seems not sufficient. We have to point out the exact the end of the record.

You have a point but if this theory is correct why we are not able to
reproduce the issue after this patch? Also, how to get the WAL
location up to which we need to flush? Is XLogCtlData->logInsertResult
the one we are looking for?

-- 
With Regards,
Amit Kapila.