From: Alexey Makhmutov <a.makhmutov@postgrespro.ru>
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Cc: shveta malik <shveta.malik@gmail.com>
Cc: Ajin Cherian <itsajin@gmail.com>
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: Standby server with cascade logical replication could not be properly stopped under load
Date: Thu, 22 May 2025 21:22:02 +0300
Message-ID: <f2575904-acfe-4037-9748-61fdb4a7abe4@postgrespro.ru> (raw)
In-Reply-To: <aC9DXHt/MwFCW6Is@ip-10-97-1-34.eu-west-3.compute.internal>
References: <52138028-7246-421c-9161-4fa108b88070@postgrespro.ru>
<CAJpy0uA9kw9WLWETPg+upwzBu145vMB7sPZwBuR_vDrbazeuag@mail.gmail.com>
<d9e1c765-7f83-4015-ade3-16506854d0b1@postgrespro.ru>
<aC9DXHt/MwFCW6Is@ip-10-97-1-34.eu-west-3.compute.internal>
Hi Bertrand, thank you for looking at this issue.
On 5/22/25 18:31, Bertrand Drouvot wrote:
> I agree with your analysis and I think that your proposed fix make sense. I
> wonder if the comment on top of GetStandbyFlushRecPtr() could be updated a bit
> though.
I've tried to update method comment as well in the updated patch:
0001-Use-only-replayed-position-as-target-flush-point-3-pg-master.patch.
The same patch could be applied on top of PG 17 as well.
PG 16 does not have the slot synchronization logic, so its
implementation of GetStandbyFlushRecPtr is slightly different and
comments need to be updated to reflect this discrepancy as well. So,
here is a variant of the same patch updated to be compatible with PG 16:
0002-Use-only-replayed-position-as-target-flush-point-3-pg-16.patch
Thanks,
Alexey
Attachments:
[text/x-patch] 0001-Use-only-replayed-position-as-target-flush-point-3-pg-master.patch (4.0K, ../f2575904-acfe-4037-9748-61fdb4a7abe4@postgrespro.ru/2-0001-Use-only-replayed-position-as-target-flush-point-3-pg-master.patch)
download | inline diff:
From 52fe2c5094c3a65ff8bafced4b0f522007068ebb Mon Sep 17 00:00:00 2001
From: Alexey Makhmutov <a.makhmutov@postgrespro.ru>
Date: Thu, 22 May 2025 04:02:50 +0300
Subject: [PATCH] Use only replayed position as target flush point for logical
replication
For physical walsender the GetStandbyFlushRecPtr function returns position
of the latest stored WAL record, as we can stream data to downstream instance
without waiting for it being locally applied. However, for logical replication
we can sent data to downstream client only up to the locally applied position.
This distinction is important when we decide whether walsender has sent all the
available data to client ('caught up'). During the shutdown process walsender
backends are allowed to work until they reach the 'caught up' state, while
recovery process is already deactivated, so the applied position won't be
moving. In this case walsender for logical replication should work only until
it caught up with the latest applied record, otherwise it will stuck in the
infinite loop and inhibit instance shutdown.
---
src/backend/replication/walsender.c | 49 ++++++++++++++++-------------
1 file changed, 28 insertions(+), 21 deletions(-)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.cindex 9fa8beb6103..49d5b3fea5a 100644--- a/src/backend/replication/walsender.c+++ b/src/backend/replication/walsender.c@@ -3519,42 +3519,49 @@ WalSndDone(WalSndSendDataCallback send_data)
}
/*
- * Returns the latest point in WAL that has been safely flushed to disk.- * This should only be called when in recovery.- *
* This is called either by cascading walsender to find WAL position to be sent
- * to a cascaded standby or by slot synchronization operation to validate remote- * slot's lsn before syncing it locally.+ * to a downstream server or by slot synchronization operation to validate remote+ * slot's lsn before syncing it locally. For streaming replication it returns+ * latest point in WAL that has been safely flushed to disk and for logical+ * replication it returns latest replayed position. This should only be called+ * when in recovery.
*
- * As a side-effect, *tli is updated to the TLI of the last- * replayed WAL record.+ * As a side-effect, *tli is updated to the TLI of the last replayed WAL record.
*/
XLogRecPtr
GetStandbyFlushRecPtr(TimeLineID *tli)
{
- XLogRecPtr replayPtr;
TimeLineID replayTLI;
- XLogRecPtr receivePtr;- TimeLineID receiveTLI;
XLogRecPtr result;
Assert(am_cascading_walsender || IsSyncingReplicationSlots());
- /*- * We can safely send what's already been replayed. Also, if walreceiver- * is streaming WAL from the same timeline, we can send anything that it- * has streamed, but hasn't been replayed yet.- */-- receivePtr = GetWalRcvFlushRecPtr(NULL, &receiveTLI);- replayPtr = GetXLogReplayRecPtr(&replayTLI);+ result = GetXLogReplayRecPtr(&replayTLI);
if (tli)
*tli = replayTLI;
- result = replayPtr;- if (receiveTLI == replayTLI && receivePtr > replayPtr)- result = receivePtr;+ /*+ * For physical replication we can send downstream anything that has+ * already been received if walreceiver is streaming WAL from the same+ * timeline. This is also true if this function is called from slot+ * synchronization routines (MyWalSnd is NULL in such case), since they+ * also only track the physically flushed position. However, for logical+ * replication the situation is different and we can only decode and send+ * downstream data that has already been applied to the current instance.+ * This distinction is especially important during shutdown, as walsender+ * is allowed to continue its work until data is sent to the client up to+ * the flush point, so it can get stuck if we return stored rather than+ * replayed data position.+ */+ if (!MyWalSnd || MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)+ {+ TimeLineID receiveTLI;+ XLogRecPtr receivePtr = GetWalRcvFlushRecPtr(NULL, &receiveTLI);++ if (receiveTLI == replayTLI && receivePtr > result)+ result = receivePtr;+ }
return result;
}
--
2.49.0
[text/x-patch] 0002-Use-only-replayed-position-as-target-flush-point-3-pg-16.patch (3.6K, ../f2575904-acfe-4037-9748-61fdb4a7abe4@postgrespro.ru/3-0002-Use-only-replayed-position-as-target-flush-point-3-pg-16.patch)
download | inline diff:
From ad8085d7646401f62d9b17af5274041bff77a720 Mon Sep 17 00:00:00 2001
From: Alexey Makhmutov <a.makhmutov@postgrespro.ru>
Date: Thu, 22 May 2025 04:02:50 +0300
Subject: [PATCH] Use only replayed position as target flush point for logical
replication
For physical walsender the GetStandbyFlushRecPtr function returns position
of the latest stored WAL record, as we can stream data to downstream instance
without waiting for it being locally applied. However, for logical replication
we can sent data to downstream client only up to the locally applied position.
This distinction is important when we decide whether walsender has sent all the
available data to client ('caught up'). During the shutdown process walsender
backends are allowed to work until they reach the 'caught up' state, while
recovery process is already deactivated, so the applied position won't be
moving. In this case walsender for logical replication should work only until
it caught up with the latest applied record, otherwise it will stuck in the
infinite loop and inhibit instance shutdown.
---
src/backend/replication/walsender.c | 43 ++++++++++++++++-------------
1 file changed, 24 insertions(+), 19 deletions(-)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.cindex 1e0480b1d55..f91ba63454c 100644--- a/src/backend/replication/walsender.c+++ b/src/backend/replication/walsender.c@@ -3189,37 +3189,42 @@ WalSndDone(WalSndSendDataCallback send_data)
}
/*
- * Returns the latest point in WAL that has been safely flushed to disk, and- * can be sent to the standby. This should only be called when in recovery,- * ie. we're streaming to a cascaded standby.+ * This is called by cascading walsender to find WAL position to be sent to a+ * downstream server. For streaming replication it returns* latest point in WAL+ * that has been safely flushed to disk and for logical replication it returns+ * latest replayed position. This should only be called when in recovery.
*
- * As a side-effect, *tli is updated to the TLI of the last- * replayed WAL record.+ * As a side-effect, *tli is updated to the TLI of the last replayed WAL record.
*/
static XLogRecPtr
GetStandbyFlushRecPtr(TimeLineID *tli)
{
- XLogRecPtr replayPtr;
TimeLineID replayTLI;
- XLogRecPtr receivePtr;- TimeLineID receiveTLI;
XLogRecPtr result;
- /*- * We can safely send what's already been replayed. Also, if walreceiver- * is streaming WAL from the same timeline, we can send anything that it- * has streamed, but hasn't been replayed yet.- */-- receivePtr = GetWalRcvFlushRecPtr(NULL, &receiveTLI);- replayPtr = GetXLogReplayRecPtr(&replayTLI);+ result = GetXLogReplayRecPtr(&replayTLI);
if (tli)
*tli = replayTLI;
- result = replayPtr;- if (receiveTLI == replayTLI && receivePtr > replayPtr)- result = receivePtr;+ /*+ * For physical replication we can send downstream anything that has+ * already been received if walreceiver is streaming WAL from the same+ * timeline. However, for logical replication the situation is different+ * and we can only decode and send downstream data that has already been+ * applied to the current instance. This distinction is especially+ * important during shutdown, as walsender is allowed to continue its work+ * until data is sent to the client up to the flush point, so it can get+ * stuck if we return stored rather than replayed data position.+ */+ if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)+ {+ TimeLineID receiveTLI;+ XLogRecPtr receivePtr = GetWalRcvFlushRecPtr(NULL, &receiveTLI);++ if (receiveTLI == replayTLI && receivePtr > result)+ result = receivePtr;+ }
return result;
}
--
2.49.0
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: a.makhmutov@postgrespro.ru, bertranddrouvot.pg@gmail.com, shveta.malik@gmail.com, itsajin@gmail.com, pgsql-bugs@lists.postgresql.org
Subject: Re: Standby server with cascade logical replication could not be properly stopped under load
In-Reply-To: <f2575904-acfe-4037-9748-61fdb4a7abe4@postgrespro.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox