From: Alexey Makhmutov <a.makhmutov@postgrespro.ru>
To: pgsql-bugs@lists.postgresql.org
Subject: Standby server with cascade logical replication could not be properly stopped under load
Date: Thu, 22 May 2025 05:19:56 +0300
Message-ID: <52138028-7246-421c-9161-4fa108b88070@postgrespro.ru> (raw)
Assuming following configuration with three connected servers A->B->C: A
(primary), B (physical standby) and C (logical replica connected to B).
If server A is under load and B is applying incoming WAL records while
also streaming data via logical replication to C, then attempt to stop
server B in 'fast' mode may by unsuccessful. In this case server will
remain in PM_SHUTDOWN state indefinitely with all 'walsender' processes
running in an infinite busy loop (consuming a CPU core each). To get
server out of this state it's required either to either stop B using
'immediate' mode or stop server C (which will cause 'walsender'
processes on server B to exit). This issue is reproducible on latest
'master', as well as on current PG 16/17 branches.
Attached is a test scenario to reproduce the issue: 'test_scenario.zip'.
This archive contains shell scripts to create
the required environment (all three serves) and then to execute required
steps to get server into incorrect state. First, edit the 'test_env.sh'
file and specify path to PG binaries in PG_PATH variable and optionally
set of ports used by test instances in 'pg_port' array. Then execute the
'test_prepare.sh' script, which will create, configure and start all
three PG instances. Servers then could be started and stopped using
corresponding start and stop scripts. To reproduce the issue, ensure
that all three servers are running and execute the 'test_execute.sh'
script. This script starts 'pgbench' instance in background for 30
seconds to create load on server A, waits for 20 seconds and then try to
stop the B instance using default 'fast' mode. Expected behavior is
normal shutdown for B, while observed behavior is different: shutdown
attempt fails and each remaining 'walsender' process consumes entire CPU
core. To get out of this state one could use 'stop-C.sh' script to stop
the server C, as it will complete shutdown process of B instance as well.
My understanding is that this issue seems to be caused by the logic in
'GetStandbyFlushRecPtr' function, which returns current flush point for
received WAL data. This position is used in 'XLogSendLogical' to
calculate whether current walsender is in 'caught up' state (i.e.
whether we send all the available data to downstream instance). During
shutdown walsenders are allowed to continue their work until they are in
'caught up' state, while 'postmaster' is waiting for their completion.
Currently 'GetStandbyFlushRecPtr' returns position of last stored
record, rather than last applied record. This is correct for physical
replication as we can send data to downstream instance without applying
it to local system. However, for logical replication this seems to be
incorrect, as we could not decode data until it's applied on current
instance. So, if current stored WAL position differs from applied
position while server is being stopped, then
'WalSndLoop'/'XLogSendLogical'/'XLogReadRecord' methods will spin in a
busy loop, waiting for applied position to advance. The recovery process
is already stopped at this moment, so this will be an infinite loop.
Probably either 'GetStandbyFlushRecPtr' or
'WalSndLoop'/'XLogSendLogical' logic need to be adjusted to take into
consideration such case with logical replication.
Attached is also a patch, which aims to fix this issue:
0001-Use-only-replayed-position-as-target-flush-point-for.patch. It
tries to to modify behavior of 'GetStandbyFlushRecPtr' function to
ensure that it returns only applied position for logical replication.
This function could be also invoked from slot synchronization routines
and in this case it retains current behavior by returning last stored
position.
Thanks,
Alexey
Attachments:
[application/zip] test_scenario.zip (5.1K, ../52138028-7246-421c-9161-4fa108b88070@postgrespro.ru/2-test_scenario.zip)
download
[text/x-patch] 0001-Use-only-replayed-position-as-target-flush-point-for.patch (2.9K, ../52138028-7246-421c-9161-4fa108b88070@postgrespro.ru/3-0001-Use-only-replayed-position-as-target-flush-point-for.patch)
download | inline diff:
From 698b13005e6fb85741b758d9f8b14065f6394092 Mon Sep 17 00:00:00 2001
From: Alexey Makhmutov <a.makhmutov@postgrespro.ru>
Date: Thu, 22 May 2025 04:02:50 +0300
Subject: [PATCH] Use only replayed position as target flush point for logical
replication
For physical walsender the GetStandbyFlushRecPtr function returns position
of the latest stored WAL record, as we can stream data to downstream instance
without waiting for it being locally applied. However, for logical replication
we can sent data to downstream client only up to the locally applied position.
This distinction is important when we decide whether walsender has sent all the
available data to client ('caught up'). During the shutdown process walsender
backends are allowed to work until they reach the 'caught up' state, while
recovery process is already deactivated, so the applied position won't be
moving. In this case walsender for logical replication should work only until
it caught up with the latest applied record, otherwise it will stuck in the
infinite loop and inhibit instance shutdown.
---
src/backend/replication/walsender.c | 31 ++++++++++++++++-------------
1 file changed, 17 insertions(+), 14 deletions(-)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.cindex 9fa8beb6103..ede985da216 100644--- a/src/backend/replication/walsender.c+++ b/src/backend/replication/walsender.c@@ -3532,29 +3532,32 @@ WalSndDone(WalSndSendDataCallback send_data)
XLogRecPtr
GetStandbyFlushRecPtr(TimeLineID *tli)
{
- XLogRecPtr replayPtr;
TimeLineID replayTLI;
- XLogRecPtr receivePtr;- TimeLineID receiveTLI;
XLogRecPtr result;
Assert(am_cascading_walsender || IsSyncingReplicationSlots());
- /*- * We can safely send what's already been replayed. Also, if walreceiver- * is streaming WAL from the same timeline, we can send anything that it- * has streamed, but hasn't been replayed yet.- */-- receivePtr = GetWalRcvFlushRecPtr(NULL, &receiveTLI);- replayPtr = GetXLogReplayRecPtr(&replayTLI);+ result = GetXLogReplayRecPtr(&replayTLI);
if (tli)
*tli = replayTLI;
- result = replayPtr;- if (receiveTLI == replayTLI && receivePtr > replayPtr)- result = receivePtr;+ /*+ * We can safely send what's already been replayed. Also, for physical+ * replication if walreceiver is streaming WAL from the same timeline, we+ * can send anything that it has streamed, but hasn't been replayed yet.+ * If this function is called from slot synchronization (MyWalSnd is NULL+ * in this case), then we should follow physical replication approach as+ * well to be consistent with synchronized_standby_slots expectations.+ */+ if (!MyWalSnd || MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)+ {+ TimeLineID receiveTLI;+ XLogRecPtr receivePtr = GetWalRcvFlushRecPtr(NULL, &receiveTLI);++ if (receiveTLI == replayTLI && receivePtr > result)+ result = receivePtr;+ }
return result;
}
--
2.49.0
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: a.makhmutov@postgrespro.ru, pgsql-bugs@lists.postgresql.org
Subject: Re: Standby server with cascade logical replication could not be properly stopped under load
In-Reply-To: <52138028-7246-421c-9161-4fa108b88070@postgrespro.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox