From: Alexander Lakhin <exclusion@gmail.com>
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Cc: Aleksander Alekseev <aleksander@timescale.com>
Subject: Re: BUG: Former primary node might stuck when started as a standby
Date: Fri, 20 Feb 2026 04:00:00 +0200
Message-ID: <045cab6f-4738-417e-b551-01adba44d6c3@gmail.com> (raw)
In-Reply-To: <OS9PR01MB12149D4F1A2BC23637688CE4DF56BA@OS9PR01MB12149.jpnprd01.prod.outlook.com>
References: <b0102688-6d6c-c86a-db79-e0e91d245b1a@gmail.com>
<CAJ7c6TOqnv7cZ53RfBv9towSDzNOUVQ74WzPeN7LwoKMuW4tOA@mail.gmail.com>
<74ea5e7e-9aa0-4d7a-85df-71a9d200a8d8@gmail.com>
<CAJ7c6TO-_=rfN0Mb1RrChmrQvLtXoWw9aDezwC9g48-Va3E2EQ@mail.gmail.com>
<180bf13e-235d-46a9-9788-7d6cee40bcdc@gmail.com>
<OS9PR01MB12149E6917AAC1560150B918BF565A@OS9PR01MB12149.jpnprd01.prod.outlook.com>
<cfd35cfb-3216-42b1-814b-cc658fd0f099@gmail.com>
<OS9PR01MB12149C0F968A40BA54BD4C16FF561A@OS9PR01MB12149.jpnprd01.prod.outlook.com>
<e1cf52d2-c344-4dfd-a918-e5f20ff04fa2@gmail.com>
<OS9PR01MB121498EFA4CBF3003B83C9BCCF56CA@OS9PR01MB12149.jpnprd01.prod.outlook.com>
<63e55743-669e-4300-a561-7b7ff63723b6@gmail.com>
<OS9PR01MB12149D4F1A2BC23637688CE4DF56BA@OS9PR01MB12149.jpnprd01.prod.outlook.com>
Dear Kuroda-san,
19.02.2026 05:50, Hayato Kuroda (Fujitsu) wrote:
> Dear Alexander,
>
>> Unfortunately, the testing procedure I shared above still produces failures
>> with the patched 009_twophase.pl.
> Hmm, I ran the test for hours, but I could nor reproduce the failure. But let me analyze
> based on your log.
Please look at the attached self-contained script. It works for me (failed
on iterations 6, 12, 2 right now, on my workstation with Ryzen 7900X) --
probably you could adjust number of parallel jobs to reproduce it on your
hardware.
> I have few experience to see the wal_debug output, but background writer seems to
> generate the RUNNING_XACTS record. It's different from my expectation. To confirm,
> did you really enable the injection point? For now 009_twophase can work without
> the `-Dinjection_points=true` but it should be set to avoid random failures.
I think it failed before the injection was set. My log contains:
2026-02-17 07:06:44.313 EET client backend[754908] 009_twophase.pl STATEMENT: PREPARE TRANSACTION 'xact_009_10';
2026-02-17 07:06:44.313 EET client backend[754908] 009_twophase.pl LOG: xlog flush request 0/030227F8; write
0/00000000; flush 0/00000000
2026-02-17 07:06:44.313 EET client backend[754908] 009_twophase.pl STATEMENT: PREPARE TRANSACTION 'xact_009_10';
2026-02-17 07:06:44.313 EET background writer[754333] LOG: INSERT @ 0/03022838: - Standby/RUNNING_XACTS: nextXid 791
latestCompletedXid 788 oldestRunningXid 789; 1 xacts: 789; 1 subxacts: 790
As far as I can see, it corresponds to this place in the test:
SAVEPOINT s1;
INSERT INTO t_009_tbl VALUES (22, 'issued to ${cur_primary_name}');
PREPARE TRANSACTION 'xact_009_10';");
+$cur_primary->wait_for_replay_catchup($cur_standby);
$cur_primary->teardown_node;
$cur_standby->promote;
And as we found out before, wait_for_replay_catchup() before teardown
doesn't help... I can't say for sure, but from my experiments, the test
didn't fail with $cur_primary->stop instead of $cur_primary->teardown_node.
Best regards,
Alexander
set -e
# git reset --hard; git clean -dfx >/dev/null
git restore src/backend/postmaster/bgwriter.c
patch -p1 << EOF
--- a/src/backend/postmaster/bgwriter.c+++ b/src/backend/postmaster/bgwriter.c@@ -69,3 +69,3 @@ int BgWriterDelay = 200;
*/
-#define LOG_SNAPSHOT_INTERVAL_MS 15000+#define LOG_SNAPSHOT_INTERVAL_MS 1@@ -307,3 +307,3 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
- BgWriterDelay /* ms */ , WAIT_EVENT_BGWRITER_MAIN);+ 1 /* ms */ , WAIT_EVENT_BGWRITER_MAIN);@@ -340,3 +340,2 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)- prev_hibernate = can_hibernate;
}
EOF
CFLAGS="-DWAL_DEBUG" ./configure -q --enable-debug --enable-cassert --enable-tap-tests --enable-injection-points
make -s -j8
PROVE_TESTS="t/009*" make -s check -C src/test/recovery
for i in {1..40}; do
cp -r src/test/recovery/ src/test/recovery_$i/;
sed "s|src/test/recovery|src/test/recovery_$i|" -i src/test/recovery_$i/Makefile;
done
echo "wal_debug = on
" >/tmp/temp.config
for i in {1..100}; do
echo "ITERATION $i";
parallel --halt now,fail=1 -j40 --linebuffer --tag TEMP_CONFIG=/tmp/temp.config PROVE_TESTS="t/009*" NO_TEMP_INSTALL=1 timeout 60 make check -s -C src/test/recovery_{} ::: `seq 20` || break;
done
Attachments:
[text/plain] promote-issue-repro.sh.txt (1.3K, ../045cab6f-4738-417e-b551-01adba44d6c3@gmail.com/2-promote-issue-repro.sh.txt)
download | inline diff:
set -e
# git reset --hard; git clean -dfx >/dev/null
git restore src/backend/postmaster/bgwriter.c
patch -p1 << EOF
--- a/src/backend/postmaster/bgwriter.c+++ b/src/backend/postmaster/bgwriter.c@@ -69,3 +69,3 @@ int BgWriterDelay = 200;
*/
-#define LOG_SNAPSHOT_INTERVAL_MS 15000+#define LOG_SNAPSHOT_INTERVAL_MS 1@@ -307,3 +307,3 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
- BgWriterDelay /* ms */ , WAIT_EVENT_BGWRITER_MAIN);+ 1 /* ms */ , WAIT_EVENT_BGWRITER_MAIN);@@ -340,3 +340,2 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)- prev_hibernate = can_hibernate;
}
EOF
CFLAGS="-DWAL_DEBUG" ./configure -q --enable-debug --enable-cassert --enable-tap-tests --enable-injection-points
make -s -j8
PROVE_TESTS="t/009*" make -s check -C src/test/recovery
for i in {1..40}; do
cp -r src/test/recovery/ src/test/recovery_$i/;
sed "s|src/test/recovery|src/test/recovery_$i|" -i src/test/recovery_$i/Makefile;
done
echo "wal_debug = on
" >/tmp/temp.config
for i in {1..100}; do
echo "ITERATION $i";
parallel --halt now,fail=1 -j40 --linebuffer --tag TEMP_CONFIG=/tmp/temp.config PROVE_TESTS="t/009*" NO_TEMP_INSTALL=1 timeout 60 make check -s -C src/test/recovery_{} ::: `seq 20` || break;
done
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: exclusion@gmail.com, kuroda.hayato@fujitsu.com, aleksander@timescale.com
Subject: Re: BUG: Former primary node might stuck when started as a standby
In-Reply-To: <045cab6f-4738-417e-b551-01adba44d6c3@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox