From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
To: Zsolt Parragi <zsolt.parragi@percona.com>
Cc: Daniel Gustafsson <daniel@yesql.se>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: Offline data checksum changes can cause incorrect checksum state on standbys
Date: Tue, 1 Sep 2026 08:38:38 +0000
Message-ID: <apaPDmrlhtXgKR+E@bdtpg> (raw)
In-Reply-To: <CAN4CZFN5sOQvWTXr7Dm1J1HXwczikBfMjkbFQ7QmHsr+VHd5Mg@mail.gmail.com>
References: <apJGv8fffeUfSkCL@bdtpg>
<CAN4CZFMoDj7mbY231r-pmPywwJpneksUiNXDwkNokkveGtobQg@mail.gmail.com>
<apUL3N4IE934qJ08@bdtpg>
<8DCA12FF-0199-403D-A203-666528A25DB6@yesql.se>
<apU0vKrffQDmmv68@bdtpg>
<CAN4CZFNqKg9Ts76r922cKWcOgtH32ocyghQa8NLB-qeCeC1RKg@mail.gmail.com>
<apVnwOuJBb4rHk+B@bdtpg>
<CAN4CZFMFcgfgJ99RYhax-T+YJH=CWLy6ZGANMxf77SrSirw3VQ@mail.gmail.com>
<apWdicFsT+iFxgAH@bdtpg>
<CAN4CZFN5sOQvWTXr7Dm1J1HXwczikBfMjkbFQ7QmHsr+VHd5Mg@mail.gmail.com>
Hi,
On Tue, Sep 01, 2026 at 12:31:42AM +0100, Zsolt Parragi wrote:
> > I found a case where the source's online enable occurs after divergence and was
> > never seen by the target, but replay skips it instead of applying it:
> > ....
> > Maybe the watermark needs timeline context, or pg_rewind needs to adjust it
> > when it comes from the target's divergent history?
>
> Thanks! v8 adds the latter, with a new test case verifying this scenario.
Thanks!
As far the new test:
=== 1
+# Clean switchover back to A; enable checksums online on it.
+$node_b->stop('fast');
+$node_a->promote;
IIUC, the preceding wait_for_catchup() does not cover the shutdown checkpoint
written by stop(). Therefore, the divergence checkpoint in scenario 2 is not
guaranteed to carry off, as described.
=== 2
+enable_data_checksums($node_b, wait => 'on');
+test_checksum_state($node_b, 'on');
...
+$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
+test_checksum_state($node_b, 'on');
The target is already "on" before pg_rewind, so the final assertion does not
prove that the source's enable record was replayed.
Please find attached a small patch addressing those two test comments to apply
on top of v8. What do you think?
=== 3
+ /*
+ * End of the newest XLOG2_CHECKSUMS record this node has written or
+ * applied.
and
+ * would skip them as already applied. Clamp it to the divergence point,
+ * so that every transition record on the source's history takes effect.
+ */
+ if (ControlFile_new.data_checksum_lsn > divergerec)
+ ControlFile_new.data_checksum_lsn = divergerec;
divergerec is not necessarily the end of an XLOG2_CHECKSUMS record, so the
comment no longer describes every value the field may contain. Maybe it should
describe it as the WAL position through which checksum transitions are covered?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.comdiff --git a/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl b/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl--- a/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl+++ b/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl@@ -40,6 +40,19 @@ sub controldata_watermark
return (hex($1) << 32) + hex($2);
}
+sub wait_for_shutdown_checkpoint_replay+{+ my ($primary, $standby) = @_;+ my ($stdout) = run_command([ 'pg_controldata', $primary->data_dir ]);+ $stdout =~ /^Latest checkpoint location:\s*([0-9A-F\/]+)$/m+ or die "checkpoint location missing from pg_controldata output";+ my $shutdown_checkpoint = $1;++ $standby->poll_query_until('postgres',+ "SELECT pg_last_wal_replay_lsn() > '$shutdown_checkpoint'::pg_lsn;")+ or die "standby never replayed the shutdown checkpoint";+}+
# Old primary, checksums off. wal_log_hints is required by pg_rewind
# on a cluster without data checksums.
my $node_a = PostgreSQL::Test::Cluster->new('node_a');
@@ -63,6 +76,7 @@ $node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
# Clean switchover to B; enable checksums online on it.
$node_a->stop('fast');
+wait_for_shutdown_checkpoint_replay($node_a, $node_b);
$node_b->promote;
enable_data_checksums($node_b, wait => 'on');
test_checksum_state($node_b, 'on');
@@ -123,9 +137,11 @@ test_checksum_state($node_a, 'off');
# Clean switchover back to A; enable checksums online on it.
$node_b->stop('fast');
+wait_for_shutdown_checkpoint_replay($node_b, $node_a);
$node_a->promote;
enable_data_checksums($node_a, wait => 'on');
test_checksum_state($node_a, 'on');
+my $source_enable_watermark = controldata_watermark($node_a);
# The old primary restarts on its old timeline and enables checksums
# online independently: both control files say "on", the divergence
@@ -162,6 +178,8 @@ is($node_b->safe_psql('postgres', "SELECT count(*) FROM t_div;"),
'0', 'divergent insert was rewound');
$node_b->stop('fast');
+is(controldata_watermark($node_b), $source_enable_watermark,+ 'rewound node replayed the source checksum transition');
command_ok([ 'pg_checksums', '--check', '-D', $node_b->data_dir ],
'checksums valid on the rewound node');
Attachments:
[text/plain] v8-021-test-fixes.txt (2.2K, ../apaPDmrlhtXgKR+E@bdtpg/2-v8-021-test-fixes.txt)
download | inline diff:diff --git a/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl b/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl--- a/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl+++ b/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl@@ -40,6 +40,19 @@ sub controldata_watermark
return (hex($1) << 32) + hex($2);
}
+sub wait_for_shutdown_checkpoint_replay+{+ my ($primary, $standby) = @_;+ my ($stdout) = run_command([ 'pg_controldata', $primary->data_dir ]);+ $stdout =~ /^Latest checkpoint location:\s*([0-9A-F\/]+)$/m+ or die "checkpoint location missing from pg_controldata output";+ my $shutdown_checkpoint = $1;++ $standby->poll_query_until('postgres',+ "SELECT pg_last_wal_replay_lsn() > '$shutdown_checkpoint'::pg_lsn;")+ or die "standby never replayed the shutdown checkpoint";+}+
# Old primary, checksums off. wal_log_hints is required by pg_rewind
# on a cluster without data checksums.
my $node_a = PostgreSQL::Test::Cluster->new('node_a');
@@ -63,6 +76,7 @@ $node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
# Clean switchover to B; enable checksums online on it.
$node_a->stop('fast');
+wait_for_shutdown_checkpoint_replay($node_a, $node_b);
$node_b->promote;
enable_data_checksums($node_b, wait => 'on');
test_checksum_state($node_b, 'on');
@@ -123,9 +137,11 @@ test_checksum_state($node_a, 'off');
# Clean switchover back to A; enable checksums online on it.
$node_b->stop('fast');
+wait_for_shutdown_checkpoint_replay($node_b, $node_a);
$node_a->promote;
enable_data_checksums($node_a, wait => 'on');
test_checksum_state($node_a, 'on');
+my $source_enable_watermark = controldata_watermark($node_a);
# The old primary restarts on its old timeline and enables checksums
# online independently: both control files say "on", the divergence
@@ -162,6 +178,8 @@ is($node_b->safe_psql('postgres', "SELECT count(*) FROM t_div;"),
'0', 'divergent insert was rewound');
$node_b->stop('fast');
+is(controldata_watermark($node_b), $source_enable_watermark,+ 'rewound node replayed the source checksum transition');
command_ok([ 'pg_checksums', '--check', '-D', $node_b->data_dir ],
'checksums valid on the rewound node');
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bertranddrouvot.pg@gmail.com, zsolt.parragi@percona.com, daniel@yesql.se, pgsql-hackers@lists.postgresql.org
Subject: Re: Offline data checksum changes can cause incorrect checksum state on standbys
In-Reply-To: <apaPDmrlhtXgKR+E@bdtpg>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox