Date: Tue, 1 Sep 2026 08:38:38 +0000
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
To: Zsolt Parragi <zsolt.parragi@percona.com>
Cc: Daniel Gustafsson <daniel@yesql.se>, pgsql-hackers@lists.postgresql.org
Subject: Re: Offline data checksum changes can cause incorrect checksum state
 on standbys
Message-ID: <apaPDmrlhtXgKR+E@bdtpg>
References: <apJGv8fffeUfSkCL@bdtpg>
 <CAN4CZFMoDj7mbY231r-pmPywwJpneksUiNXDwkNokkveGtobQg@mail.gmail.com>
 <apUL3N4IE934qJ08@bdtpg>
 <8DCA12FF-0199-403D-A203-666528A25DB6@yesql.se>
 <apU0vKrffQDmmv68@bdtpg>
 <CAN4CZFNqKg9Ts76r922cKWcOgtH32ocyghQa8NLB-qeCeC1RKg@mail.gmail.com>
 <apVnwOuJBb4rHk+B@bdtpg>
 <CAN4CZFMFcgfgJ99RYhax-T+YJH=CWLy6ZGANMxf77SrSirw3VQ@mail.gmail.com>
 <apWdicFsT+iFxgAH@bdtpg>
 <CAN4CZFN5sOQvWTXr7Dm1J1HXwczikBfMjkbFQ7QmHsr+VHd5Mg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="gk2ldDDifaa2RZdE"
Content-Disposition: inline
In-Reply-To: 
 <CAN4CZFN5sOQvWTXr7Dm1J1HXwczikBfMjkbFQ7QmHsr+VHd5Mg@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/apaPDmrlhtXgKR%2BE%40bdtpg>
Precedence: bulk


--gk2ldDDifaa2RZdE
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Hi,

On Tue, Sep 01, 2026 at 12:31:42AM +0100, Zsolt Parragi wrote:
> > I found a case where the source's online enable occurs after divergence and was
> > never seen by the target, but replay skips it instead of applying it:
> > ....
> > Maybe the watermark needs timeline context, or pg_rewind needs to adjust it
> > when it comes from the target's divergent history?
> 
> Thanks! v8 adds the latter, with a new test case verifying this scenario.

Thanks!

As far the new test:

=== 1

+# Clean switchover back to A; enable checksums online on it.
+$node_b->stop('fast');
+$node_a->promote;

IIUC, the preceding wait_for_catchup() does not cover the shutdown checkpoint
written by stop(). Therefore, the divergence checkpoint in scenario 2 is not
guaranteed to carry off, as described.

=== 2

+enable_data_checksums($node_b, wait => 'on');
+test_checksum_state($node_b, 'on');
...
+$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
+test_checksum_state($node_b, 'on');

The target is already "on" before pg_rewind, so the final assertion does not
prove that the source's enable record was replayed.

Please find attached a small patch addressing those two test comments to apply
on top of v8. What do you think?

=== 3

+   /*
+    * End of the newest XLOG2_CHECKSUMS record this node has written or
+    * applied.

and

+    * would skip them as already applied.  Clamp it to the divergence point,
+    * so that every transition record on the source's history takes effect.
+    */
+   if (ControlFile_new.data_checksum_lsn > divergerec)
+       ControlFile_new.data_checksum_lsn = divergerec;

divergerec is not necessarily the end of an XLOG2_CHECKSUMS record, so the
comment no longer describes every value the field may contain. Maybe it should
describe it as the WAL position through which checksum transitions are covered?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

--gk2ldDDifaa2RZdE
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="v8-021-test-fixes.txt"

diff --git a/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl b/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl
--- a/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl
+++ b/src/test/modules/test_checksums/t/021_rewind_divergent_transitions.pl
@@ -40,6 +40,19 @@ sub controldata_watermark
 	return (hex($1) << 32) + hex($2);
 }
 
+sub wait_for_shutdown_checkpoint_replay
+{
+	my ($primary, $standby) = @_;
+	my ($stdout) = run_command([ 'pg_controldata', $primary->data_dir ]);
+	$stdout =~ /^Latest checkpoint location:\s*([0-9A-F\/]+)$/m
+	  or die "checkpoint location missing from pg_controldata output";
+	my $shutdown_checkpoint = $1;
+
+	$standby->poll_query_until('postgres',
+		"SELECT pg_last_wal_replay_lsn() > '$shutdown_checkpoint'::pg_lsn;")
+	  or die "standby never replayed the shutdown checkpoint";
+}
+
 # Old primary, checksums off.  wal_log_hints is required by pg_rewind
 # on a cluster without data checksums.
 my $node_a = PostgreSQL::Test::Cluster->new('node_a');
@@ -63,6 +76,7 @@ $node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
 
 # Clean switchover to B; enable checksums online on it.
 $node_a->stop('fast');
+wait_for_shutdown_checkpoint_replay($node_a, $node_b);
 $node_b->promote;
 enable_data_checksums($node_b, wait => 'on');
 test_checksum_state($node_b, 'on');
@@ -123,9 +137,11 @@ test_checksum_state($node_a, 'off');
 
 # Clean switchover back to A; enable checksums online on it.
 $node_b->stop('fast');
+wait_for_shutdown_checkpoint_replay($node_b, $node_a);
 $node_a->promote;
 enable_data_checksums($node_a, wait => 'on');
 test_checksum_state($node_a, 'on');
+my $source_enable_watermark = controldata_watermark($node_a);
 
 # The old primary restarts on its old timeline and enables checksums
 # online independently: both control files say "on", the divergence
@@ -162,6 +178,8 @@ is($node_b->safe_psql('postgres', "SELECT count(*) FROM t_div;"),
 	'0', 'divergent insert was rewound');
 
 $node_b->stop('fast');
+is(controldata_watermark($node_b), $source_enable_watermark,
+	'rewound node replayed the source checksum transition');
 command_ok([ 'pg_checksums', '--check', '-D', $node_b->data_dir ],
 	'checksums valid on the rewound node');
 

--gk2ldDDifaa2RZdE--