Date: Fri, 28 Aug 2026 09:24:17 +0000
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
To: Daniel Gustafsson <daniel@yesql.se>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
	Zsolt Parragi <zsolt.parragi@percona.com>
Subject: Re: Offline data checksum changes can cause incorrect checksum state
 on standbys
Message-ID: <apFTwa68Qn0S0dIX@bdtpg>
References: <anwm6UPxoVS41QA2@bdtpg>
 <1C2BC974-44FA-487C-8A3B-8135A316A89B@yesql.se>
 <an8z8CJMa46FIxxn@bdtpg>
 <188A1307-1A92-45CA-9DBE-FB0962D3756E@yesql.se>
 <apEdrmkoNT6R9XyJ@bdtpg>
 <6370C439-408D-40F6-B956-55DF19FFE41C@yesql.se>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="5BB69PG6qBWnL8Ha"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <6370C439-408D-40F6-B956-55DF19FFE41C@yesql.se>
Archived-At: <https://www.postgresql.org/message-id/apFTwa68Qn0S0dIX%40bdtpg>
Precedence: bulk


--5BB69PG6qBWnL8Ha
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

Hi,

On Fri, Aug 28, 2026 at 08:53:38AM +0200, Daniel Gustafsson wrote:
> > On 28 Aug 2026, at 07:33, Bertrand Drouvot <bertranddrouvot.pg@gmail.com> wrote:
> 
> > Only looking at 0001 here, I've a few comments:
> 
> Thanks, I've yet to dig into it completely but below are a few quick questions
> to help me along the way.
> 
> > === 1
> > 
> > @@ -9288,17 +9572,33 @@ xlog2_redo(XLogReaderState *record)
> > 
> >        SpinLockAcquire(&XLogCtl->info_lck);
> >        XLogCtl->data_checksum_version = state.new_checksum_state;
> > +       SetLocalDataChecksumState(state.new_checksum_state);
> >        SpinLockRelease(&XLogCtl->info_lck);
> > 
> > This applies every XLOG2_CHECKSUMS record encountered during recovery, even when
> > the same record was applied before.
> > 
> > For example, a standby can replay the final "on" record and then stop cleanly
> > without advancing its restartpoint beyond that record. If checksums are subsequently
> > disabled offline, the next startup begins from the older restartpoint and replays
> > the same on record again, overriding the offline disable.
> 
> Do you mean that checksums are disabled offline across the cluster on all
> nodes, or just on the standby?

Disabling checksums offline on the standby is sufficient although that is not the
intended procedure.

Disabling offline on both the primary and standby also produce the issue.

> > === 2
> > 
> 
> If this can happen then online checksums wouldn't work at all right? 

You’re right, my previous explanation was not fully accurate.

The 0001-specific concern is that a checkpoint can capture
checkPoint.dataChecksumState as inprogress-on, then insert XLOG_CHECKPOINT_REDO
correctly carrying on. The delay protects the flush, but the earlier value remains
stale. The equality check then does not persist on, and recovery no longer adopts
it from the REDO record, so a crash before the following checkpoint completes can
resolve the state back to off.

> Have you been able to construct a repro (with injection points) where a
> REDO record after a CHECKSUM record carries the wrong state?

Not with an injection point, but you can repro that way:

In xlog.c add 3 sleeps (see repro.txt attached):

- In SetDataChecksumsOn() to hold the launcher at inprogress-on.
- In SetDataChecksumsOn() to park it at on before its own checkpoint.
- In CreateCheckPoint() sleep/spin until XLogCtl->data_checksum_version == on.

Then:

start a cluster with initdb --no-data-checksums
Run SELECT pg_enable_data_checksums()
Then within 60s run CHECKPOINT
Once the checkpoint completes (SHOW data_checksums = on but pg_controldata still shows version 3)
pkill -9 the cluster
restart
check SHOW data_checksums: it comes back off.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

--5BB69PG6qBWnL8Ha
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="repro.txt"

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index fa24b00e7d4..672ea66cb0a 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4858,6 +4858,10 @@ SetDataChecksumsOn(void)
 	SpinLockRelease(&XLogCtl->info_lck);
 
 	INJECTION_POINT("datachecksums-enable-checksums-delay", NULL);
+	ereport(LOG, (errmsg("BDTTESTHACK: launcher holding at inprogress-on for 60s")));
+	pg_usleep(60 * 1000000L);
+	ereport(LOG, (errmsg("BDTTESTHACK: launcher releasing, about to write XLOG2_CHECKSUMS(on) and flip")));
+
 	START_CRIT_SECTION();
 	MyProc->delayChkptFlags |= DELAY_CHKPT_START;
 
@@ -4874,6 +4878,10 @@ SetDataChecksumsOn(void)
 
 	INJECTION_POINT("datachecksums-on-before-checkpoint", NULL);
 
+	ereport(LOG, (errmsg("BDTTESTHACK: launcher holding at inprogress-on for 60s")));
+	pg_usleep(60 * 1000000L);
+	ereport(LOG, (errmsg("BDTTESTHACK: launcher releasing, about to write XLOG2_CHECKSUMS(on) and flip")));
+
 	RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
 
 	INJECTION_POINT("datachecksums-on-after-checkpoint", NULL);
@@ -7795,6 +7803,22 @@ CreateCheckPoint(int flags)
 	 */
 	WALInsertLockRelease();
 
+		if (!shutdown && checkPoint.dataChecksumState == PG_DATA_CHECKSUM_INPROGRESS_ON)
+		{
+			int                 i;
+			ereport(LOG, (errmsg("BDTTESTHACK: checkpoint sampled dataChecksumState=inprogress-on, waiting for flip to on")));
+			for (i = 0; i < 600; i++)   /* up to ~60s, then give up */
+			{
+				uint32            v;
+				SpinLockAcquire(&XLogCtl->info_lck);
+				v = XLogCtl->data_checksum_version;
+				SpinLockRelease(&XLogCtl->info_lck);
+				if (v == PG_DATA_CHECKSUM_VERSION)        /* "on" */
+					break;
+				pg_usleep(100 * 1000L);           /* 100ms */
+			}
+			ereport(LOG, (errmsg("TESTHACK: checkpoint resuming after %d iterations; will sample redo_rec and write XLOG_CHECKPOINT_REDO", i)));
+		}
 	/*
 	 * If this is an online checkpoint, we have not yet determined the redo
 	 * point. We do so now by inserting the special XLOG_CHECKPOINT_REDO

--5BB69PG6qBWnL8Ha--