pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedFrom: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
To: Daniel Gustafsson <daniel@yesql.se>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Zsolt Parragi <zsolt.parragi@percona.com>
Subject: Re: Offline data checksum changes can cause incorrect checksum state on standbys
Date: Fri, 28 Aug 2026 09:24:17 +0000
Message-ID: <apFTwa68Qn0S0dIX@bdtpg> (raw)
In-Reply-To: <6370C439-408D-40F6-B956-55DF19FFE41C@yesql.se>
References: <anwm6UPxoVS41QA2@bdtpg>
<1C2BC974-44FA-487C-8A3B-8135A316A89B@yesql.se>
<an8z8CJMa46FIxxn@bdtpg>
<188A1307-1A92-45CA-9DBE-FB0962D3756E@yesql.se>
<apEdrmkoNT6R9XyJ@bdtpg>
<6370C439-408D-40F6-B956-55DF19FFE41C@yesql.se>
Hi,
On Fri, Aug 28, 2026 at 08:53:38AM +0200, Daniel Gustafsson wrote:
> > On 28 Aug 2026, at 07:33, Bertrand Drouvot <bertranddrouvot.pg@gmail.com> wrote:
>
> > Only looking at 0001 here, I've a few comments:
>
> Thanks, I've yet to dig into it completely but below are a few quick questions
> to help me along the way.
>
> > === 1
> >
> > @@ -9288,17 +9572,33 @@ xlog2_redo(XLogReaderState *record)
> >
> > SpinLockAcquire(&XLogCtl->info_lck);
> > XLogCtl->data_checksum_version = state.new_checksum_state;
> > + SetLocalDataChecksumState(state.new_checksum_state);
> > SpinLockRelease(&XLogCtl->info_lck);
> >
> > This applies every XLOG2_CHECKSUMS record encountered during recovery, even when
> > the same record was applied before.
> >
> > For example, a standby can replay the final "on" record and then stop cleanly
> > without advancing its restartpoint beyond that record. If checksums are subsequently
> > disabled offline, the next startup begins from the older restartpoint and replays
> > the same on record again, overriding the offline disable.
>
> Do you mean that checksums are disabled offline across the cluster on all
> nodes, or just on the standby?
Disabling checksums offline on the standby is sufficient although that is not the
intended procedure.
Disabling offline on both the primary and standby also produce the issue.
> > === 2
> >
>
> If this can happen then online checksums wouldn't work at all right?
You’re right, my previous explanation was not fully accurate.
The 0001-specific concern is that a checkpoint can capture
checkPoint.dataChecksumState as inprogress-on, then insert XLOG_CHECKPOINT_REDO
correctly carrying on. The delay protects the flush, but the earlier value remains
stale. The equality check then does not persist on, and recovery no longer adopts
it from the REDO record, so a crash before the following checkpoint completes can
resolve the state back to off.
> Have you been able to construct a repro (with injection points) where a
> REDO record after a CHECKSUM record carries the wrong state?
Not with an injection point, but you can repro that way:
In xlog.c add 3 sleeps (see repro.txt attached):
- In SetDataChecksumsOn() to hold the launcher at inprogress-on.
- In SetDataChecksumsOn() to park it at on before its own checkpoint.
- In CreateCheckPoint() sleep/spin until XLogCtl->data_checksum_version == on.
Then:
start a cluster with initdb --no-data-checksums
Run SELECT pg_enable_data_checksums()
Then within 60s run CHECKPOINT
Once the checkpoint completes (SHOW data_checksums = on but pg_controldata still shows version 3)
pkill -9 the cluster
restart
check SHOW data_checksums: it comes back off.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index fa24b00e7d4..672ea66cb0a 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4858,6 +4858,10 @@ SetDataChecksumsOn(void)
SpinLockRelease(&XLogCtl->info_lck);
INJECTION_POINT("datachecksums-enable-checksums-delay", NULL);
+ ereport(LOG, (errmsg("BDTTESTHACK: launcher holding at inprogress-on for 60s")));
+ pg_usleep(60 * 1000000L);
+ ereport(LOG, (errmsg("BDTTESTHACK: launcher releasing, about to write XLOG2_CHECKSUMS(on) and flip")));
+
START_CRIT_SECTION();
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
@@ -4874,6 +4878,10 @@ SetDataChecksumsOn(void)
INJECTION_POINT("datachecksums-on-before-checkpoint", NULL);
+ ereport(LOG, (errmsg("BDTTESTHACK: launcher holding at inprogress-on for 60s")));
+ pg_usleep(60 * 1000000L);
+ ereport(LOG, (errmsg("BDTTESTHACK: launcher releasing, about to write XLOG2_CHECKSUMS(on) and flip")));
+
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
INJECTION_POINT("datachecksums-on-after-checkpoint", NULL);
@@ -7795,6 +7803,22 @@ CreateCheckPoint(int flags)
*/
WALInsertLockRelease();
+ if (!shutdown && checkPoint.dataChecksumState == PG_DATA_CHECKSUM_INPROGRESS_ON)
+ {
+ int i;
+ ereport(LOG, (errmsg("BDTTESTHACK: checkpoint sampled dataChecksumState=inprogress-on, waiting for flip to on")));
+ for (i = 0; i < 600; i++) /* up to ~60s, then give up */
+ {
+ uint32 v;
+ SpinLockAcquire(&XLogCtl->info_lck);
+ v = XLogCtl->data_checksum_version;
+ SpinLockRelease(&XLogCtl->info_lck);
+ if (v == PG_DATA_CHECKSUM_VERSION) /* "on" */
+ break;
+ pg_usleep(100 * 1000L); /* 100ms */
+ }
+ ereport(LOG, (errmsg("TESTHACK: checkpoint resuming after %d iterations; will sample redo_rec and write XLOG_CHECKPOINT_REDO", i)));
+ }
/*
* If this is an online checkpoint, we have not yet determined the redo
* point. We do so now by inserting the special XLOG_CHECKPOINT_REDO
Attachments:
[text/plain] repro.txt (2.0K, ../apFTwa68Qn0S0dIX@bdtpg/2-repro.txt)
download | inline diff:
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index fa24b00e7d4..672ea66cb0a 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4858,6 +4858,10 @@ SetDataChecksumsOn(void)
SpinLockRelease(&XLogCtl->info_lck);
INJECTION_POINT("datachecksums-enable-checksums-delay", NULL);
+ ereport(LOG, (errmsg("BDTTESTHACK: launcher holding at inprogress-on for 60s")));
+ pg_usleep(60 * 1000000L);
+ ereport(LOG, (errmsg("BDTTESTHACK: launcher releasing, about to write XLOG2_CHECKSUMS(on) and flip")));
+
START_CRIT_SECTION();
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
@@ -4874,6 +4878,10 @@ SetDataChecksumsOn(void)
INJECTION_POINT("datachecksums-on-before-checkpoint", NULL);
+ ereport(LOG, (errmsg("BDTTESTHACK: launcher holding at inprogress-on for 60s")));
+ pg_usleep(60 * 1000000L);
+ ereport(LOG, (errmsg("BDTTESTHACK: launcher releasing, about to write XLOG2_CHECKSUMS(on) and flip")));
+
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
INJECTION_POINT("datachecksums-on-after-checkpoint", NULL);
@@ -7795,6 +7803,22 @@ CreateCheckPoint(int flags)
*/
WALInsertLockRelease();
+ if (!shutdown && checkPoint.dataChecksumState == PG_DATA_CHECKSUM_INPROGRESS_ON)
+ {
+ int i;
+ ereport(LOG, (errmsg("BDTTESTHACK: checkpoint sampled dataChecksumState=inprogress-on, waiting for flip to on")));
+ for (i = 0; i < 600; i++) /* up to ~60s, then give up */
+ {
+ uint32 v;
+ SpinLockAcquire(&XLogCtl->info_lck);
+ v = XLogCtl->data_checksum_version;
+ SpinLockRelease(&XLogCtl->info_lck);
+ if (v == PG_DATA_CHECKSUM_VERSION) /* "on" */
+ break;
+ pg_usleep(100 * 1000L); /* 100ms */
+ }
+ ereport(LOG, (errmsg("TESTHACK: checkpoint resuming after %d iterations; will sample redo_rec and write XLOG_CHECKPOINT_REDO", i)));
+ }
/*
* If this is an online checkpoint, we have not yet determined the redo
* point. We do so now by inserting the special XLOG_CHECKPOINT_REDO
view thread (43+ messages) latest in thread
Message-ID: <apFTwa68Qn0S0dIX@bdtpg>
Permalink: ../apFTwa68Qn0S0dIX@bdtpg/
Also on: postgresql.org/message-id/apFTwa68Qn0S0dIX@bdtpg
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bertranddrouvot.pg@gmail.com, daniel@yesql.se, pgsql-hackers@lists.postgresql.org, zsolt.parragi@percona.com
Subject: Re: Offline data checksum changes can cause incorrect checksum state on standbys
In-Reply-To: <apFTwa68Qn0S0dIX@bdtpg>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox