public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tomas Vondra <[email protected]>
To: Daniel Gustafsson <[email protected]>
To: SATYANARAYANA NARLAPURAM <[email protected]>
Cc: Ayush Tiwari <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Cc: Heikki Linnakangas <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: Bernd Helmle <[email protected]>
Cc: Michael Paquier <[email protected]>
Cc: Michael Banck <[email protected]>
Subject: Re: Changing the state of data checksums in a running cluster
Date: Mon, 4 May 2026 15:16:34 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<CAHg+QDfRk4-S7DMmdbXJnQ-xF=sUpMAKuh8b83ObLqYVKx5QLA@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAHg+QDfS0Ckw2HT-TWkdvB1pq9rrFzm1Pk_AUwuFL+Er23ZOFA@mail.gmail.com>
<[email protected]>
<CAJTYsWUq96LR=DSh0jcbZLwJekT06Z2PGND6GYaS0SANFC9E7A@mail.gmail.com>
<[email protected]>
<CAHg+QDeGrpZbNZdLjd_T4b43xKEEXZN0HGhkFm-1bkBdyzK7AQ@mail.gmail.com>
<[email protected]>
Hi,
Thanks for getting this feature pushed, and for resolving the failures
reported since the feature freeze. I consider this to be an important
improvement, not just for the feature itself, but also because of all
the useful infrastructure it added.
Attached is a refined version of the TAP tests already posted by Daniel
some time ago [1]. Unfortunately, that .txt did not apply cleanly for
some reason, so here's a better version.
I found these tests quite useful when reasoning about how the patch
behaves in concurrent environment (e.g. with multiple sessions
triggering checksum enable/disable, or with a checkpoint, crashes, etc).
At this point all the tests pass, but there are a couple cases with
correct but slightly surprising behavior, worth discussing. Which is
what this e-mail is going to be about.
I'll explain what the TAP tests aim to do first, and then discuss the
slightly surprising behavior.
It's not meant for inclusion into PG19, at least not in this shape - I
wrote those TAP tests while investigating some of the earlier failures
and/or when wondering about behavior in various situations (sequence of
concurrent steps, race conditions, ...). So it's more of an exhaustive,
and the tests are somewhat redundant (N+1 is often just (N + some small
tweak)).
I can imagine distilling it into a tiny subset, and adding that. But
that's up to discussion. But that's for later.
Let me briefly explain what the various TAP tests aim to do. From the
very beginning, my main concern regarding this patch was race conditions
when updating the shared state about effective data_checksum_version.
Because the state is effectively split into about three or four places:
* LocalDataChecksumVersion (local cache)
* XLogCtl->data_checksum_version (XLogCtl->info_lck)
* ControlFile->data_checksum_version (ControlFileLock)
* state in control file on disk
These pieces are protected by different locks, the protocol for updating
and/or reading the various flags is not trivial (and some of the fixed
issues were due to ControlFile->data_checksum_version being updated from
a place that shouldn't have).
So the primary goal of the TAP tests was to check for race conditions by
leveraging injection points to step through concurrent processes in a
deterministic way. The first couple patches (0001-0004) add debug
logging and injection points into a lot of places. And by "a lot" I mean
~80 new injection points, which is about the number of injection points
we have in master now. Anyway, this allows stepping through concurrent
checksum changes, and also checksum change vs. checkpointer.
Then come the actual TAP tests:
1) 0005-TAP-10-concurrent-checksum-changes.patch
Two concurrent checksum changes. The first one gets paused at an
injection point, then the second one gets initiated.
2) 0006-TAP-11-concurrency-with-checkpoints.patch
A checksum change + checkpoint. The change gets paused at an injection
point, a synchronous checkpoint is performed.
3) 0007-TAP-12-crashes-at-injection-points.patch
Similar to 0006, but with a crash + recovery. A checksum change gets
paused at an injection point, a synchronous checkpoint is performed. The
changes gets wpken up and either completes, or pauses on a different
injection point. A restart/crash happens.
4) 0008-TAP-13-concurrency-with-checkpoint-REDO.patch
Similar to 0007, but the checkpoint is not synchronous - happens in the
background, so that the TAP can step through both sides and interleave
them in an arbitrary way. This matters because the checksum change
updates the different state pieces (XlogCtl/ControlFile), while the
checkpointer reads them to record initial state for REDO etc.
5) 0009-TAP-14-checkpoints-with-crashes.patch
Similar to 0008, except that the steps are more fine grained, and
focused on two particular cases with surprisingly different final state.
AFAIK everything works as expected, except for two cases in the "TAP
012" test. One for the "enabling" direction, one for the "disabling"
direction. I'm going to discuss the "enabling" direction, I believe the
other case is just a mirror with the same root cause.
The TAP 012 tests checksum change with a concurrent checkpoint, followed
by a crash, and tests the final state. It pauses the change at an
injection point, does a checkpoint, proceeds to the next injection
point, crashes and does recovery. The expectation is that the final
state "flips" at some injection point, once it gets further enough, and
stays there. But what actually happens is this:
a) test_checksum_transition(
'disabled', 'enable', undef,
'datachecksums-enable-inprogress-checksums-end',
'datachecksums-enable-checksums-start',
'off');
b) test_checksum_transition(
'disabled', 'enable', undef,
'datachecksums-enable-checksums-start',
'datachecksums-enable-checksums-after-xlog',
'on');
c) test_checksum_transition(
'disabled', 'enable', 'datachecksums-enable-checksums-start',
'datachecksums-enable-checksums-after-xlogs',
'datachecksums-enable-checksums-after-xlogctl',
'off');
This says that if the checkpoint happens after
'datachecksums-enable-inprogress-checksums-end' or after
'datachecksums-enable-checksums-after-xlog', we end up with 'off' (i.e.
enabling checksums fails).
But if the checkpoint happens after
'datachecksums-enable-checksums-start', we end up with "on" (after
recovery).
This is a bit surprising, because that injection point is before
'datachecksums-enable-checksums-after-xlog'. So the enabling process
gets further and further, but the final state flips off -> on -> off,
contradicting the expectation that it changes once.
I haven't quite wrapped my head around it yet, but my understanding is
this is due to a race condition between the checksum launcher (writing
XLOG2_CHECKSUMS and updating the shmem state), and the checkpointer
(reading the shmem state and generating REDO).
The launcher does this sequence of steps:
1) write XLOG2_CHECKSUMS with new state
2) update XLogCtl->data_checksum_version
3) update ControlFile->data_checksum_version
4) UpdateControlFile()
5) emits barrier
while the checkpointer (CreateCheckPoint) does this:
A) read XLogCtl->data_checksum_version (while holding insert locks)
B) insert XLOG_CHECKPOINT_REDO (reads XLogCtl->data_checksum_version)
C) UpdateControlFile()
The outcome depends on how exactly these two sequences interleave. For
example, this can happen:
1) write XLOG2_CHECKSUMS with new state
A) read XLogCtl->data_checksum_version (while holding insert locks)
B) insert XLOG_CHECKPOINT_REDO (reads XLogCtl->data_checksum_version)
C) UpdateControlFile()
2) update XLogCtl->data_checksum_version
3) update ControlFile->data_checksum_version
4) UpdateControlFile()
5) emits barrier
Which means the XLOG_CHECKPOINT_REDO will be after XLOG2_CHECKSUMS (and
so redo won't see it), but the checkpoint will still get the old
checksum state from XLogCtl. And so the outcome is "off", per case (c).
But it can also happen what case (b) does:
A) read XLogCtl->data_checksum_version (while holding insert locks)
B) insert XLOG_CHECKPOINT_REDO (reads XLogCtl->data_checksum_version)
C) UpdateControlFile()
1) write XLOG2_CHECKSUMS with new state
2) update XLogCtl->data_checksum_version
3) update ControlFile->data_checksum_version
4) UpdateControlFile()
5) emits barrier
In which case the REDO will have the old state, but the recovery will
read the XLOG2_CHECKSUMS, and so end up with "on".
This is the root cause of the surprising behavior in TAP 012, I think.
I attempted to trigger these race conditions in TAP 013, but without
much success. In the end I realized it probably needs more control,
waiting for the other process to hit the next injection point before
unpausing the current one. TAP 014 does that, and it shows that with the
right interleaving of steps the (c) case can end up with both "on" and
"off" final state.
As I said, I don't claim I fully understand this yet. But I wouldn't
call this "bug" - AFAICS it won't produce an incorrect final state (I
haven't seen any such cases).
Still, I wonder if there's a potential issue I failed to notice.
The other question I had when looking at this (concurrency with
checkpoints) is what we get by doing
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
whenever updating the state in SetDataChecksums... functions. Because
the only thing that guarantees is the updates happen on one side of the
checkpoint record. What does that give us, actually?
It does not seem to prevent this surprising behavior, and it does not
say the XLOG2_CHECKSUMS happens before/after the XLOG_CHECKPOINT_REDO.
regards
[1]
https://www.postgresql.org/message-id/9197F930-DDEB-4CAC-82A2-16FEC715CCE8%40yesql.se
--
Tomas Vondra
Attachments:
[text/x-patch] 0001-debug-checkpoint-logging.patch (1.2K, ../[email protected]/2-0001-debug-checkpoint-logging.patch)
download | inline diff:
From 696077acf842ce1de0bba320b10c18e5c22870e1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Mon, 13 Apr 2026 14:36:43 +0200
Subject: [PATCH 01/13] debug: checkpoint logging
---
src/backend/access/transam/xlog.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 18d5dee06e0..94b7afeb6c7 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -7568,6 +7568,9 @@ CreateCheckPoint(int flags)
*/
WALInsertLockRelease();
+ elog(LOG, "CreateCheckPoint start redo %X/%X checksums %d",
+ LSN_FORMAT_ARGS(RedoRecPtr), checkPoint.dataChecksumState);
+
/*
* If this is an online checkpoint, we have not yet determined the redo
* point. We do so now by inserting the special XLOG_CHECKPOINT_REDO
@@ -7811,6 +7814,10 @@ CreateCheckPoint(int flags)
*/
END_CRIT_SECTION();
+ elog(LOG, "CreateCheckPoint end lsn %X/%X redo %X/%X checksums %d",
+ LSN_FORMAT_ARGS(recptr), LSN_FORMAT_ARGS(RedoRecPtr),
+ checkPoint.dataChecksumState);
+
/*
* WAL summaries end when the next XLOG_CHECKPOINT_REDO or
* XLOG_CHECKPOINT_SHUTDOWN record is reached. This is the first point
--
2.54.0
[text/x-patch] 0002-debug-state-transitions.patch (14.2K, ../[email protected]/3-0002-debug-state-transitions.patch)
download | inline diff:
From 15c2237b02cbf93558f6f8110f046acce976e0e4 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 8 Apr 2026 22:07:21 +0200
Subject: [PATCH 02/13] debug: state transitions
---
src/backend/access/transam/xlog.c | 125 ++++++++++++++++++++++++++++++
1 file changed, 125 insertions(+)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 94b7afeb6c7..72110d92c8b 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4288,12 +4288,14 @@ InitControlFile(uint64 sysidentifier, uint32 data_checksum_version)
ControlFile->wal_log_hints = wal_log_hints;
ControlFile->track_commit_timestamp = track_commit_timestamp;
ControlFile->data_checksum_version = data_checksum_version;
+ elog(LOG, "InitControlFile ControlFile->data_checksum_version = %u", data_checksum_version);
/*
* Set the data_checksum_version value into XLogCtl, which is where all
* processes get the current value from.
*/
XLogCtl->data_checksum_version = data_checksum_version;
+ elog(LOG, "InitControlFile XLogCtl->data_checksum_version = %u", data_checksum_version);
}
static void
@@ -4753,6 +4755,9 @@ void
SetDataChecksumsOnInProgress(void)
{
uint64 barrier;
+ uint32 data_checksum_version;
+
+ elog(LOG, "SetDataChecksumsOnInProgress / start");
/*
* The state transition is performed in a critical section with
@@ -4764,20 +4769,32 @@ SetDataChecksumsOnInProgress(void)
XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_ON);
SpinLockAcquire(&XLogCtl->info_lck);
+ data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON;
SpinLockRelease(&XLogCtl->info_lck);
+ elog(LOG, "SetDataChecksumsOnInProgress XLogCtl->data_checksum_version %u => %u",
+ data_checksum_version, PG_DATA_CHECKSUM_INPROGRESS_ON);
+
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+
+ elog(LOG, "SetDataChecksumsOnInProgress ControlFile->data_checksum_version %u => %u",
+ ControlFile->data_checksum_version, PG_DATA_CHECKSUM_INPROGRESS_ON);
+
ControlFile->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON;
UpdateControlFile();
LWLockRelease(ControlFileLock);
+ elog(LOG, "SetDataChecksumsOnInProgress / EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON)");
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
+ elog(LOG, "SetDataChecksumsOnInProgress / WaitForProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON)");
WaitForProcSignalBarrier(barrier);
+
+ elog(LOG, "SetDataChecksumsOnInProgress / end");
}
/*
@@ -4806,6 +4823,9 @@ void
SetDataChecksumsOn(void)
{
uint64 barrier;
+ uint32 data_checksum_version;
+
+ elog(LOG, "SetDataChecksumsOn / start");
SpinLockAcquire(&XLogCtl->info_lck);
@@ -4833,25 +4853,39 @@ SetDataChecksumsOn(void)
XLogChecksums(PG_DATA_CHECKSUM_VERSION);
SpinLockAcquire(&XLogCtl->info_lck);
+ data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
SpinLockRelease(&XLogCtl->info_lck);
+ elog(LOG, "SetDataChecksumsOn / XLogCtl->data_checksum_version %u => %u",
+ data_checksum_version, PG_DATA_CHECKSUM_VERSION);
+
/*
* Update the controlfile before waiting since if we have an immediate
* shutdown while waiting we want to come back up with checksums enabled.
*/
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+
+ elog(LOG, "SetDataChecksumsOn / ControlFile->data_checksum_version %u => %u",
+ ControlFile->data_checksum_version, PG_DATA_CHECKSUM_VERSION);
+
ControlFile->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
UpdateControlFile();
LWLockRelease(ControlFileLock);
+ elog(LOG, "SetDataChecksumsOn / EmitProcSignalBarrier(PG_DATA_CHECKSUM_VERSION)");
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
+ elog(LOG, "SetDataChecksumsOn / RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST)");
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
+
+ elog(LOG, "SetDataChecksumsOn / WaitForProcSignalBarrier(PG_DATA_CHECKSUM_VERSION)");
WaitForProcSignalBarrier(barrier);
+
+ elog(LOG, "SetDataChecksumsOn / end");
}
/*
@@ -4871,6 +4905,9 @@ void
SetDataChecksumsOff(void)
{
uint64 barrier;
+ uint32 data_checksum_version;
+
+ elog(LOG, "SetDataChecksumsOff / start");
SpinLockAcquire(&XLogCtl->info_lck);
@@ -4899,20 +4936,32 @@ SetDataChecksumsOff(void)
XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_OFF);
SpinLockAcquire(&XLogCtl->info_lck);
+ data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF;
SpinLockRelease(&XLogCtl->info_lck);
+ elog(LOG, "SetDataChecksumsOff / XLogCtl->data_checksum_version %u => %u",
+ data_checksum_version, PG_DATA_CHECKSUM_INPROGRESS_OFF);
+
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+
+ elog(LOG, "SetDataChecksumsOff / ControlFile->data_checksum_version %u => %u",
+ ControlFile->data_checksum_version, PG_DATA_CHECKSUM_INPROGRESS_OFF);
+
ControlFile->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF;
UpdateControlFile();
LWLockRelease(ControlFileLock);
+ elog(LOG, "SetDataChecksumsOff / EmitProcSignalBarrier(PG_DATA_CHECKSUM_INPROGRESS_OFF)");
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
+ elog(LOG, "SetDataChecksumsOff / RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST)");
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
+
+ elog(LOG, "SetDataChecksumsOff / WaitForProcSignalBarrier(PG_DATA_CHECKSUM_INPROGRESS_OFF)");
WaitForProcSignalBarrier(barrier);
/*
@@ -4937,21 +4986,35 @@ SetDataChecksumsOff(void)
XLogChecksums(PG_DATA_CHECKSUM_OFF);
SpinLockAcquire(&XLogCtl->info_lck);
+ data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_OFF;
SpinLockRelease(&XLogCtl->info_lck);
+ elog(LOG, "SetDataChecksumsOff / XLogCtl->data_checksum_version %u => %u",
+ data_checksum_version, PG_DATA_CHECKSUM_OFF);
+
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+
+ elog(LOG, "SetDataChecksumsOff / ControlFile->data_checksum_version %u => %u",
+ ControlFile->data_checksum_version, PG_DATA_CHECKSUM_OFF);
+
ControlFile->data_checksum_version = PG_DATA_CHECKSUM_OFF;
UpdateControlFile();
LWLockRelease(ControlFileLock);
+ elog(LOG, "SetDataChecksumsOff / EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF)");
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
+ elog(LOG, "SetDataChecksumsOff / RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST)");
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
+
+ elog(LOG, "SetDataChecksumsOff / WaitForProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF)");
WaitForProcSignalBarrier(barrier);
+
+ elog(LOG, "SetDataChecksumsOff / end");
}
/*
@@ -4966,9 +5029,14 @@ void
InitLocalDataChecksumState(void)
{
Assert(InterruptHoldoffCount > 0);
+
+ elog(LOG, "InitLocalDataChecksumState start %d", LocalDataChecksumState);
+
SpinLockAcquire(&XLogCtl->info_lck);
SetLocalDataChecksumState(XLogCtl->data_checksum_version);
SpinLockRelease(&XLogCtl->info_lck);
+
+ elog(LOG, "InitLocalDataChecksumState end %d", LocalDataChecksumState);
}
void
@@ -5278,7 +5346,9 @@ LocalProcessControlFile(bool reset)
LocalControlFile = palloc_object(ControlFileData);
ControlFile = LocalControlFile;
ReadControlFile();
+ elog(LOG, "LocalProcessControlFile start %d", LocalDataChecksumState);
SetLocalDataChecksumState(ControlFile->data_checksum_version);
+ elog(LOG, "LocalProcessControlFile end %d", LocalDataChecksumState);
}
/*
@@ -5430,9 +5500,15 @@ XLOGShmemInit(void *arg)
XLogCtl->InstallXLogFileSegmentActive = false;
XLogCtl->WalWriterSleeping = false;
+ elog(LOG, "XLOGShmemInit / XLogCtl->data_checksum_version %u => %u",
+ XLogCtl->data_checksum_version, ControlFile->data_checksum_version);
+
/* Use the checksum info from control file */
XLogCtl->data_checksum_version = ControlFile->data_checksum_version;
+
+ elog(LOG, "XLOGShmemInit start %d", LocalDataChecksumState);
SetLocalDataChecksumState(XLogCtl->data_checksum_version);
+ elog(LOG, "XLOGShmemInit end %d", LocalDataChecksumState);
SpinLockInit(&XLogCtl->Insert.insertpos_lck);
SpinLockInit(&XLogCtl->info_lck);
@@ -6607,13 +6683,22 @@ StartupXLOG(void)
*/
if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_ON)
{
+ uint32 data_checksum_version;
XLogChecksums(PG_DATA_CHECKSUM_OFF);
+ elog(LOG, "StartupXLOG start %d", LocalDataChecksumState);
+
SpinLockAcquire(&XLogCtl->info_lck);
+ data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_OFF;
SetLocalDataChecksumState(XLogCtl->data_checksum_version);
SpinLockRelease(&XLogCtl->info_lck);
+ elog(LOG, "StartupXLOG XLogCtl->data_checksum_version %u => %u",
+ data_checksum_version, PG_DATA_CHECKSUM_OFF);
+
+ elog(LOG, "StartupXLOG end %d", LocalDataChecksumState);
+
ereport(WARNING,
errmsg("enabling data checksums was interrupted"),
errhint("Data checksum processing must be manually restarted for checksums to be enabled"));
@@ -6627,12 +6712,21 @@ StartupXLOG(void)
*/
if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_OFF)
{
+ uint32 data_checksum_version;
XLogChecksums(PG_DATA_CHECKSUM_OFF);
+ elog(LOG, "StartupXLOG start %d", LocalDataChecksumState);
+
SpinLockAcquire(&XLogCtl->info_lck);
+ data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_OFF;
SetLocalDataChecksumState(XLogCtl->data_checksum_version);
SpinLockRelease(&XLogCtl->info_lck);
+
+ elog(LOG, "StartupXLOG XLogCtl->data_checksum_version %u => %u",
+ data_checksum_version, PG_DATA_CHECKSUM_OFF);
+
+ elog(LOG, "StartupXLOG end %d", LocalDataChecksumState);
}
/*
@@ -7917,6 +8011,8 @@ CreateEndOfRecoveryRecord(void)
{
xl_end_of_recovery xlrec;
XLogRecPtr recptr;
+ uint32 old_data_checksum_version;
+ uint32 new_data_checksum_version;
/* sanity check */
if (!RecoveryInProgress())
@@ -7948,9 +8044,14 @@ CreateEndOfRecoveryRecord(void)
/* start with the latest checksum version (as of the end of recovery) */
SpinLockAcquire(&XLogCtl->info_lck);
+ old_data_checksum_version = ControlFile->data_checksum_version;
+ new_data_checksum_version = XLogCtl->data_checksum_version;
ControlFile->data_checksum_version = XLogCtl->data_checksum_version;
SpinLockRelease(&XLogCtl->info_lck);
+ elog(LOG, "CreateEndOfRecoveryRecord ControlFile->data_checksum_version %u => %u",
+ old_data_checksum_version, new_data_checksum_version);
+
UpdateControlFile();
LWLockRelease(ControlFileLock);
@@ -8293,6 +8394,9 @@ CreateRestartPoint(int flags)
ControlFile->state = DB_SHUTDOWNED_IN_RECOVERY;
}
+ elog(LOG, "CreateRestartPoint ControlFile->data_checksum_version %u => %u",
+ ControlFile->data_checksum_version, lastCheckPoint.dataChecksumState);
+
/* we shall start with the latest checksum version */
ControlFile->data_checksum_version = lastCheckPoint.dataChecksumState;
@@ -8750,6 +8854,8 @@ XLogChecksums(uint32 new_type)
recptr = XLogInsert(RM_XLOG2_ID, XLOG2_CHECKSUMS);
XLogFlush(recptr);
+
+ elog(LOG, "XLogChecksums %d %X/%X", new_type, LSN_FORMAT_ARGS(recptr));
}
/*
@@ -8862,6 +8968,7 @@ xlog_redo(XLogReaderState *record)
{
CheckPoint checkPoint;
TimeLineID replayTLI;
+ uint32 data_checksum_version;
memcpy(&checkPoint, XLogRecGetData(record), sizeof(CheckPoint));
/* In a SHUTDOWN checkpoint, believe the counters exactly */
@@ -8936,6 +9043,10 @@ xlog_redo(XLogReaderState *record)
/* ControlFile->checkPointCopy always tracks the latest ckpt XID */
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+
+ elog(LOG, "xlog_redo / ControlFile->data_checksum_version %u => %u",
+ ControlFile->data_checksum_version, checkPoint.dataChecksumState);
+
ControlFile->checkPointCopy.nextXid = checkPoint.nextXid;
ControlFile->data_checksum_version = checkPoint.dataChecksumState;
@@ -9178,16 +9289,21 @@ xlog_redo(XLogReaderState *record)
{
xl_checkpoint_redo redo_rec;
bool new_state = false;
+ uint32 data_checksum_version;
memcpy(&redo_rec, XLogRecGetData(record), sizeof(xl_checkpoint_redo));
SpinLockAcquire(&XLogCtl->info_lck);
+ data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = redo_rec.data_checksum_version;
SetLocalDataChecksumState(redo_rec.data_checksum_version);
if (redo_rec.data_checksum_version != ControlFile->data_checksum_version)
new_state = true;
SpinLockRelease(&XLogCtl->info_lck);
+ elog(LOG, "xlog_redo / XLogCtl->data_checksum_version %u => %u",
+ data_checksum_version, redo_rec.data_checksum_version);
+
if (new_state)
EmitAndWaitDataChecksumsBarrier(redo_rec.data_checksum_version);
}
@@ -9250,18 +9366,27 @@ xlog2_redo(XLogReaderState *record)
if (info == XLOG2_CHECKSUMS)
{
xl_checksum_state state;
+ uint32 data_checksum_version;
memcpy(&state, XLogRecGetData(record), sizeof(xl_checksum_state));
SpinLockAcquire(&XLogCtl->info_lck);
+ data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = state.new_checksum_state;
SpinLockRelease(&XLogCtl->info_lck);
+ elog(LOG, "xlog2_redo / XLogCtl->data_checksum_version %u => %u",
+ data_checksum_version, state.new_checksum_state);
+
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+ data_checksum_version = ControlFile->data_checksum_version;
ControlFile->data_checksum_version = state.new_checksum_state;
UpdateControlFile();
LWLockRelease(ControlFileLock);
+ elog(LOG, "xlog2_redo / ControlFile->data_checksum_version %u => %u",
+ data_checksum_version, state.new_checksum_state);
+
/*
* Block on a procsignalbarrier to await all processes having seen the
* change to checksum status. Once the barrier has been passed we can
--
2.54.0
[text/x-patch] 0003-debug-checksums-injection-points.patch (9.6K, ../[email protected]/4-0003-debug-checksums-injection-points.patch)
download | inline diff:
From 1e58b18a70502339c5608ba352f6e351f6114359 Mon Sep 17 00:00:00 2001
From: test <test>
Date: Fri, 1 May 2026 22:14:18 +0200
Subject: [PATCH 03/13] debug: checksums injection points
---
src/backend/access/transam/xlog.c | 74 ++++++++++++++++++++++++++++++-
1 file changed, 72 insertions(+), 2 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 72110d92c8b..8dd003d25e2 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4759,6 +4759,13 @@ SetDataChecksumsOnInProgress(void)
elog(LOG, "SetDataChecksumsOnInProgress / start");
+ INJECTION_POINT("datachecksums-enable-inprogress-checksums-start", NULL);
+
+ /* load before critical section */
+ INJECTION_POINT_LOAD("datachecksums-enable-inprogress-checksums-after-xlog");
+ INJECTION_POINT_LOAD("datachecksums-enable-inprogress-checksums-after-xlogctl");
+ INJECTION_POINT_LOAD("datachecksums-enable-inprogress-checksums-after-controlfile");
+
/*
* The state transition is performed in a critical section with
* checkpoints held off to provide crash safety.
@@ -4768,6 +4775,8 @@ SetDataChecksumsOnInProgress(void)
XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_ON);
+ INJECTION_POINT_CACHED("datachecksums-enable-inprogress-checksums-after-xlog", NULL);
+
SpinLockAcquire(&XLogCtl->info_lck);
data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON;
@@ -4776,6 +4785,8 @@ SetDataChecksumsOnInProgress(void)
elog(LOG, "SetDataChecksumsOnInProgress XLogCtl->data_checksum_version %u => %u",
data_checksum_version, PG_DATA_CHECKSUM_INPROGRESS_ON);
+ INJECTION_POINT_CACHED("datachecksums-enable-inprogress-checksums-after-xlogctl", NULL);
+
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
elog(LOG, "SetDataChecksumsOnInProgress ControlFile->data_checksum_version %u => %u",
@@ -4785,15 +4796,21 @@ SetDataChecksumsOnInProgress(void)
UpdateControlFile();
LWLockRelease(ControlFileLock);
+ INJECTION_POINT_CACHED("datachecksums-enable-inprogress-checksums-after-controlfile", NULL);
+
elog(LOG, "SetDataChecksumsOnInProgress / EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON)");
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
+ INJECTION_POINT("datachecksums-enable-inprogress-checksums-before-barrier-wait", NULL);
+
elog(LOG, "SetDataChecksumsOnInProgress / WaitForProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON)");
WaitForProcSignalBarrier(barrier);
+ INJECTION_POINT("datachecksums-enable-inprogress-checksums-end", NULL);
+
elog(LOG, "SetDataChecksumsOnInProgress / end");
}
@@ -4846,12 +4863,20 @@ SetDataChecksumsOn(void)
SpinLockRelease(&XLogCtl->info_lck);
- INJECTION_POINT("datachecksums-enable-checksums-delay", NULL);
+ INJECTION_POINT("datachecksums-enable-checksums-start", NULL);
+
+ /* load before critical section */
+ INJECTION_POINT_LOAD("datachecksums-enable-checksums-after-xlog");
+ INJECTION_POINT_LOAD("datachecksums-enable-checksums-after-xlogctl");
+ INJECTION_POINT_LOAD("datachecksums-enable-checksums-after-controlfile");
+
START_CRIT_SECTION();
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
XLogChecksums(PG_DATA_CHECKSUM_VERSION);
+ INJECTION_POINT_CACHED("datachecksums-enable-checksums-after-xlog", NULL);
+
SpinLockAcquire(&XLogCtl->info_lck);
data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
@@ -4860,6 +4885,8 @@ SetDataChecksumsOn(void)
elog(LOG, "SetDataChecksumsOn / XLogCtl->data_checksum_version %u => %u",
data_checksum_version, PG_DATA_CHECKSUM_VERSION);
+ INJECTION_POINT_CACHED("datachecksums-enable-checksums-after-xlogctl", NULL);
+
/*
* Update the controlfile before waiting since if we have an immediate
* shutdown while waiting we want to come back up with checksums enabled.
@@ -4873,18 +4900,26 @@ SetDataChecksumsOn(void)
UpdateControlFile();
LWLockRelease(ControlFileLock);
+ INJECTION_POINT_CACHED("datachecksums-enable-checksums-after-controlfile", NULL);
+
elog(LOG, "SetDataChecksumsOn / EmitProcSignalBarrier(PG_DATA_CHECKSUM_VERSION)");
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
+ INJECTION_POINT("datachecksums-enable-checksums-before-checkpoint", NULL);
+
elog(LOG, "SetDataChecksumsOn / RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST)");
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
+ INJECTION_POINT("datachecksums-enable-checksums-before-barrier-wait", NULL);
+
elog(LOG, "SetDataChecksumsOn / WaitForProcSignalBarrier(PG_DATA_CHECKSUM_VERSION)");
WaitForProcSignalBarrier(barrier);
+ INJECTION_POINT("datachecksums-enable-checksums-end", NULL);
+
elog(LOG, "SetDataChecksumsOn / end");
}
@@ -4930,11 +4965,20 @@ SetDataChecksumsOff(void)
{
SpinLockRelease(&XLogCtl->info_lck);
+ INJECTION_POINT("datachecksums-disable-inprogress-checksums-start", NULL);
+
+ /* load before critical section */
+ INJECTION_POINT_LOAD("datachecksums-disable-inprogress-checksums-after-xlog");
+ INJECTION_POINT_LOAD("datachecksums-disable-inprogress-checksums-after-xlogctl");
+ INJECTION_POINT_LOAD("datachecksums-disable-inprogress-checksums-after-controlfile");
+
START_CRIT_SECTION();
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_OFF);
+ INJECTION_POINT_CACHED("datachecksums-disable-inprogress-checksums-after-xlog", NULL);
+
SpinLockAcquire(&XLogCtl->info_lck);
data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF;
@@ -4943,6 +4987,8 @@ SetDataChecksumsOff(void)
elog(LOG, "SetDataChecksumsOff / XLogCtl->data_checksum_version %u => %u",
data_checksum_version, PG_DATA_CHECKSUM_INPROGRESS_OFF);
+ INJECTION_POINT_CACHED("datachecksums-disable-inprogress-checksums-after-xlogctl", NULL);
+
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
elog(LOG, "SetDataChecksumsOff / ControlFile->data_checksum_version %u => %u",
@@ -4952,15 +4998,21 @@ SetDataChecksumsOff(void)
UpdateControlFile();
LWLockRelease(ControlFileLock);
+ INJECTION_POINT_CACHED("datachecksums-disable-inprogress-checksums-after-controlfile", NULL);
+
elog(LOG, "SetDataChecksumsOff / EmitProcSignalBarrier(PG_DATA_CHECKSUM_INPROGRESS_OFF)");
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
+ INJECTION_POINT("datachecksums-disable-inprogress-checksums-before-checkpoint", NULL);
+
elog(LOG, "SetDataChecksumsOff / RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST)");
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
+ INJECTION_POINT("datachecksums-disable-inprogress-checksums-before-barrier-wait", NULL);
+
elog(LOG, "SetDataChecksumsOff / WaitForProcSignalBarrier(PG_DATA_CHECKSUM_INPROGRESS_OFF)");
WaitForProcSignalBarrier(barrier);
@@ -4979,12 +5031,21 @@ SetDataChecksumsOff(void)
SpinLockRelease(&XLogCtl->info_lck);
}
+ INJECTION_POINT("datachecksums-disable-checksums-start", NULL);
+
+ /* load before critical section */
+ INJECTION_POINT_LOAD("datachecksums-disable-checksums-after-xlog");
+ INJECTION_POINT_LOAD("datachecksums-disable-checksums-after-xlogctl");
+ INJECTION_POINT_LOAD("datachecksums-disable-checksums-after-controlfile");
+
START_CRIT_SECTION();
/* Ensure that we don't incur a checkpoint during disabling checksums */
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
XLogChecksums(PG_DATA_CHECKSUM_OFF);
+ INJECTION_POINT_CACHED("datachecksums-disable-checksums-after-xlog", NULL);
+
SpinLockAcquire(&XLogCtl->info_lck);
data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_OFF;
@@ -4993,6 +5054,8 @@ SetDataChecksumsOff(void)
elog(LOG, "SetDataChecksumsOff / XLogCtl->data_checksum_version %u => %u",
data_checksum_version, PG_DATA_CHECKSUM_OFF);
+ INJECTION_POINT_CACHED("datachecksums-disable-checksums-after-xlogctl", NULL);
+
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
elog(LOG, "SetDataChecksumsOff / ControlFile->data_checksum_version %u => %u",
@@ -5002,18 +5065,26 @@ SetDataChecksumsOff(void)
UpdateControlFile();
LWLockRelease(ControlFileLock);
+ INJECTION_POINT_CACHED("datachecksums-disable-checksums-after-controlfile", NULL);
+
elog(LOG, "SetDataChecksumsOff / EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF)");
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
+ INJECTION_POINT("datachecksums-disable-checksums-before-checkpoint", NULL);
+
elog(LOG, "SetDataChecksumsOff / RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST)");
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
+ INJECTION_POINT("datachecksums-disable-checksums-before-barrier-wait", NULL);
+
elog(LOG, "SetDataChecksumsOff / WaitForProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF)");
WaitForProcSignalBarrier(barrier);
+ INJECTION_POINT("datachecksums-disable-checksums-end", NULL);
+
elog(LOG, "SetDataChecksumsOff / end");
}
@@ -8968,7 +9039,6 @@ xlog_redo(XLogReaderState *record)
{
CheckPoint checkPoint;
TimeLineID replayTLI;
- uint32 data_checksum_version;
memcpy(&checkPoint, XLogRecGetData(record), sizeof(CheckPoint));
/* In a SHUTDOWN checkpoint, believe the counters exactly */
--
2.54.0
[text/x-patch] 0004-debug-checkpointer-injection-points.patch (6.0K, ../[email protected]/5-0004-debug-checkpointer-injection-points.patch)
download | inline diff:
From 0c3dd8ac1df7fd1686374555eb766e4517c4c1a8 Mon Sep 17 00:00:00 2001
From: test <test>
Date: Fri, 1 May 2026 22:15:52 +0200
Subject: [PATCH 04/13] debug: checkpointer injection points
---
src/backend/access/transam/xlog.c | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 8dd003d25e2..7c4e412093c 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4762,6 +4762,7 @@ SetDataChecksumsOnInProgress(void)
INJECTION_POINT("datachecksums-enable-inprogress-checksums-start", NULL);
/* load before critical section */
+ INJECTION_POINT_LOAD("datachecksums-enable-inprogress-checksums-before-xlog");
INJECTION_POINT_LOAD("datachecksums-enable-inprogress-checksums-after-xlog");
INJECTION_POINT_LOAD("datachecksums-enable-inprogress-checksums-after-xlogctl");
INJECTION_POINT_LOAD("datachecksums-enable-inprogress-checksums-after-controlfile");
@@ -4773,6 +4774,8 @@ SetDataChecksumsOnInProgress(void)
START_CRIT_SECTION();
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
+ INJECTION_POINT_CACHED("datachecksums-enable-inprogress-checksums-before-xlog", NULL);
+
XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_ON);
INJECTION_POINT_CACHED("datachecksums-enable-inprogress-checksums-after-xlog", NULL);
@@ -4866,6 +4869,7 @@ SetDataChecksumsOn(void)
INJECTION_POINT("datachecksums-enable-checksums-start", NULL);
/* load before critical section */
+ INJECTION_POINT_LOAD("datachecksums-enable-checksums-before-xlog");
INJECTION_POINT_LOAD("datachecksums-enable-checksums-after-xlog");
INJECTION_POINT_LOAD("datachecksums-enable-checksums-after-xlogctl");
INJECTION_POINT_LOAD("datachecksums-enable-checksums-after-controlfile");
@@ -4873,6 +4877,8 @@ SetDataChecksumsOn(void)
START_CRIT_SECTION();
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
+ INJECTION_POINT_CACHED("datachecksums-enable-checksums-before-xlog", NULL);
+
XLogChecksums(PG_DATA_CHECKSUM_VERSION);
INJECTION_POINT_CACHED("datachecksums-enable-checksums-after-xlog", NULL);
@@ -4968,6 +4974,7 @@ SetDataChecksumsOff(void)
INJECTION_POINT("datachecksums-disable-inprogress-checksums-start", NULL);
/* load before critical section */
+ INJECTION_POINT_LOAD("datachecksums-disable-inprogress-checksums-before-xlog");
INJECTION_POINT_LOAD("datachecksums-disable-inprogress-checksums-after-xlog");
INJECTION_POINT_LOAD("datachecksums-disable-inprogress-checksums-after-xlogctl");
INJECTION_POINT_LOAD("datachecksums-disable-inprogress-checksums-after-controlfile");
@@ -4975,6 +4982,8 @@ SetDataChecksumsOff(void)
START_CRIT_SECTION();
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
+ INJECTION_POINT_CACHED("datachecksums-disable-inprogress-checksums-before-xlog", NULL);
+
XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_OFF);
INJECTION_POINT_CACHED("datachecksums-disable-inprogress-checksums-after-xlog", NULL);
@@ -5034,6 +5043,7 @@ SetDataChecksumsOff(void)
INJECTION_POINT("datachecksums-disable-checksums-start", NULL);
/* load before critical section */
+ INJECTION_POINT_LOAD("datachecksums-disable-checksums-before-xlog");
INJECTION_POINT_LOAD("datachecksums-disable-checksums-after-xlog");
INJECTION_POINT_LOAD("datachecksums-disable-checksums-after-xlogctl");
INJECTION_POINT_LOAD("datachecksums-disable-checksums-after-controlfile");
@@ -5042,6 +5052,8 @@ SetDataChecksumsOff(void)
/* Ensure that we don't incur a checkpoint during disabling checksums */
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
+ INJECTION_POINT_CACHED("datachecksums-disable-checksums-before-xlog", NULL);
+
XLogChecksums(PG_DATA_CHECKSUM_OFF);
INJECTION_POINT_CACHED("datachecksums-disable-checksums-after-xlog", NULL);
@@ -7611,6 +7623,12 @@ CreateCheckPoint(int flags)
/* Run these points outside the critical section. */
INJECTION_POINT("create-checkpoint-initial", NULL);
INJECTION_POINT_LOAD("create-checkpoint-run");
+ INJECTION_POINT_LOAD("checkpoint-before-redo");
+ INJECTION_POINT_LOAD("checkpoint-before-xlogctl-checksums");
+ INJECTION_POINT_LOAD("checkpoint-after-xlogctl-checksums");
+ INJECTION_POINT_LOAD("checkpoint-before-redo-checksums");
+ INJECTION_POINT_LOAD("checkpoint-before-redo-wal");
+ INJECTION_POINT_LOAD("checkpoint-after-redo-wal");
/*
* Use a critical section to force system panic if we have trouble.
@@ -7676,6 +7694,8 @@ CreateCheckPoint(int flags)
else
checkPoint.PrevTimeLineID = checkPoint.ThisTimeLineID;
+ INJECTION_POINT_CACHED("checkpoint-before-redo", NULL);
+
/*
* We must block concurrent insertions while examining insert state.
*/
@@ -7684,6 +7704,8 @@ CreateCheckPoint(int flags)
checkPoint.fullPageWrites = Insert->fullPageWrites;
checkPoint.wal_level = wal_level;
+ INJECTION_POINT_CACHED("checkpoint-before-xlogctl-checksums", NULL);
+
/*
* Get the current data_checksum_version value from xlogctl, valid at the
* time of the checkpoint.
@@ -7692,6 +7714,8 @@ CreateCheckPoint(int flags)
checkPoint.dataChecksumState = XLogCtl->data_checksum_version;
SpinLockRelease(&XLogCtl->info_lck);
+ INJECTION_POINT_CACHED("checkpoint-after-xlogctl-checksums", NULL);
+
if (shutdown)
{
XLogRecPtr curInsert = XLogBytePosToRecPtr(Insert->CurrBytePos);
@@ -7749,6 +7773,8 @@ CreateCheckPoint(int flags)
{
xl_checkpoint_redo redo_rec;
+ INJECTION_POINT_CACHED("checkpoint-before-redo-checksums", NULL);
+
WALInsertLockAcquire();
redo_rec.wal_level = wal_level;
SpinLockAcquire(&XLogCtl->info_lck);
@@ -7756,6 +7782,8 @@ CreateCheckPoint(int flags)
SpinLockRelease(&XLogCtl->info_lck);
WALInsertLockRelease();
+ INJECTION_POINT_CACHED("checkpoint-before-redo-wal", NULL);
+
/* Include WAL level in record for WAL summarizer's benefit. */
XLogBeginInsert();
XLogRegisterData(&redo_rec, sizeof(xl_checkpoint_redo));
@@ -7768,6 +7796,8 @@ CreateCheckPoint(int flags)
* checkpoint is complete.
*/
checkPoint.redo = RedoRecPtr;
+
+ INJECTION_POINT_CACHED("checkpoint-after-redo-wal", NULL);
}
/* Update the info_lck-protected copy of RedoRecPtr as well */
--
2.54.0
[text/x-patch] 0005-TAP-10-concurrent-checksum-changes.patch (16.0K, ../[email protected]/6-0005-TAP-10-concurrent-checksum-changes.patch)
download | inline diff:
From 24ab81ae666ed9de683091d6feecbf80b2dd5bb1 Mon Sep 17 00:00:00 2001
From: test <test>
Date: Fri, 1 May 2026 22:14:49 +0200
Subject: [PATCH 05/13] TAP 10: concurrent checksum changes
---
.../t/010_concurrent_changes.pl | 296 ++++++++++++++++++
1 file changed, 296 insertions(+)
create mode 100644 src/test/modules/test_checksums/t/010_concurrent_changes.pl
diff --git a/src/test/modules/test_checksums/t/010_concurrent_changes.pl b/src/test/modules/test_checksums/t/010_concurrent_changes.pl
new file mode 100644
index 00000000000..1b17b496033
--- /dev/null
+++ b/src/test/modules/test_checksums/t/010_concurrent_changes.pl
@@ -0,0 +1,296 @@
+
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test suite for testing enabling data checksums in an online cluster with
+# injection point tests injecting failures into the processing
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use DataChecksums::Utils;
+
+# This test suite is expensive, or very expensive, to execute. There are two
+# PG_TEST_EXTRA options for running it, "checksum" for a pared-down test suite
+# an "checksum_extended" for the full suite. The full suite can run for hours
+# on slow or constrained systems.
+my $extended = undef;
+if ($ENV{PG_TEST_EXTRA})
+{
+ $extended = 1 if ($ENV{PG_TEST_EXTRA} =~ /\bchecksum_extended\b/);
+ plan skip_all => 'Expensive data checksums test disabled'
+ unless ($ENV{PG_TEST_EXTRA} =~ /\bchecksum(_extended)?\b/);
+}
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# ---------------------------------------------------------------------------
+# Test cluster setup
+#
+
+# Initiate testcluster
+my $node = PostgreSQL::Test::Cluster->new('injection_node');
+$node->init(no_data_checksums => 1);
+$node->start;
+
+# Set up test environment
+$node->safe_psql('postgres', 'CREATE EXTENSION test_checksums;');
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+my $pgbench = undef;
+my $scalefactor = ($extended ? 10 : 1);
+my $node_loglocation = 0;
+
+$node->command_ok(
+ [
+ 'pgbench', '-p', $node->port, '-i',
+ '-s', $scalefactor, '-q', 'postgres'
+ ]);
+
+# Start a pgbench run in the background against the server specified via the
+# port passed as parameter.
+sub background_rw_pgbench
+{
+ my $port = shift;
+
+ # If a previous pgbench is still running, start by shutting it down.
+ $pgbench->finish if $pgbench;
+
+ my $clients = 1;
+ my $runtime = 2;
+
+ if ($extended)
+ {
+ # Randomize the number of pgbench clients a bit (range 1-16)
+ $clients = 1 + int(rand(15));
+ $runtime = 600;
+ }
+ my @cmd = ('pgbench', '-p', $port, '-T', $runtime, '-c', $clients);
+
+ # Randomize whether we spawn connections or not
+ push(@cmd, '-C') if ($extended && cointoss);
+ # Finally add the database name to use
+ push(@cmd, 'postgres');
+
+ $pgbench = IPC::Run::start(
+ \@cmd,
+ '<' => '/dev/null',
+ '>' => '/dev/null',
+ '2>' => '/dev/null',
+ IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
+}
+
+sub attach_injection_point
+{
+ my ($node, $point) = @_;
+ note('attaching injection point: ' . $point);
+ $node->safe_psql('postgres',
+ "SELECT injection_points_attach('" . $point . "','wait');"
+ );
+}
+
+sub wait_injection_point
+{
+ my ($node, $point) = @_;
+
+ note("waiting for the injection point to be hit");
+ $node->poll_query_until(
+ 'postgres',
+ "SELECT COUNT(*) FROM pg_catalog.pg_stat_activity WHERE wait_event = '" . $point . "'",
+ '1');
+}
+
+sub wakeup_injection_point
+{
+ my ($node, $point) = @_;
+
+ note('waking the init injection point: ' . $point);
+ $node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('" . $point . "');"
+ );
+
+ note("detaching the init injection point");
+ $node->safe_psql('postgres',
+ "SELECT injection_points_detach('" . $point . "');");
+}
+
+# Test behavior with two concurrent checksum transitions.
+#
+# The cluster is initialized into checksum state $start (enabled/disabled),
+# and two changes are initiated in a controled way. The $first change gets
+# paused at a selected injection point $point, at which point $second change
+# gets initiated. Once the checksum resolves the cluster state, the state
+# is compared to the expected $final state.
+#
+# arguments:
+#
+# - start - starting checksum state (enabled/disabled)
+# - first - first checksum change
+# - second - second checksum change
+# - init - initial injection point (see comment later)
+# - point - injection point the first change should wait on
+# - final - expected checksum state at the end
+#
+# The injection point is triggered only by the datachecksum launcher, and
+# there can be only one such process. So there's no risk of hitting the
+# injection point by both changes.
+#
+# XXX Some of the injection points are in a critical section, which does
+# not allow memory allocations etc. INJECTION_POINT_LOAD/_CACHED handles
+# just private memory allocation, but 'wait' requires a shmem allocation.
+# To deal with that, we setup an injection point $init outside a critical
+# section to initialize the shmem stuff needed by 'wait'.
+#
+# XXX The first change gets "unpaused" after the second checksum change
+# gets initiated, but maybe that's not a sufficiently deterministic?
+#
+# XXX Should we detach the injection point $point before initiating the
+# second change? I'm not sure if there's a small race condition because
+# of waking up and detaching the injection point.
+sub test_checksum_transition
+{
+ my ($start, $first, $second, $init, $point, $final) = @_;
+
+ # print the current test instructions, both into TAP output and into
+ # the server log, to make correlation easier
+
+ my $initstr = 'undef';
+ $initstr = $init if defined($init);
+
+ $node->safe_psql('postgres',
+ "SELECT '========== " . $start . " / " . $first . " / " . $second . " / " . $initstr . " / " . $point . " / " . $final . " =========='");
+
+ note($start . " / " . $first . " / " . $second . " / " . $initstr . " / " . $point . " / " . $final);
+
+ # put the cluster into the initial checksum state, synchronously
+ note('changing checksums into initial state: ' . $start);
+ enable_data_checksums($node, wait => 'on') if ($start eq 'enabled');
+ disable_data_checksums($node, wait => 'off') if ($start eq 'disabled');
+
+ # Wait on an injection point outside a critical section, to initialize
+ # the shmem (which can't be done in critical section).
+ attach_injection_point($node, $init) if defined($init);
+
+ # Wait on the injection point, to pause the first checksum change.
+ note('attaching injection point: ' . $point);
+ $node->safe_psql('postgres',
+ "SELECT injection_points_attach('" . $point . "','wait');"
+ );
+
+ # Trigger the checksum change, asynchronously
+ note("triggering first checksum change: " . $first);
+ enable_data_checksums($node) if ($first eq 'enable');
+ disable_data_checksums($node) if ($first eq 'disable');
+
+ # Handle the initial injection point - wait, wakeup and detatch. This
+ # initializes the shmem for the 'wait' action.
+ wait_injection_point($node, $init) if defined($init);
+ wakeup_injection_point($node, $init) if defined($init);
+
+ # Wait for the actual injection point to be hit by the state change.
+ wait_injection_point($node, $point);
+
+ # The first checksum state change is waitinig on the injection point.
+ # Trigger the concurrent change (also asynchronously).
+ note("triggering second checksum change: " . $second);
+ enable_data_checksums($node) if ($second eq 'enable');
+ disable_data_checksums($node) if ($second eq 'disable');
+
+ # Wake the injection point, so that the first change can proceed.
+ wakeup_injection_point($node, $point);
+
+ # Wait until there are no ongoing checksum changes, which we determine
+ # by looking for a checksum launcher process.
+ note('wait for the checksum launcher to exit');
+ $node->poll_query_until('postgres',
+ "SELECT count(*) = 0 "
+ . "FROM pg_catalog.pg_stat_activity "
+ . "WHERE backend_type = 'datachecksum launcher';");
+
+ # Does the final checksum state match the expected state?
+ test_checksum_state($node, $final);
+
+ # Since the log isn't being written to now, parse the log and check
+ # for instances of checksum verification failures.
+ my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile,
+ $node_loglocation);
+ unlike(
+ $log,
+ qr/page verification failed,.+\d$/,
+ "no checksum validation errors in primary log (during WAL recovery)"
+ );
+ $node_loglocation = -s $node->logfile;
+}
+
+# Start the test suite with pgbench running.
+background_rw_pgbench($node->port);
+
+# concurrent enable + disable, different injection points in the "enable" process
+test_checksum_transition('disabled', 'enable', 'disable', undef, 'datachecksums-enable-inprogress-checksums-start', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-xlog', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-xlogctl', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-controlfile', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', undef, 'datachecksums-enable-inprogress-checksums-before-barrier-wait', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', undef, 'datachecksums-enable-inprogress-checksums-end', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', undef, 'datachecksums-enable-checksums-start', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-xlog', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-xlogctl', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-controlfile', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', undef, 'datachecksums-enable-checksums-before-checkpoint', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', undef, 'datachecksums-enable-checksums-before-barrier-wait', 'off');
+test_checksum_transition('disabled', 'enable', 'disable', undef, 'datachecksums-enable-checksums-end', 'off');
+
+# concurrent enable + enable, different injection points in the "enable" process
+test_checksum_transition('disabled', 'enable', 'enable', undef, 'datachecksums-enable-inprogress-checksums-start', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-xlog', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-xlogctl', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-controlfile', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', undef, 'datachecksums-enable-inprogress-checksums-before-barrier-wait', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', undef, 'datachecksums-enable-inprogress-checksums-end', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', undef, 'datachecksums-enable-checksums-start', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-xlog', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-xlogctl', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-controlfile', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', undef, 'datachecksums-enable-checksums-before-checkpoint', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', undef, 'datachecksums-enable-checksums-before-barrier-wait', 'on');
+test_checksum_transition('disabled', 'enable', 'enable', undef, 'datachecksums-enable-checksums-end', 'on');
+
+# concurrent disable + disable, different injection points in the "disable" process
+test_checksum_transition('enabled', 'disable', 'disable', undef, 'datachecksums-disable-inprogress-checksums-start', 'off');
+test_checksum_transition('enabled', 'disable', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-xlog', 'off');
+test_checksum_transition('enabled', 'disable', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-xlogctl', 'off');
+test_checksum_transition('enabled', 'disable', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-controlfile', 'off');
+test_checksum_transition('enabled', 'disable', 'disable', undef, 'datachecksums-disable-inprogress-checksums-before-barrier-wait', 'off');
+test_checksum_transition('enabled', 'disable', 'disable', undef, 'datachecksums-disable-checksums-start', 'off');
+test_checksum_transition('enabled', 'disable', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-xlog', 'off');
+test_checksum_transition('enabled', 'disable', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-xlogctl', 'off');
+test_checksum_transition('enabled', 'disable', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-controlfile', 'off');
+test_checksum_transition('enabled', 'disable', 'disable', undef, 'datachecksums-disable-checksums-before-checkpoint', 'off');
+test_checksum_transition('enabled', 'disable', 'disable', undef, 'datachecksums-disable-checksums-before-barrier-wait', 'off');
+test_checksum_transition('enabled', 'disable', 'disable', undef, 'datachecksums-disable-checksums-end', 'off');
+
+# concurrent disable + enable, different injection points in the "disable" process
+test_checksum_transition('enabled', 'disable', 'enable', undef, 'datachecksums-disable-inprogress-checksums-start', 'on');
+test_checksum_transition('enabled', 'disable', 'enable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-xlog', 'on');
+test_checksum_transition('enabled', 'disable', 'enable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-xlogctl', 'on');
+test_checksum_transition('enabled', 'disable', 'enable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-controlfile', 'on');
+test_checksum_transition('enabled', 'disable', 'enable', undef, 'datachecksums-disable-inprogress-checksums-before-barrier-wait', 'on');
+test_checksum_transition('enabled', 'disable', 'enable', undef, 'datachecksums-disable-checksums-start', 'on');
+test_checksum_transition('enabled', 'disable', 'enable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-xlog', 'on');
+test_checksum_transition('enabled', 'disable', 'enable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-xlogctl', 'on');
+test_checksum_transition('enabled', 'disable', 'enable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-controlfile', 'on');
+test_checksum_transition('enabled', 'disable', 'enable', undef, 'datachecksums-disable-checksums-before-checkpoint', 'on');
+test_checksum_transition('enabled', 'disable', 'enable', undef, 'datachecksums-disable-checksums-before-barrier-wait', 'on');
+test_checksum_transition('enabled', 'disable', 'enable', undef, 'datachecksums-disable-checksums-end', 'on');
+
+$node->stop;
+done_testing();
--
2.54.0
[text/x-patch] 0006-TAP-11-concurrency-with-checkpoints.patch (12.2K, ../[email protected]/7-0006-TAP-11-concurrency-with-checkpoints.patch)
download | inline diff:
From ae1ef6cd0a2c4d2fcddb34db282c55a7f2fe509c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Apr 2026 22:22:48 +0200
Subject: [PATCH 06/13] TAP 11: concurrency with checkpoints
---
.../t/011_concurrent_checkpoint.pl | 276 ++++++++++++++++++
1 file changed, 276 insertions(+)
create mode 100644 src/test/modules/test_checksums/t/011_concurrent_checkpoint.pl
diff --git a/src/test/modules/test_checksums/t/011_concurrent_checkpoint.pl b/src/test/modules/test_checksums/t/011_concurrent_checkpoint.pl
new file mode 100644
index 00000000000..4cea74914d4
--- /dev/null
+++ b/src/test/modules/test_checksums/t/011_concurrent_checkpoint.pl
@@ -0,0 +1,276 @@
+
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test suite for testing enabling data checksums in an online cluster with
+# injection point tests injecting failures into the processing
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use DataChecksums::Utils;
+
+# This test suite is expensive, or very expensive, to execute. There are two
+# PG_TEST_EXTRA options for running it, "checksum" for a pared-down test suite
+# an "checksum_extended" for the full suite. The full suite can run for hours
+# on slow or constrained systems.
+my $extended = undef;
+if ($ENV{PG_TEST_EXTRA})
+{
+ $extended = 1 if ($ENV{PG_TEST_EXTRA} =~ /\bchecksum_extended\b/);
+ plan skip_all => 'Expensive data checksums test disabled'
+ unless ($ENV{PG_TEST_EXTRA} =~ /\bchecksum(_extended)?\b/);
+}
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# ---------------------------------------------------------------------------
+# Test cluster setup
+#
+
+# Initiate testcluster
+my $node = PostgreSQL::Test::Cluster->new('injection_node');
+$node->init(no_data_checksums => 1);
+$node->start;
+
+# Set up test environment
+$node->safe_psql('postgres', 'CREATE EXTENSION test_checksums;');
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+my $pgbench = undef;
+my $scalefactor = ($extended ? 10 : 1);
+my $node_loglocation = 0;
+
+$node->command_ok(
+ [
+ 'pgbench', '-p', $node->port, '-i',
+ '-s', $scalefactor, '-q', 'postgres'
+ ]);
+
+# Start a pgbench run in the background against the server specified via the
+# port passed as parameter.
+sub background_rw_pgbench
+{
+ my $port = shift;
+
+ # If a previous pgbench is still running, start by shutting it down.
+ $pgbench->finish if $pgbench;
+
+ my $clients = 1;
+ my $runtime = 2;
+
+ if ($extended)
+ {
+ # Randomize the number of pgbench clients a bit (range 1-16)
+ $clients = 1 + int(rand(15));
+ $runtime = 600;
+ }
+ my @cmd = ('pgbench', '-p', $port, '-T', $runtime, '-c', $clients);
+
+ # Randomize whether we spawn connections or not
+ push(@cmd, '-C') if ($extended && cointoss);
+ # Finally add the database name to use
+ push(@cmd, 'postgres');
+
+ $pgbench = IPC::Run::start(
+ \@cmd,
+ '<' => '/dev/null',
+ '>' => '/dev/null',
+ '2>' => '/dev/null',
+ IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
+}
+
+sub attach_injection_point
+{
+ my ($node, $point) = @_;
+ note('attaching injection point: ' . $point);
+ $node->safe_psql('postgres',
+ "SELECT injection_points_attach('" . $point . "','wait');"
+ );
+}
+
+sub wait_injection_point
+{
+ my ($node, $point) = @_;
+
+ note("waiting for the injection point to be hit");
+ $node->poll_query_until(
+ 'postgres',
+ "SELECT COUNT(*) FROM pg_catalog.pg_stat_activity WHERE wait_event = '" . $point . "'",
+ '1');
+}
+
+sub wakeup_injection_point
+{
+ my ($node, $point) = @_;
+
+ note('waking the init injection point: ' . $point);
+ $node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('" . $point . "');"
+ );
+
+ note("detaching the init injection point");
+ $node->safe_psql('postgres',
+ "SELECT injection_points_detach('" . $point . "');");
+}
+
+# Test behavior with a checksum transitions and a concurrent checkpoint.
+#
+# The cluster is initialized into checksum state $start (enabled/disabled),
+# and a checksum change is initiated in a controled way. It gets paused at
+# a selected injection point $point, and a checkpoint is performed. Then
+# the injection point is waken up. Once the checksum change completes, the
+# checksum state is compared to the expected $final state.
+#
+# The checkpoint is performed synchronously, while the checksum worker
+# is paused on the injection point.
+#
+# Finally, after the checksum change completes, the instance is restarted
+# (in either fast ot immediate mode, picked randomly), and the final
+# checksum state is validated against the expected value. The server log
+# is checked for checksum failures.
+#
+# While the checksum change is happening, there's a r/w pgbench running in
+# the background, to generate writes.
+#
+# arguments:
+#
+# - start - starting checksum state (enabled/disabled)
+# - change - checksum change
+# - init - initial injection point (see comment later)
+# - point - injection point the first change should wait on
+# - final - expected checksum state at the end
+#
+# XXX Some of the injection points are in a critical section, which does
+# not allow memory allocations etc. INJECTION_POINT_LOAD/_CACHED handles
+# just private memory allocation, but 'wait' requires a shmem allocation.
+# To deal with that, we setup an injection point $init outside a critical
+# section to initialize the shmem stuff needed by 'wait'.
+#
+# XXX We could also validate the checksums using pg_checksums, if the
+# state is 'on'.
+sub test_checksum_transition
+{
+ my ($start, $change, $init, $point, $final) = @_;
+
+ # print the current test instructions, both into TAP output and into
+ # the server log, to make correlation easier
+
+ my $initstr = 'undef';
+ $initstr = $init if defined($init);
+
+ # Start the test suite with pgbench running.
+ background_rw_pgbench($node->port);
+
+ $node->safe_psql('postgres',
+ "SELECT '========== " . $start . " / " . $change . " / " . $initstr . " / " . $point . " / " . $final . " =========='");
+
+ note($start . " / " . $change . " / " . $initstr . " / " . $point . " / " . $final);
+
+ # put the cluster into the initial checksum state, synchronously
+ note('changing checksums into initial state: ' . $start);
+ enable_data_checksums($node, wait => 'on') if ($start eq 'enabled');
+ disable_data_checksums($node, wait => 'off') if ($start eq 'disabled');
+
+ # Wait on an injection point outside a critical section, to initialize
+ # the shmem (which can't be done in critical section).
+ attach_injection_point($node, $init) if defined($init);
+
+ # Wait on the injection point, to pause the checksum change.
+ attach_injection_point($node, $point);
+
+ # Trigger the checksum change, asynchronously
+ note("triggering checksum change: " . $change);
+ enable_data_checksums($node) if ($change eq 'enable');
+ disable_data_checksums($node) if ($change eq 'disable');
+
+ # Handle the initial injection point - wait, wakeup and detatch. This
+ # initializes the shmem for the 'wait' action.
+ wait_injection_point($node, $init) if defined($init);
+ wakeup_injection_point($node, $init) if defined($init);
+
+ # Wait for the injection point to be hit by the state change.
+ wait_injection_point($node, $point);
+
+ # The checksum state change is paused on the injection point. Perform
+ # the checkpoint (synchronously).
+ note('checkpoint');
+ $node->safe_psql('postgres', "CHECKPOINT");
+
+ # Wake the injection point, so that the first change can proceed.
+ wakeup_injection_point($node, $point);
+
+ # Wait until there are no ongoing checksum changes, which we determine
+ # by looking for a checksum launcher process.
+ note('wait for the checksum launcher to exit');
+ $node->poll_query_until('postgres',
+ "SELECT count(*) = 0 "
+ . "FROM pg_catalog.pg_stat_activity "
+ . "WHERE backend_type = 'datachecksum launcher';");
+
+ # Does the final checksum state match the expected state?
+ test_checksum_state($node, $final);
+
+ # restart the cluster, possibly in immediate mode, to simulate a crash
+ #
+ # XXX maybe we should do just immediate shutdowns, because the shutdown
+ # checkpoint will overwrite the state state with "current" one, not the
+ # state produced by the concurrent change + checkpoint.
+ $node->stop(stopmode());
+ $node->start;
+
+ # Does the final checksum state match the expected state?
+ test_checksum_state($node, $final);
+
+ # Since the log isn't being written to now, parse the log and check
+ # for instances of checksum verification failures.
+ my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile,
+ $node_loglocation);
+ unlike(
+ $log,
+ qr/page verification failed,.+\d$/,
+ "no checksum validation errors in primary log (during WAL recovery)"
+ );
+ $node_loglocation = -s $node->logfile;
+}
+
+# concurrent enable + checkpoint, different injection points in the "enable" process
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-inprogress-checksums-start', 'on');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-xlog', 'on');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-xlogctl', 'on');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-controlfile', 'on');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-inprogress-checksums-before-barrier-wait', 'on');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-inprogress-checksums-end', 'on');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-checksums-start', 'on');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-xlog', 'on');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-xlogctl', 'on');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-controlfile', 'on');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-checksums-before-checkpoint', 'on');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-checksums-before-barrier-wait', 'on');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-checksums-end', 'on');
+
+# concurrent disable + checkpoint, different injection points in the "disable" process
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-inprogress-checksums-start', 'off');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-xlog', 'off');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-xlogctl', 'off');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-controlfile', 'off');
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-inprogress-checksums-before-barrier-wait', 'off');
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-checksums-start', 'off');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-xlog', 'off');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-xlogctl', 'off');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-controlfile', 'off');
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-checksums-before-checkpoint', 'off');
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-checksums-before-barrier-wait', 'off');
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-checksums-end', 'off');
+
+$node->stop;
+done_testing();
--
2.54.0
[text/x-patch] 0007-TAP-12-crashes-at-injection-points.patch (13.4K, ../[email protected]/8-0007-TAP-12-crashes-at-injection-points.patch)
download | inline diff:
From 57a5e353dbfcaaac1fa7cd25dc38f3e1807ccafa Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Apr 2026 23:06:14 +0200
Subject: [PATCH 07/13] TAP 12: crashes at injection points
---
.../t/012_concurrent_checkpoint_crash.pl | 282 ++++++++++++++++++
1 file changed, 282 insertions(+)
create mode 100644 src/test/modules/test_checksums/t/012_concurrent_checkpoint_crash.pl
diff --git a/src/test/modules/test_checksums/t/012_concurrent_checkpoint_crash.pl b/src/test/modules/test_checksums/t/012_concurrent_checkpoint_crash.pl
new file mode 100644
index 00000000000..de169f47034
--- /dev/null
+++ b/src/test/modules/test_checksums/t/012_concurrent_checkpoint_crash.pl
@@ -0,0 +1,282 @@
+
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test suite for testing enabling data checksums in an online cluster with
+# injection point tests injecting failures into the processing
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use DataChecksums::Utils;
+
+# This test suite is expensive, or very expensive, to execute. There are two
+# PG_TEST_EXTRA options for running it, "checksum" for a pared-down test suite
+# an "checksum_extended" for the full suite. The full suite can run for hours
+# on slow or constrained systems.
+my $extended = undef;
+if ($ENV{PG_TEST_EXTRA})
+{
+ $extended = 1 if ($ENV{PG_TEST_EXTRA} =~ /\bchecksum_extended\b/);
+ plan skip_all => 'Expensive data checksums test disabled'
+ unless ($ENV{PG_TEST_EXTRA} =~ /\bchecksum(_extended)?\b/);
+}
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# ---------------------------------------------------------------------------
+# Test cluster setup
+#
+
+# Initiate testcluster
+my $node = PostgreSQL::Test::Cluster->new('injection_node');
+$node->init(no_data_checksums => 1);
+$node->start;
+
+# Set up test environment
+$node->safe_psql('postgres', 'CREATE EXTENSION test_checksums;');
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+my $pgbench = undef;
+my $scalefactor = ($extended ? 10 : 1);
+my $node_loglocation = 0;
+
+$node->command_ok(
+ [
+ 'pgbench', '-p', $node->port, '-i',
+ '-s', $scalefactor, '-q', 'postgres'
+ ]);
+
+# Start a pgbench run in the background against the server specified via the
+# port passed as parameter.
+sub background_rw_pgbench
+{
+ my $port = shift;
+
+ # If a previous pgbench is still running, start by shutting it down.
+ $pgbench->finish if $pgbench;
+
+ my $clients = 1;
+ my $runtime = 2;
+
+ if ($extended)
+ {
+ # Randomize the number of pgbench clients a bit (range 1-16)
+ $clients = 1 + int(rand(15));
+ $runtime = 600;
+ }
+ my @cmd = ('pgbench', '-p', $port, '-T', $runtime, '-c', $clients);
+
+ # Randomize whether we spawn connections or not
+ push(@cmd, '-C') if ($extended && cointoss);
+ # Finally add the database name to use
+ push(@cmd, 'postgres');
+
+ $pgbench = IPC::Run::start(
+ \@cmd,
+ '<' => '/dev/null',
+ '>' => '/dev/null',
+ '2>' => '/dev/null',
+ IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
+}
+
+sub attach_injection_point
+{
+ my ($node, $point) = @_;
+ note('attaching injection point: ' . $point);
+ $node->safe_psql('postgres',
+ "SELECT injection_points_attach('" . $point . "','wait');"
+ );
+}
+
+sub wait_injection_point
+{
+ my ($node, $point) = @_;
+
+ note("waiting for the injection point to be hit");
+ $node->poll_query_until(
+ 'postgres',
+ "SELECT COUNT(*) FROM pg_catalog.pg_stat_activity WHERE wait_event = '" . $point . "'",
+ '1');
+}
+
+sub wakeup_injection_point
+{
+ my ($node, $point) = @_;
+
+ note('waking the init injection point: ' . $point);
+ $node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('" . $point . "');"
+ );
+
+ note("detaching the init injection point");
+ $node->safe_psql('postgres',
+ "SELECT injection_points_detach('" . $point . "');");
+}
+
+# Test behavior with a checksum transitions and a concurrent checkpoint,
+# followed by a crash.
+#
+# The test puts the instance into the initial checksum state $start,
+# triggers a checksum change that pauses on a first injection point. Then
+# a checkpoint is performed, and the checksum change proceeds either to
+# a second injection point or finishes.
+#
+# Then the instance get restarted in immediate mode to simulate failure,
+# and the final checksum state (after recovery) is validated against the
+# expected value. The server log is checked for checksum failures.
+#
+# While the checksum change is happening, there's a r/w pgbench running in
+# the background, to generate writes.
+#
+# arguments:
+#
+# - start - start checksum state (enabled/disabled)
+# - change - checksum change to initiate
+# - point1 - injection point before checkpoint
+# - point2 - injection point after checkpoint
+# - final - expected checksum state at the end
+#
+# XXX Some of the injection points are in a critical section, which does
+# not allow memory allocations etc. INJECTION_POINT_LOAD/_CACHED handles
+# just private memory allocation, but 'wait' requires a shmem allocation.
+# To deal with that, we setup an injection point $init outside a critical
+# section to initialize the shmem stuff needed by 'wait'.
+#
+# XXX We could also validate the checksums using pg_checksums, if the
+# state is 'on'.
+sub test_checksum_transition
+{
+ my ($start, $change, $init, $point1, $point2, $final) = @_;
+
+ my $initstr = 'undef';
+ my $point2str = 'undef';
+
+ # print the current test instructions, both into TAP output and into
+ # the server log, to make correlation easier
+
+ $initstr = $init if defined($init);
+ $point2str = $point2 if defined($point2);
+
+ $node->safe_psql('postgres',
+ "SELECT '========== " . $start . " / " . $change . " / " . $initstr . " / " . $point1 . " / " . $point2str . " / " . $final . " =========='");
+
+ note($start . " / " . $change . " / " . $initstr . " / " . $point1 . " / " . $point2str . " / " . $final);
+
+ # Start the test suite with pgbench running.
+ background_rw_pgbench($node->port);
+
+ # put the cluster into the initial checksum state, synchronously
+ note('changing checksums into initial state: ' . $start);
+ enable_data_checksums($node, wait => 'on') if ($start eq 'enabled');
+ disable_data_checksums($node, wait => 'off') if ($start eq 'disabled');
+
+ # Wait on an injection point outside a critical section, to initialize
+ # the shmem (which can't be done in critical section).
+ attach_injection_point($node, $init) if defined($init);
+
+ # Wait on the two injection points, to pause the checksum change.
+ attach_injection_point($node, $point1);
+
+ # The second injection point is optional, so only attach it then.
+ attach_injection_point($node, $point2) if defined($point2);
+
+ # Trigger the checksum change, asynchronously
+ note("triggering checksum change: " . $change);
+ enable_data_checksums($node) if ($change eq 'enable');
+ disable_data_checksums($node) if ($change eq 'disable');
+
+ # Handle the initial injection point - wait, wakeup and detatch. This
+ # initializes the shmem for the 'wait' action.
+ wait_injection_point($node, $init) if defined($init);
+ wakeup_injection_point($node, $init) if defined($init);
+
+ # Wait for the first injection point to be hit by the state change.
+ wait_injection_point($node, $point1);
+
+ # The checksum state change is paused on the first injection point.
+ # Perform the checkpoint (synchronously).
+ note('checkpoint');
+ $node->safe_psql('postgres', "CHECKPOINT");
+
+ # Wake the injection point, so that the first change can proceed.
+ wakeup_injection_point($node, $point1);
+
+ # Either wait for the second injection point - if defined, or for the
+ # checksum change to complete.
+
+ if (defined($point2))
+ {
+ wait_injection_point($node, $point2);
+ }
+ else
+ {
+ # Wait until there are no ongoing checksum changes, which we determine
+ # by looking for a checksum launcher process.
+ note('wait for the checksum launcher to exit');
+ $node->poll_query_until('postgres',
+ "SELECT count(*) = 0 "
+ . "FROM pg_catalog.pg_stat_activity "
+ . "WHERE backend_type = 'datachecksum launcher';");
+ note('checksum launcher exited');
+ }
+
+ # restart the cluster, in immediate mode, to simulate a crash
+ $node->stop('immediate');
+ $node->start;
+
+ # Does the final checksum state match the expected state?
+ test_checksum_state($node, $final);
+
+ # Since the log isn't being written to now, parse the log and check
+ # for instances of checksum verification failures.
+ my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile,
+ $node_loglocation);
+ unlike(
+ $log,
+ qr/page verification failed,.+\d$/,
+ "no checksum validation errors in primary log (during WAL recovery)"
+ );
+ $node_loglocation = -s $node->logfile;
+}
+
+# concurrent enable + checkpoint, different injection points in the "enable" process
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-xlog', 'off');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-xlog', 'datachecksums-enable-inprogress-checksums-after-xlogctl', 'off');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-xlogctl', 'datachecksums-enable-inprogress-checksums-after-controlfile', 'off');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-after-controlfile', 'datachecksums-enable-inprogress-checksums-before-barrier-wait', 'off');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-inprogress-checksums-before-barrier-wait', 'datachecksums-enable-inprogress-checksums-end', 'off');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-inprogress-checksums-end', 'datachecksums-enable-checksums-start', 'off');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-xlog', 'on');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-xlog', 'datachecksums-enable-checksums-after-xlogctl', 'off');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-xlogctl', 'datachecksums-enable-checksums-after-controlfile', 'on');
+test_checksum_transition('disabled', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-after-controlfile', 'datachecksums-enable-checksums-before-checkpoint', 'on');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-checksums-before-checkpoint', 'datachecksums-enable-checksums-before-barrier-wait', 'on');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-checksums-before-barrier-wait', 'datachecksums-enable-checksums-end', 'on');
+test_checksum_transition('disabled', 'enable', undef, 'datachecksums-enable-checksums-end', undef, 'on');
+
+# concurrent disable + checkpoint, different injection points in the "disable" process
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-xlog', 'off');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-xlog', 'datachecksums-disable-inprogress-checksums-after-xlogctl', 'on');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-xlogctl', 'datachecksums-disable-inprogress-checksums-after-controlfile', 'off');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-inprogress-checksums-after-controlfile', 'datachecksums-disable-inprogress-checksums-before-barrier-wait', 'off');
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-inprogress-checksums-before-barrier-wait', 'datachecksums-disable-checksums-start', 'off');
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-xlog', 'off');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-xlog', 'datachecksums-disable-checksums-after-xlogctl', 'off');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-xlogctl', 'datachecksums-disable-checksums-after-controlfile', 'off');
+test_checksum_transition('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-after-controlfile', 'datachecksums-disable-checksums-before-checkpoint', 'off');
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-checksums-before-checkpoint', 'datachecksums-disable-checksums-before-barrier-wait', 'off');
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-checksums-before-barrier-wait', 'datachecksums-disable-checksums-end', 'off');
+test_checksum_transition('enabled', 'disable', undef, 'datachecksums-disable-checksums-end', undef, 'off');
+
+$node->stop;
+done_testing();
--
2.54.0
[text/x-patch] 0008-TAP-13-concurrency-with-checkpoint-REDO.patch (23.8K, ../[email protected]/9-0008-TAP-13-concurrency-with-checkpoint-REDO.patch)
download | inline diff:
From ddaca81f530a371a2dfab67ad9da49ef1f8aa69e Mon Sep 17 00:00:00 2001
From: test <test>
Date: Fri, 1 May 2026 22:16:00 +0200
Subject: [PATCH 08/13] TAP 13: concurrency with checkpoint REDO
---
.../t/013_async_checkpoint_crash.pl | 604 ++++++++++++++++++
1 file changed, 604 insertions(+)
create mode 100644 src/test/modules/test_checksums/t/013_async_checkpoint_crash.pl
diff --git a/src/test/modules/test_checksums/t/013_async_checkpoint_crash.pl b/src/test/modules/test_checksums/t/013_async_checkpoint_crash.pl
new file mode 100644
index 00000000000..4a310506482
--- /dev/null
+++ b/src/test/modules/test_checksums/t/013_async_checkpoint_crash.pl
@@ -0,0 +1,604 @@
+
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test suite for testing enabling data checksums in an online cluster with
+# injection point tests injecting failures into the processing
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use DataChecksums::Utils;
+
+# This test suite is expensive, or very expensive, to execute. There are two
+# PG_TEST_EXTRA options for running it, "checksum" for a pared-down test suite
+# an "checksum_extended" for the full suite. The full suite can run for hours
+# on slow or constrained systems.
+my $extended = undef;
+if ($ENV{PG_TEST_EXTRA})
+{
+ $extended = 1 if ($ENV{PG_TEST_EXTRA} =~ /\bchecksum_extended\b/);
+ plan skip_all => 'Expensive data checksums test disabled'
+ unless ($ENV{PG_TEST_EXTRA} =~ /\bchecksum(_extended)?\b/);
+}
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# ---------------------------------------------------------------------------
+# Test cluster setup
+#
+
+# Initiate testcluster
+my $node = PostgreSQL::Test::Cluster->new('injection_node');
+$node->init(no_data_checksums => 1);
+
+# make sure we don't hit checkpoints unless desired
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+checkpoint_timeout = 1h
+max_wal_size = 32GB
+]);
+$node->start;
+
+# Set up test environment
+$node->safe_psql('postgres', 'CREATE EXTENSION test_checksums;');
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+my $pgbench = undef;
+my $scalefactor = ($extended ? 10 : 1);
+my $node_loglocation = 0;
+
+$node->command_ok(
+ [
+ 'pgbench', '-p', $node->port, '-i',
+ '-s', $scalefactor, '-q', 'postgres'
+ ]);
+
+# Start a pgbench run in the background against the server specified via the
+# port passed as parameter.
+sub background_rw_pgbench
+{
+ my $port = shift;
+
+ # If a previous pgbench is still running, start by shutting it down.
+ $pgbench->finish if $pgbench;
+
+ my $clients = 1;
+ my $runtime = 2;
+
+ if ($extended)
+ {
+ # Randomize the number of pgbench clients a bit (range 1-16)
+ $clients = 1 + int(rand(15));
+ $runtime = 600;
+ }
+ my @cmd = ('pgbench', '-p', $port, '-T', $runtime, '-c', $clients);
+
+ # Randomize whether we spawn connections or not
+ push(@cmd, '-C') if ($extended && cointoss);
+ # Finally add the database name to use
+ push(@cmd, 'postgres');
+
+ $pgbench = IPC::Run::start(
+ \@cmd,
+ '<' => '/dev/null',
+ '>' => '/dev/null',
+ '2>' => '/dev/null',
+ IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
+}
+
+# Start a checkpoint in the background against the server specified via the
+# port passed as parameter.
+sub background_checkpoint
+{
+ my $port = shift;
+
+ my @cmd = ('psql', '-p', $port, '-c', 'checkpoint', 'postgres');
+
+ $pgbench = IPC::Run::start(
+ \@cmd,
+ '<' => '/dev/null',
+ '>' => '/dev/null',
+ '2>' => '/dev/null',
+ IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
+}
+
+sub attach_injection_point
+{
+ my ($node, $point) = @_;
+ note('attaching injection point: ' . $point);
+ $node->safe_psql('postgres',
+ "SELECT injection_points_attach('" . $point . "','wait');"
+ );
+}
+
+sub wait_injection_point
+{
+ my ($node, $point) = @_;
+
+ note("waiting for the injection point to be hit");
+ $node->poll_query_until(
+ 'postgres',
+ "SELECT COUNT(*) FROM pg_catalog.pg_stat_activity WHERE wait_event = '" . $point . "'",
+ '1');
+}
+
+sub wakeup_injection_point
+{
+ my ($node, $point) = @_;
+
+ # detach before wakeup, so that we can't hit it right away
+ note("detaching the init injection point");
+ $node->safe_psql('postgres',
+ "SELECT injection_points_detach('" . $point . "');");
+
+ note('waking the init injection point: ' . $point);
+ $node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('" . $point . "');"
+ );
+}
+
+# Test behavior with a checksum transitions and a concurrent checkpoint,
+# followed by a crash.
+#
+# The test puts the instance into the initial checksum state $start,
+# initiates a checksum change and waits for it to pause on the $init
+# injection point. Then a checkpoint is started (in the background), and
+# the two processes go through a list of injection points to interleave
+# them in a particular way.
+#
+# After reaching a $stop injection point, the cluster get restarted in an
+# immediate mode to simulate failure. The checksum state (after recovery)
+# is validated against the expected value. The server log is checked for
+# checksum failures.
+#
+# While the checksum change is happening, there's a r/w pgbench running in
+# the background, to generate writes.
+#
+# arguments:
+#
+# - start - start checksum state (enabled/disabled)
+# - change - checksum change to initiate
+# - init - initial injection point to wait on
+# - stop - injection point to wait on before restarting
+# - final - expected checksum state at the end
+# - steps - a list of injection points to wait on / unpause
+#
+# XXX Some of the injection points are in a critical section, which does
+# not allow memory allocations etc. INJECTION_POINT_LOAD/_CACHED handles
+# just private memory allocation, but 'wait' requires a shmem allocation.
+# To deal with that, we first need to pause on an injection point outside
+# a critical section to initialize the shmem stuff needed by 'wait'.
+#
+# XXX We could also validate the checksums using pg_checksums, if the
+# state is 'on'.
+#
+# XXX The steps are not entirely deterministic, because when unpausing
+# an injection point, we don't know if the other process is already
+# paused on the following point.
+#
+sub test_checksum_sequence
+{
+ my ($start, $change, $init, $stop, $final, @steps) = @_;
+
+ # print the current test instructions, both into TAP output and into
+ # the server log, to make correlation easier
+
+ $node->safe_psql('postgres',
+ "SELECT '========== " . $start . " / " . $change . " / (" . @steps . ") / " . $init . " / " . $stop . " / " . $final . " =========='");
+
+ note($start . " / " . $change . " / (" . @steps . ") / " . $init . " / " . $stop . " / " . $final);
+
+ # Start the test suite with pgbench running.
+ background_rw_pgbench($node->port);
+
+ # put the cluster into the initial checksum state, synchronously
+ note('changing checksums into initial state: ' . $start);
+ enable_data_checksums($node, wait => 'on') if ($start eq 'enabled');
+ disable_data_checksums($node, wait => 'off') if ($start eq 'disabled');
+
+ # attach the initial injection point
+ attach_injection_point($node, $init);
+
+ # when we're at it, attach the stop injection point
+ attach_injection_point($node, $stop);
+
+ # Do a checkpoint first, so that we don't accidentaly start the next
+ # checkpoint at an unecpected time. We want the next checkpoint to be
+ # the background one.
+ $node->safe_psql('postgres', "CHECKPOINT");
+
+ # Trigger the checksum change, asynchronously
+ note("triggering checksum change: " . $change);
+ enable_data_checksums($node) if ($change eq 'enable');
+ disable_data_checksums($node) if ($change eq 'disable');
+
+ # wait for the initial injection point
+ wait_injection_point($node, $init);
+
+ # now attach all the regular injection points
+ my $n = @steps;
+ for ($a = 0; $a < $n; $a++)
+ {
+ my $point = $steps[$a];
+ attach_injection_point($node, $point);
+ }
+
+ # initiate a background checkpoint
+ background_checkpoint($node->port);
+
+ # wakeup the initial injection point, to start the main part
+ wakeup_injection_point($node, $init);
+
+ # wait for regular injection points in sequence, wake and detach them
+ for ($a = 0; $a < $n; $a++)
+ {
+ my $point = $steps[$a];
+
+ wait_injection_point($node, $point);
+ wakeup_injection_point($node, $point);
+ }
+
+ # wait for the stop injection point
+ wait_injection_point($node, $stop);
+
+ # restart the cluster, in immediate mode, to simulate a crash
+ $node->stop('immediate');
+ $node->start;
+
+ # Does the final checksum state match the expected state?
+ test_checksum_state($node, $final);
+
+ # Since the log isn't being written to now, parse the log and check
+ # for instances of checksum verification failures.
+ my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile,
+ $node_loglocation);
+ unlike(
+ $log,
+ qr/page verification failed,.+\d$/,
+ "no checksum validation errors in primary log (during WAL recovery)"
+ );
+ $node_loglocation = -s $node->logfile;
+}
+
+my @steps = undef;
+
+## checksums INPROGRESS ON
+
+note('TEST INPROGRESS-ON/1');
+@steps = qw(datachecksums-enable-inprogress-checksums-before-xlog
+ datachecksums-enable-inprogress-checksums-after-xlog
+ datachecksums-enable-inprogress-checksums-after-xlogctl
+ datachecksums-enable-inprogress-checksums-after-controlfile
+ create-checkpoint-initial
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal);
+test_checksum_sequence('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-end', 'off', @steps);
+
+note('TEST INPROGRESS-ON/2');
+@steps = qw(create-checkpoint-initial
+ datachecksums-enable-inprogress-checksums-before-xlog
+ datachecksums-enable-inprogress-checksums-after-xlog
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ datachecksums-enable-inprogress-checksums-after-xlogctl
+ datachecksums-enable-inprogress-checksums-after-controlfile
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal);
+test_checksum_sequence('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-end', 'off', @steps);
+
+note('TEST INPROGRESS-ON/3');
+@steps = qw(create-checkpoint-initial
+ datachecksums-enable-inprogress-checksums-before-xlog
+ datachecksums-enable-inprogress-checksums-after-xlog
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ datachecksums-enable-inprogress-checksums-after-xlogctl
+ datachecksums-enable-inprogress-checksums-after-controlfile
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal);
+test_checksum_sequence('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-end', 'off', @steps);
+
+note('TEST INPROGRESS-ON/4');
+@steps = qw(create-checkpoint-initial
+ datachecksums-enable-inprogress-checksums-before-xlog
+ datachecksums-enable-inprogress-checksums-after-xlog
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ datachecksums-enable-inprogress-checksums-after-xlogctl
+ datachecksums-enable-inprogress-checksums-after-controlfile
+ checkpoint-before-old-wal-removal);
+test_checksum_sequence('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-end', 'off', @steps);
+
+note('TEST INPROGRESS-ON/5');
+@steps = qw(create-checkpoint-initial
+ datachecksums-enable-inprogress-checksums-before-xlog
+ datachecksums-enable-inprogress-checksums-after-xlog
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal
+ datachecksums-enable-inprogress-checksums-after-xlogctl
+ datachecksums-enable-inprogress-checksums-after-controlfile);
+test_checksum_sequence('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-end', 'off', @steps);
+
+note('TEST INPROGRESS-ON/6');
+@steps = qw(create-checkpoint-initial
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ datachecksums-enable-inprogress-checksums-before-xlog
+ datachecksums-enable-inprogress-checksums-after-xlog
+ datachecksums-enable-inprogress-checksums-after-xlogctl
+ datachecksums-enable-inprogress-checksums-after-controlfile
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal);
+test_checksum_sequence('disabled', 'enable', 'datachecksums-enable-inprogress-checksums-start', 'datachecksums-enable-inprogress-checksums-end', 'off', @steps);
+
+## checksums ON
+
+note('TEST ON/1');
+@steps = qw(datachecksums-enable-checksums-before-xlog
+ datachecksums-enable-checksums-after-xlog
+ datachecksums-enable-checksums-after-xlogctl
+ datachecksums-enable-checksums-after-controlfile
+ create-checkpoint-initial
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal);
+test_checksum_sequence('disabled', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-end', 'on', @steps);
+
+note('TEST ON/2');
+@steps = qw(create-checkpoint-initial
+ datachecksums-enable-checksums-before-xlog
+ datachecksums-enable-checksums-after-xlog
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal
+ datachecksums-enable-checksums-after-xlogctl
+ datachecksums-enable-checksums-after-controlfile);
+test_checksum_sequence('disabled', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-end', 'on', @steps);
+
+note('TEST ON/3');
+@steps = qw(create-checkpoint-initial
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ datachecksums-enable-checksums-before-xlog
+ datachecksums-enable-checksums-after-xlog
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal
+ datachecksums-enable-checksums-after-xlogctl
+ datachecksums-enable-checksums-after-controlfile);
+test_checksum_sequence('disabled', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-end', 'on', @steps);
+
+note('TEST ON/4');
+@steps = qw(create-checkpoint-initial
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ datachecksums-enable-checksums-before-xlog
+ datachecksums-enable-checksums-after-xlog
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal
+ datachecksums-enable-checksums-after-xlogctl
+ datachecksums-enable-checksums-after-controlfile);
+test_checksum_sequence('disabled', 'enable', 'datachecksums-enable-checksums-start', 'datachecksums-enable-checksums-end', 'on', @steps);
+
+
+## checksums INPROGRESS OFF
+
+note('TEST INPROGRESS-OFF/1');
+@steps = qw(datachecksums-disable-inprogress-checksums-before-xlog
+ datachecksums-disable-inprogress-checksums-after-xlog
+ datachecksums-disable-inprogress-checksums-after-xlogctl
+ datachecksums-disable-inprogress-checksums-after-controlfile
+ datachecksums-disable-inprogress-checksums-before-checkpoint
+ create-checkpoint-initial
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal
+ datachecksums-disable-inprogress-checksums-before-barrier-wait);
+test_checksum_sequence('enabled', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-checksums-start', 'off', @steps);
+
+note('TEST INPROGRESS-OFF/2');
+@steps = qw(create-checkpoint-initial
+ datachecksums-disable-inprogress-checksums-before-xlog
+ datachecksums-disable-inprogress-checksums-after-xlog
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal
+ datachecksums-disable-inprogress-checksums-after-xlogctl
+ datachecksums-disable-inprogress-checksums-after-controlfile
+ datachecksums-disable-inprogress-checksums-before-checkpoint
+ datachecksums-disable-inprogress-checksums-before-barrier-wait);
+test_checksum_sequence('enabled', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-checksums-start', 'off', @steps);
+
+note('TEST INPROGRESS-OFF/3');
+@steps = qw(create-checkpoint-initial
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ datachecksums-disable-inprogress-checksums-before-xlog
+ datachecksums-disable-inprogress-checksums-after-xlog
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal
+ datachecksums-disable-inprogress-checksums-after-xlogctl
+ datachecksums-disable-inprogress-checksums-after-controlfile
+ datachecksums-disable-inprogress-checksums-before-checkpoint
+ datachecksums-disable-inprogress-checksums-before-barrier-wait);
+test_checksum_sequence('enabled', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-checksums-start', 'off', @steps);
+
+note('TEST INPROGRESS-OFF/4');
+@steps = qw(create-checkpoint-initial
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ datachecksums-disable-inprogress-checksums-before-xlog
+ datachecksums-disable-inprogress-checksums-after-xlog
+ datachecksums-disable-inprogress-checksums-after-xlogctl
+ datachecksums-disable-inprogress-checksums-after-controlfile
+ datachecksums-disable-inprogress-checksums-before-checkpoint
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal
+ datachecksums-disable-inprogress-checksums-before-barrier-wait);
+test_checksum_sequence('enabled', 'disable', 'datachecksums-disable-inprogress-checksums-start', 'datachecksums-disable-checksums-start', 'off', @steps);
+
+## checksums OFF
+
+note('TEST OFF/1');
+@steps = qw(datachecksums-disable-checksums-before-xlog
+ datachecksums-disable-checksums-after-xlog
+ datachecksums-disable-checksums-after-xlogctl
+ datachecksums-disable-checksums-after-controlfile
+ datachecksums-disable-checksums-before-checkpoint
+ create-checkpoint-initial
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal
+ datachecksums-disable-checksums-before-barrier-wait);
+test_checksum_sequence('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-end', 'off', @steps);
+
+note('TEST OFF/2');
+@steps = qw(create-checkpoint-initial
+ datachecksums-disable-checksums-before-xlog
+ datachecksums-disable-checksums-after-xlog
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ datachecksums-disable-checksums-after-xlogctl
+ datachecksums-disable-checksums-after-controlfile
+ datachecksums-disable-checksums-before-checkpoint
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ checkpoint-before-old-wal-removal
+ datachecksums-disable-checksums-before-barrier-wait);
+test_checksum_sequence('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-end', 'off', @steps);
+
+note('TEST OFF/3');
+@steps = qw(create-checkpoint-initial
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ datachecksums-disable-checksums-before-xlog
+ datachecksums-disable-checksums-after-xlog
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ datachecksums-disable-checksums-after-xlogctl
+ datachecksums-disable-checksums-after-controlfile
+ datachecksums-disable-checksums-before-checkpoint
+ checkpoint-before-old-wal-removal
+ datachecksums-disable-checksums-before-barrier-wait);
+test_checksum_sequence('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-end', 'off', @steps);
+
+note('TEST OFF/4');
+@steps = qw(create-checkpoint-initial
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ datachecksums-disable-checksums-before-xlog
+ datachecksums-disable-checksums-after-xlog
+ datachecksums-disable-checksums-after-xlogctl
+ datachecksums-disable-checksums-after-controlfile
+ datachecksums-disable-checksums-before-checkpoint
+ checkpoint-before-old-wal-removal
+ datachecksums-disable-checksums-before-barrier-wait);
+test_checksum_sequence('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-end', 'off', @steps);
+
+note('TEST OFF/5');
+@steps = qw(create-checkpoint-initial
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ checkpoint-before-redo-checksums
+ datachecksums-disable-checksums-before-xlog
+ datachecksums-disable-checksums-after-xlog
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ datachecksums-disable-checksums-after-xlogctl
+ datachecksums-disable-checksums-after-controlfile
+ datachecksums-disable-checksums-before-checkpoint
+ checkpoint-before-old-wal-removal
+ datachecksums-disable-checksums-before-barrier-wait);
+test_checksum_sequence('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-end', 'off', @steps);
+
+note('TEST OFF/6');
+@steps = qw(create-checkpoint-initial
+ datachecksums-disable-checksums-before-xlog
+ datachecksums-disable-checksums-after-xlog
+ checkpoint-before-redo
+ checkpoint-before-xlogctl-checksums
+ checkpoint-after-xlogctl-checksums
+ datachecksums-disable-checksums-after-xlogctl
+ checkpoint-before-redo-checksums
+ checkpoint-before-redo-wal
+ checkpoint-after-redo-wal
+ datachecksums-disable-checksums-after-controlfile
+ datachecksums-disable-checksums-before-checkpoint
+ checkpoint-before-old-wal-removal
+ datachecksums-disable-checksums-before-barrier-wait);
+test_checksum_sequence('enabled', 'disable', 'datachecksums-disable-checksums-start', 'datachecksums-disable-checksums-end', 'off', @steps);
+
+$node->stop;
+done_testing();
--
2.54.0
[text/x-patch] 0009-TAP-14-checkpoints-with-crashes.patch (12.6K, ../[email protected]/10-0009-TAP-14-checkpoints-with-crashes.patch)
download | inline diff:
From ff3cd3e661d508db102e18752fee3335fb4ceaca Mon Sep 17 00:00:00 2001
From: test <test>
Date: Fri, 1 May 2026 13:35:56 +0200
Subject: [PATCH 09/13] TAP 14: checkpoints with crashes
This shows the non-determinism / race condition with the step in test
012_crashes, thanks to better sequencing of the two sides (checksum
worker + checkpointer). The related 012 test is this:
test_checksum_transition('disabled', 'enable',
'datachecksums-enable-checksums-start',
'datachecksums-enable-checksums-after-xlog',
'datachecksums-enable-checksums-after-xlogctl',
'off');
---
...ncurrent_checkpoint_crash_deterministic.pl | 359 ++++++++++++++++++
1 file changed, 359 insertions(+)
create mode 100644 src/test/modules/test_checksums/t/014_concurrent_checkpoint_crash_deterministic.pl
diff --git a/src/test/modules/test_checksums/t/014_concurrent_checkpoint_crash_deterministic.pl b/src/test/modules/test_checksums/t/014_concurrent_checkpoint_crash_deterministic.pl
new file mode 100644
index 00000000000..bfca3983965
--- /dev/null
+++ b/src/test/modules/test_checksums/t/014_concurrent_checkpoint_crash_deterministic.pl
@@ -0,0 +1,359 @@
+
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test suite for testing enabling data checksums in an online cluster with
+# injection point tests injecting failures into the processing
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use DataChecksums::Utils;
+
+# This test suite is expensive, or very expensive, to execute. There are two
+# PG_TEST_EXTRA options for running it, "checksum" for a pared-down test suite
+# an "checksum_extended" for the full suite. The full suite can run for hours
+# on slow or constrained systems.
+my $extended = undef;
+if ($ENV{PG_TEST_EXTRA})
+{
+ $extended = 1 if ($ENV{PG_TEST_EXTRA} =~ /\bchecksum_extended\b/);
+ plan skip_all => 'Expensive data checksums test disabled'
+ unless ($ENV{PG_TEST_EXTRA} =~ /\bchecksum(_extended)?\b/);
+}
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# ---------------------------------------------------------------------------
+# Test cluster setup
+#
+
+# Initiate testcluster
+my $node = PostgreSQL::Test::Cluster->new('injection_node');
+$node->init(no_data_checksums => 1);
+$node->start;
+
+# Set up test environment
+$node->safe_psql('postgres', 'CREATE EXTENSION test_checksums;');
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+my $pgbench = undef;
+my $scalefactor = ($extended ? 10 : 1);
+my $node_loglocation = 0;
+
+$node->command_ok(
+ [
+ 'pgbench', '-p', $node->port, '-i',
+ '-s', $scalefactor, '-q', 'postgres'
+ ]);
+
+# Start a pgbench run in the background against the server specified via the
+# port passed as parameter.
+sub background_rw_pgbench
+{
+ my $port = shift;
+
+ # If a previous pgbench is still running, start by shutting it down.
+ $pgbench->finish if $pgbench;
+
+ my $clients = 1;
+ my $runtime = 2;
+
+ if ($extended)
+ {
+ # Randomize the number of pgbench clients a bit (range 1-16)
+ $clients = 1 + int(rand(15));
+ $runtime = 600;
+ }
+ my @cmd = ('pgbench', '-p', $port, '-T', $runtime, '-c', $clients);
+
+ # Randomize whether we spawn connections or not
+ push(@cmd, '-C') if ($extended && cointoss);
+ # Finally add the database name to use
+ push(@cmd, 'postgres');
+
+ $pgbench = IPC::Run::start(
+ \@cmd,
+ '<' => '/dev/null',
+ '>' => '/dev/null',
+ '2>' => '/dev/null',
+ IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
+}
+
+# Run a SQL in the background against the server specified via the
+# port passed as parameter.
+sub background_sql
+{
+ my ($port, $sql) = @_;
+
+ my @cmd = ('psql', '-p', $port, '-c', $sql, 'postgres');
+
+ $pgbench = IPC::Run::start(
+ \@cmd,
+ '<' => '/dev/null',
+ '>' => '/dev/null',
+ '2>' => '/dev/null',
+ IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
+}
+
+sub attach_injection_point
+{
+ my ($node, $point) = @_;
+ note('attaching injection point: ' . $point);
+ $node->safe_psql('postgres',
+ "SELECT injection_points_attach('" . $point . "','wait');"
+ );
+}
+
+sub wait_injection_point
+{
+ my ($node, $point) = @_;
+
+ note("waiting for the injection point to be hit");
+ $node->poll_query_until(
+ 'postgres',
+ "SELECT COUNT(*) FROM pg_catalog.pg_stat_activity WHERE wait_event = '" . $point . "'",
+ '1');
+}
+
+sub wakeup_injection_point
+{
+ my ($node, $point) = @_;
+
+ # detach before wakeup, so that we can't hit it right away
+ note("detaching the init injection point");
+ $node->safe_psql('postgres',
+ "SELECT injection_points_detach('" . $point . "');");
+
+ note('waking the init injection point: ' . $point);
+ $node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('" . $point . "');"
+ );
+}
+
+# Test behavior with a checksum transitions and a concurrent checkpoint,
+# followed by a crash.
+#
+# The test puts the instance into the initial checksum state $start,
+# triggers a checksum change. A concurrent checkpoint is performed, and
+# and the test steps through the checksum change and the checkpoint in
+# a deterministic way.
+#
+# Then the instance get restarted in immediate mode to simulate failure,
+# and the final checksum state (after recovery) is validated against the
+# expected value. The server log is checked for checksum failures.
+#
+# While the checksum change is happening, there's a r/w pgbench running in
+# the background, to generate writes.
+#
+# arguments:
+#
+# - name - name of the test sequence (for easy identification)
+# - start - start checksum state (enabled/disabled)
+# - change - checksum change to initiate
+# - final - expected checksum state at the end
+# - steps - steps to go through (wait, wakeup, sql)
+#
+# XXX This is similar to 012_concurrent_checkpoint_crash, except that the
+# checkpoint happens asynchronously (in the background), and the steps
+# step through the injection points in a deterministic way.
+#
+# XXX Some of the injection points are in a critical section, which does
+# not allow memory allocations etc. INJECTION_POINT_LOAD/_CACHED handles
+# just private memory allocation, but 'wait' requires a shmem allocation.
+# To deal with that, we setup an injection point $init outside a critical
+# section to initialize the shmem stuff needed by 'wait'. This needs to
+# be done for individual processes (e.g. the checkpointer needs to init a
+# 'wait' point too, it's not enough to init one in the checksum worker).
+#
+# XXX We could also validate the checksums using pg_checksums, if the
+# state is 'on'.
+sub test_checksum_sequence
+{
+ my ($name, $start, $change, $final, @steps) = @_;
+
+ # Start the test suite with pgbench running.
+ background_rw_pgbench($node->port);
+
+ # print the current test instructions, both into TAP output and into
+ # the server log, to make correlation easier
+
+ note($name);
+ $node->safe_psql('postgres',
+ "SELECT '========== " . $name . " =========='");
+
+ # put the cluster into the initial checksum state, synchronously
+ note('changing checksums into initial state: ' . $start);
+ enable_data_checksums($node, wait => 'on') if ($start eq 'enabled');
+ disable_data_checksums($node, wait => 'off') if ($start eq 'disabled');
+
+ # attach all the injection points mentioned in 'wait' steps
+ my $n = @steps;
+ my $s;
+ note('processing ' . $n . ' steps');
+ for ($s = 0; $s < $n; $s++)
+ {
+ my @step = $steps[$s];
+ my $action = $steps[$s][0];
+ my $value = $steps[$s][1];
+
+ note('step ' . $s . ' action ' . $action . ' / ' . $value);
+ if ($action eq 'wait')
+ {
+ attach_injection_point($node, $value);
+ }
+ }
+
+ # Trigger the checksum change, asynchronously
+ note("triggering checksum change: " . $change);
+ enable_data_checksums($node) if ($change eq 'enable');
+ disable_data_checksums($node) if ($change eq 'disable');
+
+ # now process all the steps - wait, wakeup, sql, etc.
+ $n = @steps;
+ note('processing ' . $n . ' steps');
+ for ($s = 0; $s < $n; $s++)
+ {
+ my @step = $steps[$s];
+ my $action = $steps[$s][0];
+ my $value = $steps[$s][1];
+
+ note('step ' . $s . ' action ' . $action . ' / ' . $value);
+ if ($action eq 'wait')
+ {
+ wait_injection_point($node, $value);
+ }
+ elsif ($action eq 'wakeup')
+ {
+ wakeup_injection_point($node, $value);
+ }
+ elsif ($action eq 'sql')
+ {
+ note('sql: ' . $value);
+
+ # initiate a background sql
+ background_sql($node->port, $value);
+ }
+ }
+
+ # restart the cluster, in immediate mode, to simulate a crash
+ $node->stop('immediate');
+ $node->start;
+
+ # Does the final checksum state match the expected state?
+ test_checksum_state($node, $final);
+
+ # Since the log isn't being written to now, parse the log and check
+ # for instances of checksum verification failures.
+ my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile,
+ $node_loglocation);
+ unlike(
+ $log,
+ qr/page verification failed,.+\d$/,
+ "no checksum validation errors in primary log (during WAL recovery)"
+ );
+ $node_loglocation = -s $node->logfile;
+}
+
+# sequence of steps to perform, each step is defined as an array of actions
+#
+# - type of an action (wait, wakeup, sql)
+# - injection point or SQL command
+#
+# after processing all commands, the instance gets killed / shut down with
+# immediate mode
+my @steps = undef;
+
+# CHECKSUMS-ON-1
+#
+# checkpoint after: datachecksums-enable-checksums-after-xlog
+# crash after: datachecksums-enable-checksums-after-xlogctl
+# checkpoint completes
+#
+# steps:
+# 1) checksums: write XLOG2_CHECKSUMS to WAL
+# 2) start checkpoint
+# 3) checkpointer: read XLogCtl->data_checksum_version
+# 4) checksums: update XLogCtl->data_checksum_version
+# 5) checkpointer: write CHECKPOINT_REDO
+# 6) checkpointer: complete the checkpoint
+# 7) crash
+@steps = (
+ ['wait', 'datachecksums-enable-checksums-start'], # initialize the wait
+ ['wakeup', 'datachecksums-enable-checksums-start'],
+ ['wait', 'datachecksums-enable-checksums-after-xlog'],
+ ['sql', 'checkpoint'],
+ ['wait', 'create-checkpoint-initial'], # initialize the wait
+ ['wakeup', 'create-checkpoint-initial'],
+ ['wait', 'checkpoint-before-redo-wal'],
+ ['wakeup', 'datachecksums-enable-checksums-after-xlog'],
+ ['wait', 'datachecksums-enable-checksums-after-xlogctl'],
+ ['wakeup', 'checkpoint-before-redo-wal'],
+ ['wait', 'checkpoint-before-old-wal-removal'],
+ ['wakeup', 'checkpoint-before-old-wal-removal']
+);
+
+test_checksum_sequence('CHECKSUMS-ON-1', 'disabled', 'enable', 'off', @steps);
+
+# CHECKSUMS-ON-2
+#
+# checkpoint after: datachecksums-enable-checksums-after-xlog
+# crash after: datachecksums-enable-checksums-after-xlogctl
+# checkpoint completes
+#
+# steps:
+# 1) checksums: write XLOG2_CHECKSUMS to WAL
+# 2) start checkpoint
+# 3) checksums: update XLogCtl->data_checksum_version
+# 4) checkpointer: read XLogCtl->data_checksum_version
+# 5) checkpointer: write CHECKPOINT_REDO
+# 6) checkpointer: complete the checkpoint
+# 7) crash
+@steps = (
+ ['wait', 'datachecksums-enable-checksums-start'], # initialize the wait
+ ['wakeup', 'datachecksums-enable-checksums-start'],
+ ['wait', 'datachecksums-enable-checksums-after-xlog'],
+ ['sql', 'checkpoint'],
+ ['wait', 'create-checkpoint-initial'], # initialize the wait
+ ['wakeup', 'datachecksums-enable-checksums-after-xlog'],
+ ['wakeup', 'create-checkpoint-initial'],
+ ['wait', 'checkpoint-before-xlogctl-checksums'],
+ ['wait', 'datachecksums-enable-checksums-after-xlogctl'],
+ ['wakeup', 'checkpoint-before-xlogctl-checksums'],
+ ['wait', 'checkpoint-before-old-wal-removal'],
+ ['wakeup', 'checkpoint-before-old-wal-removal']
+);
+
+test_checksum_sequence('CHECKSUMS-ON-2', 'disabled', 'enable', 'off', @steps);
+
+# CHECKSUMS-ON-3
+#
+# checkpoint happens after checksum worker updates XLogCtl
+@steps = (
+ ['wait', 'datachecksums-enable-checksums-start'], # initialize the wait
+ ['wakeup', 'datachecksums-enable-checksums-start'],
+ ['wait', 'datachecksums-enable-checksums-after-xlogctl'],
+ ['sql', 'checkpoint'],
+ ['wait', 'create-checkpoint-initial'], # initialize the wait
+ ['wakeup', 'create-checkpoint-initial'],
+ ['wait', 'checkpoint-before-redo-checksums'],
+ ['wakeup', 'checkpoint-before-redo-checksums'],
+ ['wait', 'checkpoint-before-old-wal-removal'],
+ ['wakeup', 'checkpoint-before-old-wal-removal']
+);
+
+test_checksum_sequence('CHECKSUMS-ON-3', 'disabled', 'enable', 'off', @steps);
+
+## FIXME do similar sequences for the opposite direction (enabled -> disabled)
+
+$node->stop;
+done_testing();
--
2.54.0
view thread (60+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Changing the state of data checksums in a running cluster
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox