public inbox for [email protected]
help / color / mirror / Atom feedFrom: Daniel Gustafsson <[email protected]>
To: Ayush Tiwari <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Cc: Heikki Linnakangas <[email protected]>
Cc: Tomas Vondra <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: Bernd Helmle <[email protected]>
Cc: Michael Paquier <[email protected]>
Cc: Michael Banck <[email protected]>
Cc: SATYANARAYANA NARLAPURAM <[email protected]>
Subject: Re: Changing the state of data checksums in a running cluster
Date: Wed, 29 Apr 2026 00:06:14 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <CAJTYsWWn9_n2CGJOPNTvnqQUNtE3d4EXm5hy51CtEXptykdGUA@mail.gmail.com>
References: <[email protected]>
<[email protected]>
<5mv5vtbqj2xm2wmevsi22smyn32jtznva3ya3daih3ixh3onex@s5fyab63xt3c>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<CAHg+QDfRk4-S7DMmdbXJnQ-xF=sUpMAKuh8b83ObLqYVKx5QLA@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAJTYsWWn9_n2CGJOPNTvnqQUNtE3d4EXm5hy51CtEXptykdGUA@mail.gmail.com>
> On 25 Apr 2026, at 13:59, Ayush Tiwari <[email protected]> wrote:
> I don't think 0006 fully addresses the duplicate-launcher exit race [1]. I was
> able to reproduce the issue with a deterministic test using the existing
> test_checksums delay hooks: delay
> the initial StartDataChecksumsWorkerLauncher()
> path so two quick pg_enable_data_checksums() calls can queue two launchers,
> delay the final transition to on, then fire follow-up pg_enable_data_checksums()
> calls while the real launcher is still active.
Thanks to the really great reproducer script I got offlist (which is attached
here) I was able to reproduce and figure out what was wrong. This sequence
exposed a thinko in the cleanup handler where a process exiting due to an
already running launcher would enter cleanup, and thus wipe state from the
running launcher. It needed this quite narrow window to hit, but it was good
to catch that. Fixed in the attached by giving processes a way to register
themselves as needing no cleanup when exiting.
> A few other smaller comments/nits:
>
> 1. In src/backend/access/transam/xlog.c, the comment in SetDataChecksumsOff()
> says the branch implies the state is inprogress-on or inprogress-off, but
> after the patch inprogress-on is handled by the preceding if condition. It
> looks like this should probably only mention inprogress-off.
Fixed.
> 2. There is a repeated typo in comments:
>
> src/backend/utils/init/postinit.c: "interrups" -> "interrupts"
> src/backend/postmaster/auxprocess.c: "interrups" -> "interrupts"
Fixed.
> 3. A few commit messages could use a quick polish pass:
>
> 0002: "onine" -> "online"
> 0004: "Additionelly" -> "Additionally"
> 0006: "being enabled of disabled" -> "being enabled or disabled"
> 0006: "change it's operation" -> "change its operation"
> 0006: "ss now avoided" -> "is now avoided"
Fixed.
> 4. One minor question on 0006: the runtime cost-parameter update path is only
> meaningful while checksums are being enabled. That looks fine from the current
> control flow, but would it be worth adding a short comment or assertion near
> that update path to make the assumption explicit?
We have this assertion which makes sure that there are no costs passed for
disabling, not sure if we need much more since there is now way for the user to
inject any cost parameters.
#ifdef USE_ASSERT_CHECKING
/* The cost delay settings have no effect when disabling */
if (op == DISABLE_DATACHECKSUMS)
Assert(cost_delay == 0 && cost_limit == 0);
#endif
--
Daniel Gustafsson
#!/usr/bin/env bash
set -euo pipefail
prefix="${PGPREFIX:-$PWD/tmp_install/usr/local/pgsql}"
bindir="$prefix/bin"
sharedir="$prefix/share"
libdir="$prefix/lib"
rows="${ROWS:-80000}"
followup_calls="${FOLLOWUP_CALLS:-5}"
followup_sleep="${FOLLOWUP_SLEEP:-0.2}"
post_run_sleep="${POST_RUN_SLEEP:-8}"
if [[ ! -x "$bindir/initdb" || ! -x "$bindir/pg_ctl" || ! -x "$bindir/psql" ]]; then
cat >&2 <<EOF
could not find PostgreSQL binaries under: $bindir
Set PGPREFIX to the install prefix, or install into:
$PWD/tmp_install/usr/local/pgsql
EOF
exit 1
fi
if [[ ! -f "$sharedir/extension/test_checksums.control" || ! -f "$libdir/test_checksums.so" ]]; then
cat >&2 <<EOF
test_checksums is not installed under: $prefix
Install the test modules first, for example:
make -s -C src/test/modules/injection_points install DESTDIR="$PWD/tmp_install"
make -s -C src/test/modules/test_checksums install DESTDIR="$PWD/tmp_install"
EOF
exit 1
fi
export PATH="$bindir:$PATH"
export LD_LIBRARY_PATH="$libdir${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
base="/tmp/pg-dc-launcher-race-$$"
datadir="$base/data"
logfile="$base/server.log"
port="${PGPORT:-$((55432 + RANDOM % 1000))}"
started=0
mkdir -p "$base"
cleanup()
{
if [[ "$started" -eq 1 ]]; then
pg_ctl -D "$datadir" -m immediate -w stop >"$base/pg_ctl_stop.out" 2>&1 || true
fi
echo "repro artifacts left in: $base"
}
trap cleanup EXIT
initdb --no-data-checksums --auth trust --no-sync --no-instructions -D "$datadir" >"$base/initdb.out"
cat >>"$datadir/postgresql.conf" <<EOF
port = $port
listen_addresses = ''
unix_socket_directories = '$base'
max_worker_processes = 20
log_line_prefix = '%m [%p] '
log_min_messages = debug1
EOF
pg_ctl -D "$datadir" -l "$logfile" -w start >"$base/pg_ctl_start.out"
started=1
psql_args=(-h "$base" -p "$port" -v ON_ERROR_STOP=1 -X postgres)
count_log()
{
grep -F -c "$1" "$logfile" 2>/dev/null || true
}
wait_for_log_count()
{
local needle="$1"
local expected="$2"
local timeout_seconds="$3"
local end=$((SECONDS + timeout_seconds))
local count
while (( SECONDS < end )); do
count=$(count_log "$needle")
if (( count >= expected )); then
return 0
fi
sleep 0.1
done
count=$(count_log "$needle")
echo "timed out waiting for at least $expected occurrence(s) of: $needle" >&2
echo "saw $count occurrence(s)" >&2
return 1
}
psql "${psql_args[@]}" -c "CREATE EXTENSION test_checksums;" >"$base/setup-extension.out"
psql "${psql_args[@]}" -c "CREATE TABLE t AS SELECT g, repeat('x', 2000) AS payload FROM generate_series(1, $rows) g;" >"$base/setup-table.out"
psql "${psql_args[@]}" -c "CHECKPOINT;" >"$base/checkpoint.out"
# Delay the two initial pg_enable_data_checksums() callers at the launcher
# registration entry point, so both callers enter the launch path together.
psql "${psql_args[@]}" -c "SELECT dcw_inject_startup_delay(true);" >"$base/attach-startup-delay.out"
# Delay launcher main before it claims DataChecksumState->launcher_running. This
# makes the same race window deterministic instead of relying on scheduler luck.
psql "${psql_args[@]}" -c "SELECT dcw_inject_launcher_delay(true);" >"$base/attach-launcher-delay.out"
# Delay the final transition to on, keeping the winning launcher active long
# enough for follow-up pg_enable_data_checksums() calls to probe the state.
psql "${psql_args[@]}" -c "SELECT dcw_inject_delay_barrier(true);" >"$base/attach-final-delay.out"
psql "${psql_args[@]}" -c "SELECT pg_enable_data_checksums(20, 1);" >"$base/enable1.out" 2>&1 &
pid1=$!
psql "${psql_args[@]}" -c "SELECT pg_enable_data_checksums(20, 1);" >"$base/enable2.out" 2>&1 &
pid2=$!
wait "$pid1" || true
wait "$pid2" || true
# Later pg_enable_data_checksums() calls should not be delayed at the SQL entry
# point; they need to probe the shared launcher_running state promptly.
psql "${psql_args[@]}" -c "SELECT dcw_inject_startup_delay(false);" >"$base/detach-startup-delay.out"
wait_for_log_count 'background worker "datachecksums launcher" started' 2 15
wait_for_log_count 'background worker "datachecksums launcher" already running, exiting' 1 10
wait_for_log_count 'initiating data checksum processing in database "postgres"' 1 10
# Stop delaying later launchers. Follow-up calls are fired immediately after
# the redundant launcher has exited, while the owning launcher is still
# processing or waiting at the final checksum-on barrier.
psql "${psql_args[@]}" -c "SELECT dcw_inject_launcher_delay(false);" >"$base/detach-launcher-delay.out"
# A single follow-up call is often enough after the redundant launcher exits.
# Use a small handful by default to avoid depending on exact worker timing; set
# FOLLOWUP_CALLS=1 for the smallest variant.
pids=()
for _ in $(seq 1 "$followup_calls"); do
psql "${psql_args[@]}" -c "SELECT pg_enable_data_checksums(20, 1);" >"$base/enable-followup-$_.out" 2>&1 &
pids+=("$!")
sleep "$followup_sleep"
done
for pid in "${pids[@]}"; do
wait "$pid" || true
done
sleep "$post_run_sleep"
echo
echo "--- relevant server log lines ---"
grep -E 'datachecksums launcher|data checksum processing already running|data checksums already in desired state|enabling data checksums requested|initiating data checksum processing|cannot set data checksums|data checksums are now|data checksum processing was aborted|disabling data checksums requested' "$logfile" || true
echo
echo "--- summary ---"
launcher_workers_started=$(count_log 'background worker "datachecksums launcher" started')
checksum_processing_starts=$(count_log 'enabling data checksums requested')
already_running_responses=$(count_log 'data checksum processing already running')
checksum_state_warnings=$(count_log 'cannot set data checksums')
echo "launcher workers started: $launcher_workers_started"
echo "checksum processing starts: $checksum_processing_starts"
echo "already-running responses: $already_running_responses"
echo "checksum state warnings: $checksum_state_warnings"
echo "final data_checksums setting: $(psql "${psql_args[@]}" -At -c 'SHOW data_checksums;' || true)"
echo
if (( checksum_processing_starts > 1 )); then
echo "BUG REPRODUCED: more than one launcher started checksum processing."
else
echo "BUG NOT REPRODUCED: only one launcher started checksum processing."
exit 1
fi
echo
cat <<'EOF'
Expected result on the unfixed patchset:
After the redundant launcher exits, a follow-up call can start another
datachecksums launcher while the original launcher is still active. The log
shows more than one "enabling data checksums requested" line. Depending on
timing, it may also show "cannot set data checksums to \"on\"...".
Expected result with the launcher_exit() ownership fix:
Two launcher workers may start, but the redundant one only logs "already
running, exiting". While the owning launcher is still active, follow-up
calls log "data checksum processing already running"; after it completes,
they may log "data checksums already in desired state, exiting". No
additional launcher starts checksum processing.
EOF
Attachments:
[application/octet-stream] v2-0001-Prevent-pg_enable-disable_data_checksums-on-stand.patch (1.5K, ../[email protected]/2-v2-0001-Prevent-pg_enable-disable_data_checksums-on-stand.patch)
download | inline diff:
From 80c612cbfffc1115a701f4ef35953608de678ca1 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Tue, 28 Apr 2026 23:48:30 +0200
Subject: [PATCH v2 1/8] Prevent pg_enable/disable_data_checksums() on standby
These functions missed a RecoveryInProgress() check, allowing them to
be called on a hot standby. Enabling, or disabling, checksums on the
standby only would cause the cluster to get out of sync and replaying
checksum transitions to fail.
Author: Satyanarayana Narlapuram <[email protected]>
Discussion: https://postgr.es/m/CAHg+QDfRk4-S7DMmdbXJnQ-xF=sUpMAKuh8b83ObLqYVKx5QLA@mail.gmail.com
---
src/backend/postmaster/datachecksum_state.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/postmaster/datachecksum_state.c b/src/backend/postmaster/datachecksum_state.c
index 5556a9ca893..ea102086144 100644
--- a/src/backend/postmaster/datachecksum_state.c
+++ b/src/backend/postmaster/datachecksum_state.c
@@ -487,6 +487,8 @@ AbsorbDataChecksumsBarrier(ProcSignalBarrierType barrier)
Datum
disable_data_checksums(PG_FUNCTION_ARGS)
{
+ PreventCommandDuringRecovery("pg_disable_data_checksums()");
+
if (!superuser())
ereport(ERROR,
errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
@@ -507,6 +509,8 @@ enable_data_checksums(PG_FUNCTION_ARGS)
int cost_delay = PG_GETARG_INT32(0);
int cost_limit = PG_GETARG_INT32(1);
+ PreventCommandDuringRecovery("pg_enable_data_checksums()");
+
if (!superuser())
ereport(ERROR,
errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
--
2.39.3 (Apple Git-146)
[application/octet-stream] v2-0002-Test-improvements-for-online-checksums.patch (7.3K, ../[email protected]/3-v2-0002-Test-improvements-for-online-checksums.patch)
download | inline diff:
From 1d7b0b4db2db1191a1aa89a62965c80633137d16 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Tue, 28 Apr 2026 23:48:39 +0200
Subject: [PATCH v2 2/8] Test improvements for online checksums
This includes a number of smaller fixups to the online checksums test
module which were found during postcommit review and stabilization
work.
* Fix scope increase for PG_TEST_EXTRA: The online checksums tests
have two levels of PG_TEST_EXTRA, checksum and checksums_extended
for extra test runs and test runs with increased randomization.
The logic for increasing the number of test iterations was however
backwards.
* Change stopmode for PITR test: The pitr suite used immediate stop
mode which caused problems on slower machines where the sigquit
would interrupt archive commands leaving partial WAL files behind.
This would then prevent restart. Fix by using fast mode which is
the appropriate mode for the test at hand. Also increase timeouts
to help slower test systems since an expired timeout will incur
the same effect as an immediate standby with a partial WAL left
behind. This issue was observed when running the test suites on
a Raspberry Pi 4 machine.
* Improve logging: The test suite for data checksums use a set of
helper functions in a Perl module to avoid repeating code, this
makes sure that the helper functions do a better job of logging
their test output to make debug easier.
* Remove unused code: wait_for_cluster_crash was used during the
development of online checksums but was never used in any test
which shipped, so remove the function.
* Standby fixes: Ensure no vacuum on pgbench init on standby with
-n to avoid bogus error message in the log, and enable
hot_standby_feedback to prevent queries from getting cancelled
due to recovery on slower systems.
Author: Daniel Gustafsson <[email protected]>
Author: Tomas Vondra <[email protected]>
Discussion: https://postgr.es/m/xxx
---
.../test_checksums/t/007_pgbench_standby.pl | 12 +++--
src/test/modules/test_checksums/t/008_pitr.pl | 5 +-
.../test_checksums/t/DataChecksums/Utils.pm | 53 +++----------------
3 files changed, 20 insertions(+), 50 deletions(-)
diff --git a/src/test/modules/test_checksums/t/007_pgbench_standby.pl b/src/test/modules/test_checksums/t/007_pgbench_standby.pl
index f3611e7ce25..0b3996f1d69 100644
--- a/src/test/modules/test_checksums/t/007_pgbench_standby.pl
+++ b/src/test/modules/test_checksums/t/007_pgbench_standby.pl
@@ -49,8 +49,8 @@ my $node_standby_loglocation = 0;
# of tests performed and the wall time taken is non-deterministic as the test
# performs a lot of randomized actions, but 5 iterations will be a long test
# run regardless.
-my $TEST_ITERATIONS = 5;
-$TEST_ITERATIONS = 1 if ($extended);
+my $TEST_ITERATIONS = 1;
+$TEST_ITERATIONS = 5 if ($extended);
# Variables which record the current state of the cluster
my $data_checksum_state = 'off';
@@ -83,6 +83,7 @@ sub background_pgbench
push(@cmd, '-C') if ($extended && cointoss());
# If we run on a standby it needs to be a read-only benchmark
push(@cmd, '-S') if ($standby);
+ push(@cmd, '-n') if ($standby);
# Finally add the database name to use
push(@cmd, 'postgres');
@@ -146,8 +147,10 @@ sub flip_data_checksums
. "FROM pg_catalog.pg_settings "
. "WHERE name = 'data_checksums';");
- is(($result eq 'inprogress-on' || $result eq 'on'),
- 1, 'ensure checksums are on, or in progress, on standby_1');
+ is( ($result eq 'inprogress-on' || $result eq 'on'),
+ 1,
+ 'ensure checksums are on, or in progress, on standby_1, got: '
+ . $result);
# Wait for checksums enabled on the primary and standby
wait_for_checksum_state($node_primary, 'on');
@@ -210,6 +213,7 @@ $node_primary->append_conf(
qq[
max_connections = 30
log_statement = none
+hot_standby_feedback = on
]);
$node_primary->start;
$node_primary->safe_psql('postgres', 'CREATE EXTENSION test_checksums;');
diff --git a/src/test/modules/test_checksums/t/008_pitr.pl b/src/test/modules/test_checksums/t/008_pitr.pl
index e8cb2b0ed96..1f8176686fd 100644
--- a/src/test/modules/test_checksums/t/008_pitr.pl
+++ b/src/test/modules/test_checksums/t/008_pitr.pl
@@ -124,11 +124,14 @@ $node_primary->init(
has_archiving => 1,
allows_streaming => 1,
no_data_checksums => 1);
+my $timeout_unit = 's';
$node_primary->append_conf(
'postgresql.conf',
qq[
max_connections = 100
log_statement = none
+wal_sender_timeout = $PostgreSQL::Test::Utils::timeout_default$timeout_unit
+wal_receiver_timeout = $PostgreSQL::Test::Utils::timeout_default$timeout_unit
]);
$node_primary->start;
@@ -154,7 +157,7 @@ my ($pre_lsn, $post_lsn) = flip_data_checksums();
$node_primary->safe_psql('postgres', "UPDATE t SET a = a + 1;");
$node_primary->safe_psql('postgres', "SELECT pg_create_restore_point('a');");
$node_primary->safe_psql('postgres', "UPDATE t SET a = a + 1;");
-$node_primary->stop('immediate');
+$node_primary->stop('fast');
my $node_pitr = PostgreSQL::Test::Cluster->new('pitr_backup');
$node_pitr->init_from_backup(
diff --git a/src/test/modules/test_checksums/t/DataChecksums/Utils.pm b/src/test/modules/test_checksums/t/DataChecksums/Utils.pm
index fb704623a60..cb78dd6ecfb 100644
--- a/src/test/modules/test_checksums/t/DataChecksums/Utils.pm
+++ b/src/test/modules/test_checksums/t/DataChecksums/Utils.pm
@@ -43,7 +43,6 @@ our @EXPORT = qw(
stopmode
test_checksum_state
wait_for_checksum_state
- wait_for_cluster_crash
);
=pod
@@ -67,7 +66,10 @@ sub test_checksum_state
my $result = $postgresnode->safe_psql('postgres',
"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
);
- is($result, $state, 'ensure checksums are set to ' . $state);
+ is($result, $state,
+ 'ensure checksums are set to '
+ . $state . ' on '
+ . $postgresnode->name());
return $result eq $state;
}
@@ -89,52 +91,13 @@ sub wait_for_checksum_state
'postgres',
"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
$state);
- is($res, 1, 'ensure data checksums are transitioned to ' . $state);
+ is($res, 1,
+ 'ensure data checksums are transitioned to '
+ . $state . ' on '
+ . $postgresnode->name());
return $res == 1;
}
-=item wait_for_cluster_crash(node, params)
-
-Repeatedly test if the cluster running at B<node> responds to connections
-and return when it no longer does so, or when it times out. Processing will
-run for $PostgreSQL::Test::Utils::timeout_default seconds unless a timeout
-value is specified as a parameter. Returns True if the cluster crashed, else
-False if the process timed out.
-
-=over
-
-=item timeout
-
-Approximate number of seconds to wait for cluster to crash, default is
-$PostgreSQL::Test::Utils::timeout_default. There are no real-time guarantees
-that the total process time won't exceed the timeout.
-
-=back
-
-=cut
-
-sub wait_for_cluster_crash
-{
- my $postgresnode = shift;
- my %params = @_;
- my $crash = 0;
-
- $params{timeout} = $PostgreSQL::Test::Utils::timeout_default
- unless (defined($params{timeout}));
-
- for (my $naps = 0; $naps < $params{timeout}; $naps++)
- {
- if (!$postgresnode->is_alive)
- {
- $crash = 1;
- last;
- }
- sleep(1);
- }
-
- return $crash == 1;
-}
-
=item enable_data_checksums($node, %params)
Function for enabling data checksums in the cluster running at B<node>.
--
2.39.3 (Apple Git-146)
[application/octet-stream] v2-0003-Handle-data_checksum-state-changes-during-launche.patch (6.5K, ../[email protected]/4-v2-0003-Handle-data_checksum-state-changes-during-launche.patch)
download | inline diff:
From 015b9095011b3e606a5a7e8fbb70936cd7c2c747 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Tue, 28 Apr 2026 23:49:03 +0200
Subject: [PATCH v2 3/8] Handle data_checksum state changes during
launcher_exit
When erroring out from the datachecksums launcher during data checksum
enabling, before state has transitioned to "on", we revert back to the
"off" state. Since checksums weren't enabled, there is no use staying
in an inprogress state since the checksum launcher currently doesn't
support restarting from where it left off. Should restartability get
added in the future, this would need to be revisited. This state
transition was however missing from the allowed transitions in the
statemachine causing an error.
Author: Daniel Gustafsson <[email protected]>
Discussion: https://postgr.es/m/xxx
---
src/backend/access/transam/xlog.c | 15 +++---
src/backend/postmaster/datachecksum_state.c | 51 +++++++++++++++++++--
2 files changed, 54 insertions(+), 12 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index e39af79c03b..f74d7a2ab1a 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4871,13 +4871,14 @@ SetDataChecksumsOff(void)
}
/*
- * If data checksums are currently enabled we first transition to the
- * "inprogress-off" state during which backends continue to write
- * checksums without verifying them. When all backends are in
- * "inprogress-off" the next transition to "off" can be performed, after
- * which all data checksum processing is disabled.
- */
- if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_VERSION)
+ * If data checksums are currently enabled, or in the process of being
+ * enabled, we first transition to the "inprogress-off" state during which
+ * backends continue to write checksums without verifying them. When all
+ * backends are in "inprogress-off" the next transition to "off" can be
+ * performed, after which all data checksum processing is disabled.
+ */
+ if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_VERSION ||
+ XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_ON)
{
SpinLockRelease(&XLogCtl->info_lck);
diff --git a/src/backend/postmaster/datachecksum_state.c b/src/backend/postmaster/datachecksum_state.c
index ea102086144..e26803bc501 100644
--- a/src/backend/postmaster/datachecksum_state.c
+++ b/src/backend/postmaster/datachecksum_state.c
@@ -235,7 +235,7 @@ typedef struct ChecksumBarrierCondition
int to;
} ChecksumBarrierCondition;
-static const ChecksumBarrierCondition checksum_barriers[6] =
+static const ChecksumBarrierCondition checksum_barriers[7] =
{
/*
* Disabling checksums: If checksums are currently enabled, disabling must
@@ -261,6 +261,12 @@ static const ChecksumBarrierCondition checksum_barriers[6] =
* checksums, we can go straight back to 'on'
*/
{PG_DATA_CHECKSUM_INPROGRESS_OFF, PG_DATA_CHECKSUM_VERSION},
+
+ /*
+ * If checksums are being enabled when launcher_exit is executed, state
+ * is set to off since we cannot reach on at that point.
+ */
+ {PG_DATA_CHECKSUM_INPROGRESS_ON, PG_DATA_CHECKSUM_INPROGRESS_OFF},
};
/*
@@ -323,6 +329,13 @@ typedef struct DataChecksumsStateStruct
* catalogs
*/
bool process_shared_catalogs;
+
+ /*
+ * List of PIDs for which the launcher_exit should avoid doing any abort
+ * cleanup, as they are exiting gracefully due to being launched while
+ * another launcher is already running.
+ */
+ List *no_abort;
} DataChecksumsStateStruct;
/* Shared memory segment for datachecksumsworker */
@@ -348,6 +361,7 @@ static DataChecksumsWorkerOperation operation;
/* Prototypes */
static void DataChecksumsShmemRequest(void *arg);
+static void DataChecksumsShmemInit(void *arg);
static bool DatabaseExists(Oid dboid);
static List *BuildDatabaseList(void);
static List *BuildRelationList(bool temp_relations, bool include_shared);
@@ -360,6 +374,7 @@ static void WaitForAllTransactionsToFinish(void);
const ShmemCallbacks DataChecksumsShmemCallbacks = {
.request_fn = DataChecksumsShmemRequest,
+ .init_fn = DataChecksumsShmemInit,
};
/*****************************************************************************
@@ -771,7 +786,9 @@ ProcessDatabase(DataChecksumsWorkerDatabase *db)
pid_t pid;
char activity[NAMEDATALEN + 64];
+ LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
DataChecksumState->success = DATACHECKSUMSWORKER_FAILED;
+ LWLockRelease(DataChecksumsWorkerLock);
memset(&bgw, 0, sizeof(bgw));
bgw.bgw_flags = BGWORKER_SHMEM_ACCESS | BGWORKER_BACKEND_DATABASE_CONNECTION;
@@ -881,14 +898,29 @@ ProcessDatabase(DataChecksumsWorkerDatabase *db)
/*
* launcher_exit
*
- * Internal routine for cleaning up state when the launcher process exits. We
- * need to clean up the abort flag to ensure that processing started again if
- * it was previously aborted (note: started again, *not* restarted from where
- * it left off).
+ * Internal routine for cleaning up state when the launcher process exits. If
+ * the process is exiting due to a duplicate started launcher, cleanup should
+ * not be done as that would interfere with the running launcher. Otherwise,
+ * we need to clean up the abort flag to ensure that processing started again
+ * if it was previously aborted (note: started again, *not* restarted from
+ * where it left off).
*/
static void
launcher_exit(int code, Datum arg)
{
+ /* Check for processes which are exiting gracefully */
+ LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
+ foreach_int(dup, DataChecksumState->no_abort)
+ {
+ if (dup == MyProcPid)
+ {
+ DataChecksumState->no_abort = list_delete_int(DataChecksumState->no_abort, MyProcPid);
+ LWLockRelease(DataChecksumsWorkerLock);
+ return;
+ }
+ }
+ LWLockRelease(DataChecksumsWorkerLock);
+
abort_requested = false;
if (launcher_running)
@@ -1040,6 +1072,7 @@ DataChecksumsWorkerLauncherMain(Datum arg)
ereport(LOG,
errmsg("background worker \"datachecksums launcher\" already running, exiting"));
/* Launcher was already running, let it finish */
+ DataChecksumState->no_abort = lappend_int(DataChecksumState->no_abort, MyProcPid);
LWLockRelease(DataChecksumsWorkerLock);
return;
}
@@ -1278,6 +1311,14 @@ DataChecksumsShmemRequest(void *arg)
);
}
+static void
+DataChecksumsShmemInit(void *arg)
+{
+ DataChecksumState->no_abort = NIL;
+ DataChecksumState->launcher_running = false;
+ DataChecksumState->worker_pid = InvalidPid;
+}
+
/*
* DatabaseExists
*
--
2.39.3 (Apple Git-146)
[application/octet-stream] v2-0004-Fix-invalid-checksum-state-transition-in-checkpoi.patch (13.2K, ../[email protected]/5-v2-0004-Fix-invalid-checksum-state-transition-in-checkpoi.patch)
download | inline diff:
From e3accf52288b20ffba3648d1071fd569673470f6 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Tue, 28 Apr 2026 23:49:05 +0200
Subject: [PATCH v2 4/8] Fix invalid checksum state transition in checkpoints
Commit 78e950cb8 added checksum state handling to all XLOG_CHECKPOINT
records which caused unnecessary state transitions and emission of
procsignal barriers. Remove as only the _REDO record need to handle
checksum state. Barrier emission is also consistently made after
controlfile updates to avoid race conditions.
Additionally, interrupts are held between calling ProcSignalInit and
InitLocalDataChecksumState to remove a window where otherwise invalid
state transitions can happen.
Also remove a pointless assertion on Controlfile which will never hit.
Author: Tomas Vondra <[email protected]>
Author: Daniel Gustafsson <[email protected]>
Discussion: https://postgr.es/m/xxx
---
src/backend/access/transam/xlog.c | 106 ++++++--------------
src/backend/postmaster/auxprocess.c | 10 ++
src/backend/postmaster/datachecksum_state.c | 8 +-
src/backend/utils/init/postinit.c | 10 ++
4 files changed, 55 insertions(+), 79 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f74d7a2ab1a..9b1a10dbd1c 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4723,8 +4723,6 @@ SetDataChecksumsOnInProgress(void)
{
uint64 barrier;
- Assert(ControlFile != NULL);
-
/*
* The state transition is performed in a critical section with
* checkpoints held off to provide crash safety.
@@ -4738,25 +4736,16 @@ SetDataChecksumsOnInProgress(void)
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON;
SpinLockRelease(&XLogCtl->info_lck);
- barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);
-
- MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
- END_CRIT_SECTION();
-
- /*
- * Update the controlfile before waiting since if we have an immediate
- * shutdown while waiting we want to come back up with checksums enabled.
- */
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
ControlFile->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON;
UpdateControlFile();
LWLockRelease(ControlFileLock);
- /*
- * Await state change in all backends to ensure that all backends are in
- * "inprogress-on". Once done we know that all backends are writing data
- * checksums.
- */
+ barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);
+
+ MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
+ END_CRIT_SECTION();
+
WaitForProcSignalBarrier(barrier);
}
@@ -4787,8 +4776,6 @@ SetDataChecksumsOn(void)
{
uint64 barrier;
- Assert(ControlFile != NULL);
-
SpinLockAcquire(&XLogCtl->info_lck);
/*
@@ -4818,11 +4805,6 @@ SetDataChecksumsOn(void)
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
SpinLockRelease(&XLogCtl->info_lck);
- barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);
-
- MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
- END_CRIT_SECTION();
-
/*
* Update the controlfile before waiting since if we have an immediate
* shutdown while waiting we want to come back up with checksums enabled.
@@ -4832,12 +4814,12 @@ SetDataChecksumsOn(void)
UpdateControlFile();
LWLockRelease(ControlFileLock);
- RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
+ barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);
- /*
- * Await state transition to "on" in all backends. When done we know that
- * data checksums are both written and verified in all backends.
- */
+ MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
+ END_CRIT_SECTION();
+
+ RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
WaitForProcSignalBarrier(barrier);
}
@@ -4859,8 +4841,6 @@ SetDataChecksumsOff(void)
{
uint64 barrier;
- Assert(ControlFile != NULL);
-
SpinLockAcquire(&XLogCtl->info_lck);
/* If data checksums are already disabled there is nothing to do */
@@ -4891,22 +4871,17 @@ SetDataChecksumsOff(void)
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF;
SpinLockRelease(&XLogCtl->info_lck);
+ LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+ ControlFile->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF;
+ UpdateControlFile();
+ LWLockRelease(ControlFileLock);
+
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
- LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
- ControlFile->data_checksum_version = PG_DATA_CHECKSUM_OFF;
- UpdateControlFile();
- LWLockRelease(ControlFileLock);
-
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
-
- /*
- * Update local state in all backends to ensure that any backend in
- * "on" state is changed to "inprogress-off".
- */
WaitForProcSignalBarrier(barrier);
/*
@@ -4935,18 +4910,17 @@ SetDataChecksumsOff(void)
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_OFF;
SpinLockRelease(&XLogCtl->info_lck);
- barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);
-
- MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
- END_CRIT_SECTION();
-
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
ControlFile->data_checksum_version = PG_DATA_CHECKSUM_OFF;
UpdateControlFile();
LWLockRelease(ControlFileLock);
- RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
+ barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);
+ MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
+ END_CRIT_SECTION();
+
+ RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
WaitForProcSignalBarrier(barrier);
}
@@ -4961,6 +4935,7 @@ SetDataChecksumsOff(void)
void
InitLocalDataChecksumState(void)
{
+ Assert(InterruptHoldoffCount > 0);
SpinLockAcquire(&XLogCtl->info_lck);
SetLocalDataChecksumState(XLogCtl->data_checksum_version);
SpinLockRelease(&XLogCtl->info_lck);
@@ -5427,7 +5402,6 @@ XLOGShmemInit(void *arg)
/* Use the checksum info from control file */
XLogCtl->data_checksum_version = ControlFile->data_checksum_version;
-
SetLocalDataChecksumState(XLogCtl->data_checksum_version);
SpinLockInit(&XLogCtl->Insert.insertpos_lck);
@@ -7518,7 +7492,9 @@ CreateCheckPoint(int flags)
* Get the current data_checksum_version value from xlogctl, valid at the
* time of the checkpoint.
*/
+ SpinLockAcquire(&XLogCtl->info_lck);
checkPoint.dataChecksumState = XLogCtl->data_checksum_version;
+ SpinLockRelease(&XLogCtl->info_lck);
if (shutdown)
{
@@ -7639,10 +7615,6 @@ CreateCheckPoint(int flags)
checkPoint.nextOid += TransamVariables->oidCount;
LWLockRelease(OidGenLock);
- SpinLockAcquire(&XLogCtl->info_lck);
- checkPoint.dataChecksumState = XLogCtl->data_checksum_version;
- SpinLockRelease(&XLogCtl->info_lck);
-
checkPoint.logicalDecodingEnabled = IsLogicalDecodingEnabled();
MultiXactGetCheckptMulti(shutdown,
@@ -7792,9 +7764,6 @@ CreateCheckPoint(int flags)
ControlFile->minRecoveryPoint = InvalidXLogRecPtr;
ControlFile->minRecoveryPointTLI = 0;
- /* make sure we start with the checksum version as of the checkpoint */
- ControlFile->data_checksum_version = checkPoint.dataChecksumState;
-
/*
* Persist unloggedLSN value. It's reset on crash recovery, so this goes
* unused on non-shutdown checkpoints, but seems useful to store it always
@@ -8871,11 +8840,6 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceOldest(checkPoint.oldestMulti,
checkPoint.oldestMultiDB);
- SpinLockAcquire(&XLogCtl->info_lck);
- XLogCtl->data_checksum_version = checkPoint.dataChecksumState;
- SetLocalDataChecksumState(checkPoint.dataChecksumState);
- SpinLockRelease(&XLogCtl->info_lck);
-
/*
* No need to set oldestClogXid here as well; it'll be set when we
* redo an xl_clog_truncate if it changed since initialization.
@@ -8936,6 +8900,8 @@ xlog_redo(XLogReaderState *record)
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
ControlFile->checkPointCopy.nextXid = checkPoint.nextXid;
ControlFile->data_checksum_version = checkPoint.dataChecksumState;
+
+ UpdateControlFile();
LWLockRelease(ControlFileLock);
/*
@@ -8962,8 +8928,6 @@ xlog_redo(XLogReaderState *record)
{
CheckPoint checkPoint;
TimeLineID replayTLI;
- bool new_state = false;
- int old_state;
memcpy(&checkPoint, XLogRecGetData(record), sizeof(CheckPoint));
/* In an ONLINE checkpoint, treat the XID counter as a minimum */
@@ -9002,8 +8966,6 @@ xlog_redo(XLogReaderState *record)
/* ControlFile->checkPointCopy always tracks the latest ckpt XID */
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
ControlFile->checkPointCopy.nextXid = checkPoint.nextXid;
- old_state = ControlFile->data_checksum_version;
- ControlFile->data_checksum_version = checkPoint.dataChecksumState;
LWLockRelease(ControlFileLock);
/* TLI should not change in an on-line checkpoint */
@@ -9015,18 +8977,6 @@ xlog_redo(XLogReaderState *record)
RecoveryRestartPoint(&checkPoint, record);
- /*
- * If the data checksum state change we need to emit a barrier.
- */
- SpinLockAcquire(&XLogCtl->info_lck);
- XLogCtl->data_checksum_version = checkPoint.dataChecksumState;
- if (checkPoint.dataChecksumState != old_state)
- new_state = true;
- SpinLockRelease(&XLogCtl->info_lck);
-
- if (new_state)
- EmitAndWaitDataChecksumsBarrier(checkPoint.dataChecksumState);
-
/*
* After replaying a checkpoint record, free all smgr objects.
* Otherwise we would never do so for dropped relations, as the
@@ -9195,6 +9145,7 @@ xlog_redo(XLogReaderState *record)
SpinLockAcquire(&XLogCtl->info_lck);
XLogCtl->data_checksum_version = redo_rec.data_checksum_version;
+ SetLocalDataChecksumState(redo_rec.data_checksum_version);
if (redo_rec.data_checksum_version != ControlFile->data_checksum_version)
new_state = true;
SpinLockRelease(&XLogCtl->info_lck);
@@ -9268,6 +9219,11 @@ xlog2_redo(XLogReaderState *record)
XLogCtl->data_checksum_version = state.new_checksum_state;
SpinLockRelease(&XLogCtl->info_lck);
+ LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+ ControlFile->data_checksum_version = state.new_checksum_state;
+ UpdateControlFile();
+ LWLockRelease(ControlFileLock);
+
/*
* Block on a procsignalbarrier to await all processes having seen the
* change to checksum status. Once the barrier has been passed we can
diff --git a/src/backend/postmaster/auxprocess.c b/src/backend/postmaster/auxprocess.c
index 8fdc518b3a1..ba8c9add67a 100644
--- a/src/backend/postmaster/auxprocess.c
+++ b/src/backend/postmaster/auxprocess.c
@@ -68,6 +68,14 @@ AuxiliaryProcessMainCommon(void)
BaseInit();
+ /*
+ * Prevent consuming interrupts between setting ProcSignalInit and setting
+ * the initial local data checksum value. If a barrier is emitted, and
+ * absorbed, before local cached state is initialized the state transition
+ * can be invalid.
+ */
+ HOLD_INTERRUPTS();
+
ProcSignalInit(NULL, 0);
/*
@@ -88,6 +96,8 @@ AuxiliaryProcessMainCommon(void)
*/
InitLocalDataChecksumState();
+ RESUME_INTERRUPTS();
+
/*
* Auxiliary processes don't run transactions, but they may need a
* resource owner anyway to manage buffer pins acquired outside
diff --git a/src/backend/postmaster/datachecksum_state.c b/src/backend/postmaster/datachecksum_state.c
index e26803bc501..0499cc005b3 100644
--- a/src/backend/postmaster/datachecksum_state.c
+++ b/src/backend/postmaster/datachecksum_state.c
@@ -263,8 +263,8 @@ static const ChecksumBarrierCondition checksum_barriers[7] =
{PG_DATA_CHECKSUM_INPROGRESS_OFF, PG_DATA_CHECKSUM_VERSION},
/*
- * If checksums are being enabled when launcher_exit is executed, state
- * is set to off since we cannot reach on at that point.
+ * If checksums are being enabled when launcher_exit is executed, state is
+ * set to off since we cannot reach on at that point.
*/
{PG_DATA_CHECKSUM_INPROGRESS_ON, PG_DATA_CHECKSUM_INPROGRESS_OFF},
};
@@ -1322,8 +1322,8 @@ DataChecksumsShmemInit(void *arg)
/*
* DatabaseExists
*
- * Scans the system catalog to check if a database with the given Oid exist
- * and returns true if it is found, else false.
+ * Scans the system catalog to check if a database with the given Oid exists
+ * and returns true if it is found else false.
*/
static bool
DatabaseExists(Oid dboid)
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 6f074013aa9..ecf78b9a986 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -756,6 +756,14 @@ InitPostgres(const char *in_dbname, Oid dboid,
*/
SharedInvalBackendInit(false);
+ /*
+ * Prevent consuming interrupts between setting ProcSignalInit and setting
+ * the initial local data checksum value. If a barrier is emitted, and
+ * absorbed, before local cached state is initialized the state transition
+ * can be invalid.
+ */
+ HOLD_INTERRUPTS();
+
ProcSignalInit(MyCancelKey, MyCancelKeyLength);
/*
@@ -776,6 +784,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
*/
InitLocalDataChecksumState();
+ RESUME_INTERRUPTS();
+
/*
* Also set up timeout handlers needed for backend operation. We need
* these in every case except bootstrap.
--
2.39.3 (Apple Git-146)
[application/octet-stream] v2-0005-Typo-and-spelling-fixups-for-online-checksums.patch (3.6K, ../[email protected]/6-v2-0005-Typo-and-spelling-fixups-for-online-checksums.patch)
download | inline diff:
From 4ceb94bb3dec9e7255c83878367c187eba76ac1a Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Tue, 28 Apr 2026 23:49:28 +0200
Subject: [PATCH v2 5/8] Typo and spelling fixups for online checksums
A collection of spelling, wording and punctuation fixups for the code
documentation from postcommit review.
Author: Daniel Gustafsson <[email protected]>
Discussion: https://postgr.es/m/xxx
---
src/backend/access/transam/xlog.c | 5 ++---
src/backend/postmaster/datachecksum_state.c | 12 ++++++------
2 files changed, 8 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 9b1a10dbd1c..7eb61158f26 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4893,9 +4893,8 @@ SetDataChecksumsOff(void)
else
{
/*
- * Ending up here implies that the checksums state is "inprogress-on"
- * or "inprogress-off" and we can transition directly to "off" from
- * there.
+ * Ending up here implies that the checksums state is "inprogress-off"
+ * and we can transition directly to "off" from there.
*/
SpinLockRelease(&XLogCtl->info_lck);
}
diff --git a/src/backend/postmaster/datachecksum_state.c b/src/backend/postmaster/datachecksum_state.c
index 0499cc005b3..27cdbb3ed1e 100644
--- a/src/backend/postmaster/datachecksum_state.c
+++ b/src/backend/postmaster/datachecksum_state.c
@@ -86,7 +86,7 @@
* failing to validate the data checksum on the page when reading it.
*
* When processing starts all backends belong to one of the below sets, with
- * one if Bd and Bi being empty:
+ * one of Bd and Bi being empty:
*
* Bg: Backend updating the global state and emitting the procsignalbarrier
* Bd: Backends in "off" state
@@ -286,7 +286,7 @@ typedef struct DataChecksumsStateStruct
int launch_cost_limit;
/*
- * Is a launcher process is currently running? This is set by the main
+ * Is a launcher process currently running? This is set by the main
* launcher process, after it has read the above launch_* parameters.
*/
bool launcher_running;
@@ -325,7 +325,7 @@ typedef struct DataChecksumsStateStruct
DataChecksumsWorkerResult success;
/*
- * tells the worker process whether it should also process the shared
+ * Tells the worker process whether it should also process the shared
* catalogs
*/
bool process_shared_catalogs;
@@ -820,7 +820,7 @@ ProcessDatabase(DataChecksumsWorkerDatabase *db)
{
/*
* If the worker managed to start, and stop, before we got to waiting
- * for it we can se a STOPPED status here without it being a failure.
+ * for it we can see a STOPPED status here without it being a failure.
*/
if (DataChecksumState->success == DATACHECKSUMSWORKER_SUCCESSFUL)
{
@@ -1323,7 +1323,7 @@ DataChecksumsShmemInit(void *arg)
* DatabaseExists
*
* Scans the system catalog to check if a database with the given Oid exists
- * and returns true if it is found else false.
+ * and returns true if it is found, else false.
*/
static bool
DatabaseExists(Oid dboid)
@@ -1486,7 +1486,7 @@ BuildRelationList(bool temp_relations, bool include_shared)
/*
* DataChecksumsWorkerMain
*
- * Main function for enabling checksums in a single database, This is the
+ * Main function for enabling checksums in a single database. This is the
* function set as the bgw_function_name in the dynamic background worker
* process initiated for each database by the worker launcher. After enabling
* data checksums in each applicable relation in the database, it will wait for
--
2.39.3 (Apple Git-146)
[application/octet-stream] v2-0006-Improve-handling-of-concurrent-checksum-requests.patch (9.8K, ../[email protected]/7-v2-0006-Improve-handling-of-concurrent-checksum-requests.patch)
download | inline diff:
From 6aaf7739d439c99ef7aa8dcd6c916792a2aa1eef Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Tue, 28 Apr 2026 23:49:30 +0200
Subject: [PATCH v2 6/8] Improve handling of concurrent checksum requests
When pg_{enable|disable}_data_checksums is called while checksums are
being enabled or disabled, the already running launcher is detected
and the new desired state is recorded. Processing will then pick up
the new state and change its operation to fulfill the new request.
If the same state is requested but updated cost values, the new cost
values will take effect on the next relation processed. The previous
coding had a complex logic of starting a new launcher for this, which
is now avoided with the shared mem structure instead used to signal
current processing.
This makes the logic more robust, and fixes a bug where the launcher
would erroneously revert back to the "off" state.
Author: Daniel Gustafsson <[email protected]>
Discussion: https://postgr.es/m/xxx
---
src/backend/access/transam/xlog.c | 33 +++++++-
src/backend/postmaster/datachecksum_state.c | 85 +++++++++++++++------
src/include/access/xlog.h | 2 +
3 files changed, 96 insertions(+), 24 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 7eb61158f26..757f309016c 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4682,10 +4682,41 @@ DataChecksumsNeedWrite(void)
LocalDataChecksumState == PG_DATA_CHECKSUM_INPROGRESS_OFF);
}
+
+bool
+DataChecksumsOff(void)
+{
+ bool ret;
+
+ SpinLockAcquire(&XLogCtl->info_lck);
+ ret = (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_OFF);
+ SpinLockRelease(&XLogCtl->info_lck);
+
+ return ret;
+}
+
+bool
+DataChecksumsOn(void)
+{
+ bool ret;
+
+ SpinLockAcquire(&XLogCtl->info_lck);
+ ret = (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_VERSION);
+ SpinLockRelease(&XLogCtl->info_lck);
+
+ return ret;
+}
+
bool
DataChecksumsInProgressOn(void)
{
- return LocalDataChecksumState == PG_DATA_CHECKSUM_INPROGRESS_ON;
+ bool ret;
+
+ SpinLockAcquire(&XLogCtl->info_lck);
+ ret = (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_ON);
+ SpinLockRelease(&XLogCtl->info_lck);
+
+ return ret;
}
/*
diff --git a/src/backend/postmaster/datachecksum_state.c b/src/backend/postmaster/datachecksum_state.c
index 27cdbb3ed1e..be1b0a2d926 100644
--- a/src/backend/postmaster/datachecksum_state.c
+++ b/src/backend/postmaster/datachecksum_state.c
@@ -235,7 +235,7 @@ typedef struct ChecksumBarrierCondition
int to;
} ChecksumBarrierCondition;
-static const ChecksumBarrierCondition checksum_barriers[7] =
+static const ChecksumBarrierCondition checksum_barriers[9] =
{
/*
* Disabling checksums: If checksums are currently enabled, disabling must
@@ -267,6 +267,13 @@ static const ChecksumBarrierCondition checksum_barriers[7] =
* set to off since we cannot reach on at that point.
*/
{PG_DATA_CHECKSUM_INPROGRESS_ON, PG_DATA_CHECKSUM_INPROGRESS_OFF},
+
+ /*
+ * Transitions that can happen when a new request is made while another is
+ * currently being processed.
+ */
+ {PG_DATA_CHECKSUM_INPROGRESS_OFF, PG_DATA_CHECKSUM_INPROGRESS_ON},
+ {PG_DATA_CHECKSUM_OFF, PG_DATA_CHECKSUM_INPROGRESS_OFF},
};
/*
@@ -377,6 +384,15 @@ const ShmemCallbacks DataChecksumsShmemCallbacks = {
.init_fn = DataChecksumsShmemInit,
};
+#define CHECK_FOR_ABORT_REQUEST() \
+ do { \
+ LWLockAcquire(DataChecksumsWorkerLock, LW_SHARED); \
+ if (DataChecksumState->launch_operation != operation) \
+ abort_requested = true; \
+ LWLockRelease(DataChecksumsWorkerLock); \
+ } while (0)
+
+
/*****************************************************************************
* Functionality for manipulating the data checksum state in the cluster
*/
@@ -566,7 +582,6 @@ StartDataChecksumsWorkerLauncher(DataChecksumsWorkerOperation op,
BackgroundWorker bgw;
BackgroundWorkerHandle *bgw_handle;
bool running;
- DataChecksumsWorkerOperation launcher_running_op;
#ifdef USE_ASSERT_CHECKING
/* The cost delay settings have no effect when disabling */
@@ -585,8 +600,6 @@ StartDataChecksumsWorkerLauncher(DataChecksumsWorkerOperation op,
/* Is the launcher already running? If so, what is it doing? */
running = DataChecksumState->launcher_running;
- if (running)
- launcher_running_op = DataChecksumState->operation;
LWLockRelease(DataChecksumsWorkerLock);
@@ -603,13 +616,17 @@ StartDataChecksumsWorkerLauncher(DataChecksumsWorkerOperation op,
* the launcher has had a chance to start up, we still end up launching it
* twice. That's OK, the second invocation will see that a launcher is
* already running and exit quickly.
- *
- * TODO: We could optimize here and skip launching the launcher, if we are
- * already in the desired state, i.e. if the checksums are already enabled
- * and you call pg_enable_data_checksums().
*/
if (!running)
{
+ if ((op == ENABLE_DATACHECKSUMS && DataChecksumsOn()) ||
+ (op == DISABLE_DATACHECKSUMS && DataChecksumsOff()))
+ {
+ ereport(LOG,
+ errmsg("data checksums already in desired state, exiting"));
+ return;
+ }
+
/*
* Prepare the BackgroundWorker and launch it.
*/
@@ -631,9 +648,8 @@ StartDataChecksumsWorkerLauncher(DataChecksumsWorkerOperation op,
}
else
{
- if (launcher_running_op == op)
- ereport(ERROR,
- errmsg("data checksum processing already running"));
+ ereport(LOG,
+ errmsg("data checksum processing already running"));
}
}
@@ -1024,11 +1040,8 @@ WaitForAllTransactionsToFinish(void)
errhint("Data checksums processing must be restarted manually after cluster restart."));
CHECK_FOR_INTERRUPTS();
+ CHECK_FOR_ABORT_REQUEST();
- LWLockAcquire(DataChecksumsWorkerLock, LW_SHARED);
- if (DataChecksumState->launch_operation != operation)
- abort_requested = true;
- LWLockRelease(DataChecksumsWorkerLock);
if (abort_requested)
break;
}
@@ -1212,7 +1225,9 @@ ProcessAllDatabases(void)
int cumulative_total = 0;
/* Set up so first run processes shared catalogs, not once in every db */
+ LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
DataChecksumState->process_shared_catalogs = true;
+ LWLockRelease(DataChecksumsWorkerLock);
/* Get a list of all databases to process */
WaitForAllTransactionsToFinish();
@@ -1288,7 +1303,9 @@ ProcessAllDatabases(void)
* When one database has completed, it will have done shared catalogs
* so we don't have to process them again.
*/
+ LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
DataChecksumState->process_shared_catalogs = false;
+ LWLockRelease(DataChecksumsWorkerLock);
}
FreeDatabaseList(DatabaseList);
@@ -1538,7 +1555,6 @@ DataChecksumsWorkerMain(Datum arg)
* implementation detail and care should be taken to avoid it bleeding
* through to the user to avoid confusion.
*/
- Assert(DataChecksumState->operation == ENABLE_DATACHECKSUMS);
VacuumCostDelay = DataChecksumState->cost_delay;
VacuumCostLimit = DataChecksumState->cost_limit;
VacuumCostActive = (VacuumCostDelay > 0);
@@ -1574,8 +1590,6 @@ DataChecksumsWorkerMain(Datum arg)
rels_done = 0;
foreach_oid(reloid, RelationList)
{
- CHECK_FOR_INTERRUPTS();
-
if (!ProcessSingleRelationByOid(reloid, strategy))
{
aborted = true;
@@ -1584,12 +1598,38 @@ DataChecksumsWorkerMain(Datum arg)
pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_RELS_DONE,
++rels_done);
+ CHECK_FOR_INTERRUPTS();
+ CHECK_FOR_ABORT_REQUEST();
+
+ if (abort_requested)
+ break;
+
+ /* Check if the cost settings changed during runtime */
+ LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
+ if ((DataChecksumState->launch_cost_delay != DataChecksumState->cost_delay)
+ || (DataChecksumState->launch_cost_limit != DataChecksumState->cost_limit))
+ {
+ VacuumCostDelay = DataChecksumState->launch_cost_delay;
+ VacuumCostLimit = DataChecksumState->launch_cost_limit;
+ VacuumCostActive = (VacuumCostDelay > 0);
+
+ FreeAccessStrategy(strategy);
+ strategy = GetAccessStrategy(BAS_VACUUM);
+ DataChecksumState->cost_delay = DataChecksumState->launch_cost_delay;
+ DataChecksumState->cost_limit = DataChecksumState->launch_cost_limit;
+ }
+ LWLockRelease(DataChecksumsWorkerLock);
+
}
+
list_free(RelationList);
+ FreeAccessStrategy(strategy);
- if (aborted)
+ if (aborted || abort_requested)
{
+ LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
DataChecksumState->success = DATACHECKSUMSWORKER_ABORTED;
+ LWLockRelease(DataChecksumsWorkerLock);
ereport(DEBUG1,
errmsg("data checksum processing aborted in database OID %u",
dboid));
@@ -1654,15 +1694,14 @@ DataChecksumsWorkerMain(Datum arg)
3000,
WAIT_EVENT_CHECKSUM_ENABLE_TEMPTABLE_WAIT);
- LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
- aborted = DataChecksumState->launch_operation != operation;
- LWLockRelease(DataChecksumsWorkerLock);
-
CHECK_FOR_INTERRUPTS();
+ CHECK_FOR_ABORT_REQUEST();
if (aborted || abort_requested)
{
+ LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
DataChecksumState->success = DATACHECKSUMSWORKER_ABORTED;
+ LWLockRelease(DataChecksumsWorkerLock);
ereport(LOG,
errmsg("data checksum processing aborted in database OID %u",
dboid));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 437b4f32349..4dd98624204 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -249,6 +249,8 @@ extern uint64 GetSystemIdentifier(void);
extern char *GetMockAuthenticationNonce(void);
extern bool DataChecksumsNeedWrite(void);
extern bool DataChecksumsNeedVerify(void);
+extern bool DataChecksumsOn(void);
+extern bool DataChecksumsOff(void);
extern bool DataChecksumsInProgressOn(void);
extern void SetDataChecksumsOnInProgress(void);
extern void SetDataChecksumsOn(void);
--
2.39.3 (Apple Git-146)
[application/octet-stream] v2-0007-Improve-database-detection-logic-in-datachecksums.patch (2.3K, ../[email protected]/8-v2-0007-Improve-database-detection-logic-in-datachecksums.patch)
download | inline diff:
From 20e1c46fb33300954e102b37f13753d71180a185 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Tue, 28 Apr 2026 23:50:01 +0200
Subject: [PATCH v2 7/8] Improve database detection logic in
datachecksumsworker
The worker need to know whether a database which failed checksum
processing still exists, or has been dropped. This improves the
detection logic by checking for being partially dropped.
Author: Daniel Gustafsson <[email protected]>
Discussion: https://postgr.es/m/xxx
---
src/backend/postmaster/datachecksum_state.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/src/backend/postmaster/datachecksum_state.c b/src/backend/postmaster/datachecksum_state.c
index be1b0a2d926..352a0f6139e 100644
--- a/src/backend/postmaster/datachecksum_state.c
+++ b/src/backend/postmaster/datachecksum_state.c
@@ -854,8 +854,7 @@ ProcessDatabase(DataChecksumsWorkerDatabase *db)
/*
* Heuristic to see if the database was dropped, and if it was we can
- * treat it as not an error, else treat as fatal and error out. TODO:
- * this could probably be improved with a tighter check.
+ * treat it as not an error, else treat as fatal and error out.
*/
if (DatabaseExists(db->dboid))
return DATACHECKSUMSWORKER_FAILED;
@@ -1340,7 +1339,9 @@ DataChecksumsShmemInit(void *arg)
* DatabaseExists
*
* Scans the system catalog to check if a database with the given Oid exists
- * and returns true if it is found, else false.
+ * and returns true if it is found and valid, else false. Note, we cannot use
+ * database_is_invalid_oid here as it will ERROR out, and we want to gracefully
+ * handle errors.
*/
static bool
DatabaseExists(Oid dboid)
@@ -1350,6 +1351,7 @@ DatabaseExists(Oid dboid)
SysScanDesc scan;
bool found;
HeapTuple tuple;
+ Form_pg_database pg_database_tuple;
StartTransactionCommand();
@@ -1363,6 +1365,14 @@ DatabaseExists(Oid dboid)
tuple = systable_getnext(scan);
found = HeapTupleIsValid(tuple);
+ /* If the Oid exists, ensure that it's not partially dropped */
+ if (found)
+ {
+ pg_database_tuple = (Form_pg_database) GETSTRUCT(tuple);
+ if (database_is_invalid_form(pg_database_tuple))
+ found = false;
+ }
+
systable_endscan(scan);
table_close(rel, AccessShareLock);
--
2.39.3 (Apple Git-146)
[application/octet-stream] v2-0008-Fix-data_checksum-GUC-show_hook.patch (974B, ../[email protected]/9-v2-0008-Fix-data_checksum-GUC-show_hook.patch)
download | inline diff:
From 59585a0a316f569d847f37addd7022e045d8bfcb Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Tue, 28 Apr 2026 23:50:03 +0200
Subject: [PATCH v2 8/8] Fix data_checksum GUC show_hook
Commit f19c0eccae erroneously omitted the show_hook for the
data_checksum GUC.
Author: Daniel Gustafsson <[email protected]>
Discussion: https://postgr.es/m/xxx
---
src/backend/utils/misc/guc_parameters.dat | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 83af594d4af..afaa058b046 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -585,6 +585,7 @@
variable => 'data_checksums',
boot_val => 'PG_DATA_CHECKSUM_OFF',
options => 'data_checksums_options',
+ show_hook => 'show_data_checksums',
},
# Can't be set by ALTER SYSTEM as it can lead to recursive definition
--
2.39.3 (Apple Git-146)
[text/plain] 0006_bug_test.bash.txt (6.9K, ../[email protected]/10-0006_bug_test.bash.txt)
download | inline:
#!/usr/bin/env bash
set -euo pipefail
prefix="${PGPREFIX:-$PWD/tmp_install/usr/local/pgsql}"
bindir="$prefix/bin"
sharedir="$prefix/share"
libdir="$prefix/lib"
rows="${ROWS:-80000}"
followup_calls="${FOLLOWUP_CALLS:-5}"
followup_sleep="${FOLLOWUP_SLEEP:-0.2}"
post_run_sleep="${POST_RUN_SLEEP:-8}"
if [[ ! -x "$bindir/initdb" || ! -x "$bindir/pg_ctl" || ! -x "$bindir/psql" ]]; then
cat >&2 <<EOF
could not find PostgreSQL binaries under: $bindir
Set PGPREFIX to the install prefix, or install into:
$PWD/tmp_install/usr/local/pgsql
EOF
exit 1
fi
if [[ ! -f "$sharedir/extension/test_checksums.control" || ! -f "$libdir/test_checksums.so" ]]; then
cat >&2 <<EOF
test_checksums is not installed under: $prefix
Install the test modules first, for example:
make -s -C src/test/modules/injection_points install DESTDIR="$PWD/tmp_install"
make -s -C src/test/modules/test_checksums install DESTDIR="$PWD/tmp_install"
EOF
exit 1
fi
export PATH="$bindir:$PATH"
export LD_LIBRARY_PATH="$libdir${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
base="/tmp/pg-dc-launcher-race-$$"
datadir="$base/data"
logfile="$base/server.log"
port="${PGPORT:-$((55432 + RANDOM % 1000))}"
started=0
mkdir -p "$base"
cleanup()
{
if [[ "$started" -eq 1 ]]; then
pg_ctl -D "$datadir" -m immediate -w stop >"$base/pg_ctl_stop.out" 2>&1 || true
fi
echo "repro artifacts left in: $base"
}
trap cleanup EXIT
initdb --no-data-checksums --auth trust --no-sync --no-instructions -D "$datadir" >"$base/initdb.out"
cat >>"$datadir/postgresql.conf" <<EOF
port = $port
listen_addresses = ''
unix_socket_directories = '$base'
max_worker_processes = 20
log_line_prefix = '%m [%p] '
log_min_messages = debug1
EOF
pg_ctl -D "$datadir" -l "$logfile" -w start >"$base/pg_ctl_start.out"
started=1
psql_args=(-h "$base" -p "$port" -v ON_ERROR_STOP=1 -X postgres)
count_log()
{
grep -F -c "$1" "$logfile" 2>/dev/null || true
}
wait_for_log_count()
{
local needle="$1"
local expected="$2"
local timeout_seconds="$3"
local end=$((SECONDS + timeout_seconds))
local count
while (( SECONDS < end )); do
count=$(count_log "$needle")
if (( count >= expected )); then
return 0
fi
sleep 0.1
done
count=$(count_log "$needle")
echo "timed out waiting for at least $expected occurrence(s) of: $needle" >&2
echo "saw $count occurrence(s)" >&2
return 1
}
psql "${psql_args[@]}" -c "CREATE EXTENSION test_checksums;" >"$base/setup-extension.out"
psql "${psql_args[@]}" -c "CREATE TABLE t AS SELECT g, repeat('x', 2000) AS payload FROM generate_series(1, $rows) g;" >"$base/setup-table.out"
psql "${psql_args[@]}" -c "CHECKPOINT;" >"$base/checkpoint.out"
# Delay the two initial pg_enable_data_checksums() callers at the launcher
# registration entry point, so both callers enter the launch path together.
psql "${psql_args[@]}" -c "SELECT dcw_inject_startup_delay(true);" >"$base/attach-startup-delay.out"
# Delay launcher main before it claims DataChecksumState->launcher_running. This
# makes the same race window deterministic instead of relying on scheduler luck.
psql "${psql_args[@]}" -c "SELECT dcw_inject_launcher_delay(true);" >"$base/attach-launcher-delay.out"
# Delay the final transition to on, keeping the winning launcher active long
# enough for follow-up pg_enable_data_checksums() calls to probe the state.
psql "${psql_args[@]}" -c "SELECT dcw_inject_delay_barrier(true);" >"$base/attach-final-delay.out"
psql "${psql_args[@]}" -c "SELECT pg_enable_data_checksums(20, 1);" >"$base/enable1.out" 2>&1 &
pid1=$!
psql "${psql_args[@]}" -c "SELECT pg_enable_data_checksums(20, 1);" >"$base/enable2.out" 2>&1 &
pid2=$!
wait "$pid1" || true
wait "$pid2" || true
# Later pg_enable_data_checksums() calls should not be delayed at the SQL entry
# point; they need to probe the shared launcher_running state promptly.
psql "${psql_args[@]}" -c "SELECT dcw_inject_startup_delay(false);" >"$base/detach-startup-delay.out"
wait_for_log_count 'background worker "datachecksums launcher" started' 2 15
wait_for_log_count 'background worker "datachecksums launcher" already running, exiting' 1 10
wait_for_log_count 'initiating data checksum processing in database "postgres"' 1 10
# Stop delaying later launchers. Follow-up calls are fired immediately after
# the redundant launcher has exited, while the owning launcher is still
# processing or waiting at the final checksum-on barrier.
psql "${psql_args[@]}" -c "SELECT dcw_inject_launcher_delay(false);" >"$base/detach-launcher-delay.out"
# A single follow-up call is often enough after the redundant launcher exits.
# Use a small handful by default to avoid depending on exact worker timing; set
# FOLLOWUP_CALLS=1 for the smallest variant.
pids=()
for _ in $(seq 1 "$followup_calls"); do
psql "${psql_args[@]}" -c "SELECT pg_enable_data_checksums(20, 1);" >"$base/enable-followup-$_.out" 2>&1 &
pids+=("$!")
sleep "$followup_sleep"
done
for pid in "${pids[@]}"; do
wait "$pid" || true
done
sleep "$post_run_sleep"
echo
echo "--- relevant server log lines ---"
grep -E 'datachecksums launcher|data checksum processing already running|data checksums already in desired state|enabling data checksums requested|initiating data checksum processing|cannot set data checksums|data checksums are now|data checksum processing was aborted|disabling data checksums requested' "$logfile" || true
echo
echo "--- summary ---"
launcher_workers_started=$(count_log 'background worker "datachecksums launcher" started')
checksum_processing_starts=$(count_log 'enabling data checksums requested')
already_running_responses=$(count_log 'data checksum processing already running')
checksum_state_warnings=$(count_log 'cannot set data checksums')
echo "launcher workers started: $launcher_workers_started"
echo "checksum processing starts: $checksum_processing_starts"
echo "already-running responses: $already_running_responses"
echo "checksum state warnings: $checksum_state_warnings"
echo "final data_checksums setting: $(psql "${psql_args[@]}" -At -c 'SHOW data_checksums;' || true)"
echo
if (( checksum_processing_starts > 1 )); then
echo "BUG REPRODUCED: more than one launcher started checksum processing."
else
echo "BUG NOT REPRODUCED: only one launcher started checksum processing."
exit 1
fi
echo
cat <<'EOF'
Expected result on the unfixed patchset:
After the redundant launcher exits, a follow-up call can start another
datachecksums launcher while the original launcher is still active. The log
shows more than one "enabling data checksums requested" line. Depending on
timing, it may also show "cannot set data checksums to \"on\"...".
Expected result with the launcher_exit() ownership fix:
Two launcher workers may start, but the redundant one only logs "already
running, exiting". While the owning launcher is still active, follow-up
calls log "data checksum processing already running"; after it completes,
they may log "data checksums already in desired state, exiting". No
additional launcher starts checksum processing.
EOF
view thread (60+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Changing the state of data checksums in a running cluster
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox