Subject: Re: Changing the state of data checksums in a running cluster

From: Tomas Vondra <[email-masked]>
Date: Tue, 26 Nov 2024 23:07:12 +0100

Hi,

I spent a bit more time doing some testing on the last version of the
patch from [1]. And I ran into this assert in PostmasterStateMachine()
when stopping the cluster:

  /* All types should be included in targetMask or remainMask */
  Assert((remainMask.mask | targetMask.mask) == BTYPE_MASK_ALL.mask);

At first I was puzzled as this happens on every shutdown, but that's
because these checks were introduced by a78af0427015 a week ago. So it's
more a matter of rebasing.

However, I also noticed the progress monitoring does not really work. I
get stuff like this:

    + psql -x test -c 'select * from pg_stat_progress_data_checksums'
    -[ RECORD 1 ]---------------------+---------
    pid                               | 56811
    datid                             | 0
    datname                           |
    phase                             | enabling
    databases_total                   | 4
    relations_total                   |
    databases_processed               | 0
    relations_processed               | 0
    databases_current                 | 16384
    relation_current                  | 0
    relation_current_blocks           | 0
    relation_current_blocks_processed | 0

But I've never seen any of the "processed" fields to be non-zero (and
relations is even NULL), and the same thing applies to relation_. Also
what is the datid/datname about? It's empty, not mentioned in sgml docs,
and we already have databases_current ...

The message [2] from 10/08 says:

> I did remove parts of the progress reporting for now since it can't be
> used from the dynamic backgroundworker it seems.  I need to regroup
> and figure out a better way there, but I wanted to address your above
> find sooner rather than wait for that.

And I guess that would explain why some of the fields are not updated.
But then the later patch versions seem to imply there are no outstanding
issues / missing stuff.

regards

[1]
https://www.postgresql.org/message-id/CA226DE1-DC9A-4675-A83C-32270C473F0B%40yesql.se

[2]
https://www.postgresql.org/message-id/DD25705F-E75F-4DCA-B49A-5578F4F55D94%40yesql.se

-- 
Tomas Vondra

---

From: Michael Paquier <[email-masked]>
Date: Wed, 11 Dec 2024 13:47:08 +0900

> more a matter of rebasing.

Looks like the CI is not really happy about this point..  (Please make
sure to refresh the patch status after a review.)
--
Michael

---

From: Tomas Vondra <[email-masked]>
Date: Sun, 9 Mar 2025 18:57:52 +0100

Here's a rebased version of the patch series, addressing the issues I've
pointed out in the last round of reviews. I've kept the changes in
separate patches for clarity, but it should be squashed into a single
patch in the end.

1) v20250309-0001-Online-enabling-and-disabling-of-data-chec.patch
------------------------------------------------------------------

Original patch, rebased, resolving merge conflicts.

2) v20250309-0002-simple-post-rebase-fixes.patch
------------------------------------------------

A couple minor fixes, addressing test failures due to stuff committed
since the previous patch version. Mostly mechanical, the main change is
I don't think the pgstat_bestart() call is necessary. Or is it?

3) v20250309-0003-sync-the-data_checksums-GUC-with-the-local.patch
------------------------------------------------------------------

This is the main change, fixing failures in 002_actions.pl - the short
version is that test does "-C data_checksums", but AFAICS that does not
quite work because it does not call show_data_checksums() that early,
and instead just grabs the variable backing the GUC. Which may be out of
sync, so this patch fixes that by updating them both.

That fixes the issue, but it's it a bit strange we now have three places
tracking the state of data checksums? We have data_checksum_version in
the control file, and then data_checksums and LocalDataChecksumVersion
in the backends.

Would it be possible to "unify" the latter two? That would also mean we
don't have the duplicate constants like PG_DATA_CHECKSUM_VERSION and
DATA_CHECKSUM_VERSION. Or why do we need that?

4) v20250309-0004-make-progress-reporting-work.patch
----------------------------------------------------

The progress reporting got "mostly disabled" in an earlier version, due
to issues with the bgworkers. AFAICS the issue is the "status" row can
be updated only by a single process, which does not quite work with the
launcher + per-db workers architecture.

I've considered a couple different approaches:

a) updating the status only from the launcher

This is mostly what CREATE INDEX does with parallel builds, and there
it's mostly sufficient. But for checksums it'd mean we only have the
number of databases to process/done, and that seems unsatisfactory,
considering large clusters often have only a single large database. So
not good enough, IMHO.

b) allowing workers to update the status row, created by the launcher

I guess this would be better, we'd know the relations/blocks counts. And
I haven't tried coding this, but there would need to be some locking so
that the workers don't overwrite increments from other workers, etc.

But I don't think it'd work nicely with parallel per-db workers (which
we don't do now, but we might).

c) having one status entry per worker

I ended up doing this, it didn't require any changes to the progress
infrastructure, and it will work naturally even with multiple workers.
There will always be one row for launcher status (which only has the
number of databases total/done), and then one row per worker, with
database-level info (datid, datname, #relations, #blocks).

I removed the "DONE" phase, because that's right before the launcher
exists, and I don't think we have that for similar cases. And I added
"waiting on checkpoint" state, because that's often a long wait when the
launcher seems to do nothing, so it seems useful to communicate the
reason for that wait.

5) v20250309-0005-update-docs.patch
-----------------------------------

Minor tweaks to docs, to reflect the changes to the progress reporting
changes, and also some corrections (no resume after restart, ...).

So far this passed all my tests - both chekc-world and stress testing
(no failures / assert crashes, ...). One thing that puzzles me is I
wasn't able to reproduce the failures reported in [1] - not even with
just the rebase + minimal fixes (0001 + 0002).  My best theory is this
is somehow machine-specific, and my laptop is too slow or something.
I'll try with the machine I used before once it gets available.

regards

[1]
https://www.postgresql.org/message-id/e4dbcb2c-e04a-4ba2-bff0-8d979f55960e%40vondra.me

-- 
Tomas Vondra
From c8e48010604a0042d22d1b0d3b31c335277b1ceb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 19:21:22 +0100
Subject: [PATCH v20250309 1/5] Online enabling and disabling of data checksums

This allows data checksums to be enabled, or disabled, in a running
cluster without restricting access to the cluster during processing.

Data checksums could prior to this only be enabled during initdb or
when the cluster is offline using the pg_checksums app. This commit
introduce functionality to enable, or disable, data checksums while
the cluster is running regardless of how it was initialized.

A background worker launcher process is responsible for launching a
dynamic per-database background worker which will mark all buffers
dirty for all relation with storage in order for them to have data
checksums calcuated on write.  Once all relations in all databases
have been processed, the data_checksums state will be set to on and
the cluster will at that point be identical to one which had data
checksums enabled during initialization or via offline processing.

When data checksums are being enabled, concurrent I/O operations
from backends other than the data checksums worker will write the
checksums but not verify them on reading.  Only when all backends
have absorbed the procsignalbarrier for setting data_checksums to
on will they also start verifying checksums on reading.  The same
process is repeated during disabling; all backends write checksums
but do not verify them until the barrier for setting the state to
off has been absorbed by all.  This in-progress state is used to
ensure there are no false negatives (or positives) due to reading
a checksum which is not in sync with the page.

This work is based on an earlier version of this patch which was
reviewed by among others Heikki Linnakangas, Robert Haas, Andres
Freund, Tomas Vondra, Michael Banck and Andrey Borodin.

Author: Daniel Gustafsson <[email-masked]>
Author: Magnus Hagander <[email-masked]>
Reviewed-by: Tomas Vondra <[email-masked]>
Discussion: https://postgr.es/m/CABUevExz9hUUOLnJVr2kpw9Cx=[email-masked]
Discussion: https://postgr.es/m/[email-masked]
Discussion: https://postgr.es/m/[email-masked]
---
 doc/src/sgml/func.sgml                        |   71 +
 doc/src/sgml/glossary.sgml                    |   23 +
 doc/src/sgml/monitoring.sgml                  |  207 ++-
 doc/src/sgml/ref/pg_checksums.sgml            |    6 +
 doc/src/sgml/wal.sgml                         |   57 +-
 src/backend/access/rmgrdesc/xlogdesc.c        |   24 +
 src/backend/access/transam/xlog.c             |  450 +++++-
 src/backend/access/transam/xlogfuncs.c        |   41 +
 src/backend/backup/basebackup.c               |    6 +-
 src/backend/catalog/system_functions.sql      |   11 +
 src/backend/catalog/system_views.sql          |   21 +
 src/backend/postmaster/Makefile               |    1 +
 src/backend/postmaster/bgworker.c             |    7 +
 src/backend/postmaster/datachecksumsworker.c  | 1376 +++++++++++++++++
 src/backend/postmaster/meson.build            |    1 +
 src/backend/replication/logical/decode.c      |    1 +
 src/backend/storage/ipc/ipci.c                |    4 +
 src/backend/storage/ipc/procsignal.c          |   13 +
 src/backend/storage/page/README               |    4 +-
 src/backend/storage/page/bufpage.c            |    6 +-
 src/backend/utils/activity/pgstat_io.c        |    2 +
 .../utils/activity/wait_event_names.txt       |    3 +
 src/backend/utils/adt/pgstatfuncs.c           |    8 +-
 src/backend/utils/init/miscinit.c             |    9 +-
 src/backend/utils/init/postinit.c             |    7 +-
 src/backend/utils/misc/guc_tables.c           |   32 +-
 src/bin/pg_checksums/pg_checksums.c           |    2 +-
 src/bin/pg_upgrade/controldata.c              |    9 +
 src/include/access/xlog.h                     |   16 +-
 src/include/access/xlog_internal.h            |    7 +
 src/include/catalog/pg_control.h              |    1 +
 src/include/catalog/pg_proc.dat               |   17 +
 src/include/commands/progress.h               |   18 +
 src/include/miscadmin.h                       |    4 +
 src/include/postmaster/datachecksumsworker.h  |   31 +
 src/include/storage/bufpage.h                 |   10 +
 src/include/storage/checksum.h                |    8 +
 src/include/storage/lwlocklist.h              |    1 +
 src/include/storage/proc.h                    |    5 +-
 src/include/storage/procsignal.h              |    5 +
 src/include/utils/backend_progress.h          |    1 +
 src/test/Makefile                             |   11 +-
 src/test/checksum/.gitignore                  |    2 +
 src/test/checksum/Makefile                    |   23 +
 src/test/checksum/README                      |   22 +
 src/test/checksum/meson.build                 |   15 +
 src/test/checksum/t/001_basic.pl              |   88 ++
 src/test/checksum/t/002_restarts.pl           |   92 ++
 src/test/checksum/t/003_standby_restarts.pl   |  131 ++
 src/test/checksum/t/004_offline.pl            |  101 ++
 src/test/meson.build                          |    1 +
 src/test/perl/PostgreSQL/Test/Cluster.pm      |   34 +
 src/test/regress/expected/rules.out           |   28 +
 src/tools/pgindent/typedefs.list              |    6 +
 54 files changed, 3031 insertions(+), 49 deletions(-)
 create mode 100644 src/backend/postmaster/datachecksumsworker.c
 create mode 100644 src/include/postmaster/datachecksumsworker.h
 create mode 100644 src/test/checksum/.gitignore
 create mode 100644 src/test/checksum/Makefile
 create mode 100644 src/test/checksum/README
 create mode 100644 src/test/checksum/meson.build
 create mode 100644 src/test/checksum/t/001_basic.pl
 create mode 100644 src/test/checksum/t/002_restarts.pl
 create mode 100644 src/test/checksum/t/003_standby_restarts.pl
 create mode 100644 src/test/checksum/t/004_offline.pl

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 51dd8ad6571..a09843e4ecf 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -29865,6 +29865,77 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
 
   </sect2>
 
+  <sect2 id="functions-admin-checksum">
+   <title>Data Checksum Functions</title>
+
+   <para>
+    The functions shown in <xref linkend="functions-checksums-table" /> can
+    be used to enable or disable data checksums in a running cluster.
+    See <xref linkend="checksums" /> for details.
+   </para>
+
+   <table id="functions-checksums-table">
+    <title>Data Checksum Functions</title>
+    <tgroup cols="1">
+     <thead>
+      <row>
+       <entry role="func_table_entry"><para role="func_signature">
+        Function
+       </para>
+       <para>
+        Description
+       </para></entry>
+      </row>
+     </thead>
+
+     <tbody>
+      <row>
+       <entry role="func_table_entry"><para role="func_signature">
+        <indexterm>
+         <primary>pg_enable_data_checksums</primary>
+        </indexterm>
+        <function>pg_enable_data_checksums</function> ( <optional><parameter>cost_delay</parameter> <type>int</type>, <parameter>cost_limit</parameter> <type>int</type></optional> )
+        <returnvalue>void</returnvalue>
+       </para>
+       <para>
+        Initiates data checksums for the cluster. This will switch the data
+        checksums mode to <literal>inprogress-on</literal> as well as start a
+        background worker that will process all pages in the database and
+        enable checksums on them. When all data pages have had checksums
+        enabled, the cluster will automatically switch data checksums mode to
+        <literal>on</literal>.
+       </para>
+       <para>
+        If <parameter>cost_delay</parameter> and <parameter>cost_limit</parameter> are
+        specified, the speed of the process is throttled using the same principles as
+        <link linkend="runtime-config-resource-vacuum-cost">Cost-based Vacuum Delay</link>.
+       </para></entry>
+      </row>
+
+      <row>
+       <entry role="func_table_entry"><para role="func_signature">
+        <indexterm>
+         <primary>pg_disable_data_checksums</primary>
+        </indexterm>
+        <function>pg_disable_data_checksums</function> ()
+        <returnvalue>void</returnvalue>
+       </para>
+       <para>
+        Disables data checksum validation and calculation for the cluster. This
+        will switch the data checksum mode to <literal>inprogress-off</literal>
+        while data checksums are being disabled. When all active backends have
+        stopped validating data checksums, the data checksum mode will be
+        changed to <literal>off</literal>.  At this point the data pages will
+        still have checksums recorded but they are not updated when pages are
+        modified.
+       </para></entry>
+      </row>
+     </tbody>
+    </tgroup>
+   </table>
+
+  </sect2>
+
   <sect2 id="functions-admin-dbobject">
    <title>Database Object Management Functions</title>
 
diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml
index c0f812e3f5e..547e8586d4b 100644
--- a/doc/src/sgml/glossary.sgml
+++ b/doc/src/sgml/glossary.sgml
@@ -159,6 +159,8 @@
      (but not the autovacuum workers),
      the <glossterm linkend="glossary-background-writer">background writer</glossterm>,
      the <glossterm linkend="glossary-checkpointer">checkpointer</glossterm>,
+     the <glossterm linkend="glossary-data-checksums-worker">data checksums worker</glossterm>,
+     the <glossterm linkend="glossary-data-checksums-worker-launcher">data checksums worker launcher</glossterm>,
      the <glossterm linkend="glossary-logger">logger</glossterm>,
      the <glossterm linkend="glossary-startup-process">startup process</glossterm>,
      the <glossterm linkend="glossary-wal-archiver">WAL archiver</glossterm>,
@@ -548,6 +550,27 @@
    </glossdef>
   </glossentry>
 
+  <glossentry id="glossary-data-checksums-worker">
+   <glossterm>Data Checksums Worker</glossterm>
+   <glossdef>
+    <para>
+     An <glossterm linkend="glossary-auxiliary-proc">auxiliary process</glossterm>
+     which enables or disables data checksums in a specific database.
+    </para>
+   </glossdef>
+  </glossentry>
+
+  <glossentry id="glossary-data-checksums-worker-launcher">
+   <glossterm>Data Checksums Worker Launcher</glossterm>
+   <glossdef>
+    <para>
+     An <glossterm linkend="glossary-auxiliary-proc">auxiliary process</glossterm>
+     which starts <glossterm linkend="glossary-data-checksums-worker"> processes</glossterm>
+     for each database.
+    </para>
+   </glossdef>
+  </glossentry>
+
   <glossentry id="glossary-db-cluster">
    <glossterm>Database cluster</glossterm>
    <glossdef>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 16646f560e8..4a6ef5f1605 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3497,8 +3497,9 @@ description | Waiting for a newly initialized WAL file to reach durable storage
       </para>
       <para>
        Number of data page checksum failures detected in this
-       database (or on a shared object), or NULL if data checksums are
-       disabled.
+       database (or on a shared object).
+       Detected failures are reported regardless of the
+       <xref linkend="guc-data-checksums"/> setting.
       </para></entry>
      </row>
 
@@ -3508,8 +3509,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
       </para>
       <para>
        Time at which the last data page checksum failure was detected in
-       this database (or on a shared object), or NULL if data checksums are
-       disabled.
+       this database (or on a shared object). Last failure is reported
+       regardless of the <xref linkend="guc-data-checksums"/> setting.
       </para></entry>
      </row>
 
@@ -6793,6 +6794,204 @@ FROM pg_stat_get_backend_idset() AS backendid;
 
  </sect2>
 
+ <sect2 id="data-checksum-progress-reporting">
+  <title>Data Checksum Progress Reporting</title>
+
+  <indexterm>
+   <primary>pg_stat_progress_data_checksums</primary>
+  </indexterm>
+
+  <para>
+   When data checksums are being enabled on a running cluster, the
+   <structname>pg_stat_progress_data_checksums</structname> view will contain
+   a row for each background worker which is currently calculating checksums
+   for the data pages.
+  </para>
+
+  <table id="pg-stat-progress-data-checksums-view" xreflabel="pg_stat_progress_data_checksums">
+   <title><structname>pg_stat_progress_data_checksums</structname> View</title>
+   <tgroup cols="1">
+    <thead>
+     <row>
+      <entry role="catalog_table_entry">
+       <para role="column_definition">
+        Column Type
+       </para>
+       <para>
+        Description>
+       </para>
+      </entry>
+     </row>
+    </thead>
+
+    <tbody>
+     <row>
+      <entry role="catalog_table_entry">
+       <para role="column_definition">
+        <structfield>pid</structfield> <type>integer</type>
+       </para>
+       <para>
+        Process ID of a datachecksumworker process.
+       </para>
+      </entry>
+     </row>
+
+     <row>
+      <entry role="catalog_table_entry">
+       <para role="column_definition">
+        <structfield>phase</structfield> <type>text</type>
+       </para>
+       <para>
+        Current processing phase, see <xref linkend="datachecksum-phases"/>
+        for description of the phases.
+       </para>
+      </entry>
+     </row>
+
+     <row>
+      <entry role="catalog_table_entry">
+       <para role="column_definition">
+        <structfield>databases_total</structfield> <type>integer</type>
+       </para>
+       <para>
+        The total number of databases which will be processed.
+       </para>
+      </entry>
+     </row>
+
+     <row>
+      <entry role="catalog_table_entry">
+       <para role="column_definition">
+        <structfield>relations_total</structfield> <type>integer</type>
+       </para>
+       <para>
+        The total number of relations which will be processed, or
+        <literal>NULL</literal> if the data checksums worker process hasn't
+        calculated the number of relations yet.
+       </para>
+      </entry>
+     </row>
+
+     <row>
+      <entry role="catalog_table_entry">
+       <para role="column_definition">
+        <structfield>databases_processed</structfield> <type>integer</type>
+       </para>
+       <para>
+        The number of databases which have been processed.
+       </para>
+      </entry>
+     </row>
+
+     <row>
+      <entry role="catalog_table_entry">
+       <para role="column_definition">
+        <structfield>relations_processed</structfield> <type>integer</type>
+       </para>
+       <para>
+        The number of relations which have been processed.
+       </para>
+      </entry>
+     </row>
+
+     <row>
+      <entry role="catalog_table_entry">
+       <para role="column_definition">
+        <structfield>databases_current</structfield> <type>oid</type>
+       </para>
+       <para>
+        OID of the database currently being processed.
+       </para>
+      </entry>
+     </row>
+
+     <row>
+      <entry role="catalog_table_entry">
+       <para role="column_definition">
+        <structfield>relation_current</structfield> <type>oid</type>
+       </para>
+       <para>
+        OID of the relation currently being processed.
+       </para>
+      </entry>
+     </row>
+
+     <row>
+      <entry role="catalog_table_entry">
+       <para role="column_definition">
+        <structfield>relation_current_blocks</structfield> <type>integer</type>
+       </para>
+       <para>
+        The total number of blocks in the relation currently being processed.
+       </para>
+      </entry>
+     </row>
+
+     <row>
+      <entry role="catalog_table_entry">
+       <para role="column_definition">
+        <structfield>relation_current_blocks_processed</structfield> <type>integer</type>
+       </para>
+       <para>
+        The number of blocks which have been processed in the relation currently
+        being processed.
+       </para>
+      </entry>
+     </row>
+
+    </tbody>
+   </tgroup>
+  </table>
+
+  <table id="datachecksum-phases">
+   <title>Data Checksum Phases</title>
+   <tgroup cols="2">
+    <colspec colname="col1" colwidth="1*"/>
+    <colspec colname="col2" colwidth="2*"/>
+    <thead>
+     <row>
+      <entry>Phase</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+    <tbody>
+     <row>
+      <entry><literal>enabling</literal></entry>
+      <entry>
+       The command is currently enabling data checksums on the cluster.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>disabling</literal></entry>
+      <entry>
+       The command is currently disabling data checksums on the cluster.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting on backends</literal></entry>
+      <entry>
+       The command is currently waiting for backends to acknowledge the data
+       checksum operation.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting on temporary tables</literal></entry>
+      <entry>
+       The command is currently waiting for all temporary tables which existed
+       at the time the command was started to be removed.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>done</literal></entry>
+      <entry>
+       The command has finished processing and is exiting.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+ </sect2>
+
  </sect1>
 
  <sect1 id="dynamic-trace">
diff --git a/doc/src/sgml/ref/pg_checksums.sgml b/doc/src/sgml/ref/pg_checksums.sgml
index 95043aa329c..0343710af53 100644
--- a/doc/src/sgml/ref/pg_checksums.sgml
+++ b/doc/src/sgml/ref/pg_checksums.sgml
@@ -45,6 +45,12 @@ PostgreSQL documentation
    exit status is nonzero if the operation failed.
   </para>
 
+  <para>
+   When enabling checksums, if checksums were in the process of being enabled
+   when the cluster was shut down, <application>pg_checksums</application>
+   will still process all relations regardless of the online processing.
+  </para>
+
   <para>
    When verifying checksums, every file in the cluster is scanned. When
    enabling checksums, each relation file block with a changed checksum is
diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgml
index f3b86b26be9..2562b1a002c 100644
--- a/doc/src/sgml/wal.sgml
+++ b/doc/src/sgml/wal.sgml
@@ -246,9 +246,10 @@
   <para>
    Checksums can be disabled when the cluster is initialized using <link
    linkend="app-initdb-data-checksums"><application>initdb</application></link>.
-   They can also be enabled or disabled at a later time as an offline
-   operation. Data checksums are enabled or disabled at the full cluster
-   level, and cannot be specified individually for databases or tables.
+   They can also be enabled or disabled at a later time either as an offline
+   operation or online in a running cluster allowing concurrent access. Data
+   checksums are enabled or disabled at the full cluster level, and cannot be
+   specified individually for databases or tables.
   </para>
 
   <para>
@@ -265,7 +266,7 @@
   </para>
 
   <sect2 id="checksums-offline-enable-disable">
-   <title>Off-line Enabling of Checksums</title>
+   <title>Offline Enabling of Checksums</title>
 
    <para>
     The <link linkend="app-pgchecksums"><application>pg_checksums</application></link>
@@ -274,6 +275,54 @@
    </para>
 
   </sect2>
+
+  <sect2 id="checksums-online-enable-disable">
+   <title>Online Enabling of Checksums</title>
+
+   <para>
+    Checksums can be enabled or disabled online, by calling the appropriate
+    <link linkend="functions-admin-checksum">functions</link>.
+   </para>
+
+   <para>
+    Enabling checksums will put the cluster checksum mode in
+    <literal>inprogress-on</literal> mode.  During this time, checksums will be
+    written but not verified. In addition to this, a background worker process
+    is started that enables checksums on all existing data in the cluster. Once
+    this worker has completed processing all databases in the cluster, the
+    checksum mode will automatically switch to <literal>on</literal>. The
+    processing will consume two background worker processes, make sure that
+    <varname>max_worker_processes</varname> allows for at least two more
+    additional processes.
+   </para>
+
+   <para>
+    The process will initially wait for all open transactions to finish before
+    it starts, so that it can be certain that there are no tables that have been
+    created inside a transaction that has not committed yet and thus would not
+    be visible to the process enabling checksums. It will also, for each database,
+    wait for all pre-existing temporary tables to get removed before it finishes.
+    If long-lived temporary tables are used in the application it may be necessary
+    to terminate these application connections to allow the process to complete.
+   </para>
+
+   <para>
+    If the cluster is stopped while in <literal>inprogress-on</literal> mode, for
+    any reason, then this process must be restarted manually. To do this,
+    re-execute the function <function>pg_enable_data_checksums()</function>
+    once the cluster has been restarted. The background worker will attempt
+    to resume the work from where it was interrupted.
+   </para>
+
+   <note>
+    <para>
+     Enabling checksums can cause significant I/O to the system, as most of the
+     database pages will need to be rewritten, and will be written both to the
+     data files and the WAL.
+    </para>
+   </note>
+
+  </sect2>
  </sect1>
 
   <sect1 id="wal-intro">
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 58040f28656..1e3ad2ab68d 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -18,6 +18,7 @@
 #include "access/xlog.h"
 #include "access/xlog_internal.h"
 #include "catalog/pg_control.h"
+#include "storage/bufpage.h"
 #include "utils/guc.h"
 #include "utils/timestamp.h"
 
@@ -167,6 +168,26 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
 		memcpy(&wal_level, rec, sizeof(int));
 		appendStringInfo(buf, "wal_level %s", get_wal_level_string(wal_level));
 	}
+	else if (info == XLOG_CHECKSUMS)
+	{
+		xl_checksum_state xlrec;
+
+		memcpy(&xlrec, rec, sizeof(xl_checksum_state));
+		switch (xlrec.new_checksumtype)
+		{
+			case PG_DATA_CHECKSUM_VERSION:
+				appendStringInfoString(buf, "on");
+				break;
+			case PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION:
+				appendStringInfoString(buf, "inprogress-off");
+				break;
+			case PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION:
+				appendStringInfoString(buf, "inprogress-on");
+				break;
+			default:
+				appendStringInfoString(buf, "off");
+		}
+	}
 }
 
 const char *
@@ -218,6 +239,9 @@ xlog_identify(uint8 info)
 		case XLOG_CHECKPOINT_REDO:
 			id = "CHECKPOINT_REDO";
 			break;
+		case XLOG_CHECKSUMS:
+			id = "CHECKSUMS";
+			break;
 	}
 
 	return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 799fc739e18..b5190a7e104 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -647,6 +647,16 @@ static XLogRecPtr LocalMinRecoveryPoint;
 static TimeLineID LocalMinRecoveryPointTLI;
 static bool updateMinRecoveryPoint = true;
 
+/*
+ * Local state fror Controlfile data_checksum_version. After initialization
+ * this is only updated when absorbing a procsignal barrier during interrupt
+ * processing. The reason for keeping a copy in backend-private memory is to
+ * avoid locking for interrogating checksum state. Possible values are the
+ * checksum versions defined in storage/bufpage.h as well as zero when data
+ * checksums are disabled.
+ */
+static uint32 LocalDataChecksumVersion = 0;
+
 /* For WALInsertLockAcquire/Release functions */
 static int	MyLockNo = 0;
 static bool holdingAllLocks = false;
@@ -715,6 +725,8 @@ static void WALInsertLockAcquireExclusive(void);
 static void WALInsertLockRelease(void);
 static void WALInsertLockUpdateInsertingAt(XLogRecPtr insertingAt);
 
+static void XLogChecksums(ChecksumType new_type);
+
 /*
  * Insert an XLOG record represented by an already-constructed chain of data
  * chunks.  This is a low-level routine; to construct the WAL record header
@@ -828,9 +840,10 @@ XLogInsertRecord(XLogRecData *rdata,
 		 * only happen just after a checkpoint, so it's better to be slow in
 		 * this case and fast otherwise.
 		 *
-		 * Also check to see if fullPageWrites was just turned on or there's a
-		 * running backup (which forces full-page writes); if we weren't
-		 * already doing full-page writes then go back and recompute.
+		 * Also check to see if fullPageWrites was just turned on, there's a
+		 * running backup or if checksums are enabled (all of which forces
+		 * full-page writes); if we weren't already doing full-page writes
+		 * then go back and recompute.
 		 *
 		 * If we aren't doing full-page writes then RedoRecPtr doesn't
 		 * actually affect the contents of the XLOG record, so we'll update
@@ -843,7 +856,9 @@ XLogInsertRecord(XLogRecData *rdata,
 			Assert(RedoRecPtr < Insert->RedoRecPtr);
 			RedoRecPtr = Insert->RedoRecPtr;
 		}
-		doPageWrites = (Insert->fullPageWrites || Insert->runningBackups > 0);
+		doPageWrites = (Insert->fullPageWrites ||
+						Insert->runningBackups > 0 ||
+						DataChecksumsNeedWrite());
 
 		if (doPageWrites &&
 			(!prevDoPageWrites ||
@@ -4579,9 +4594,7 @@ ReadControlFile(void)
 
 	CalculateCheckpointSegments();
 
-	/* Make the initdb settings visible as GUC variables, too */
-	SetConfigOption("data_checksums", DataChecksumsEnabled() ? "yes" : "no",
-					PGC_INTERNAL, PGC_S_DYNAMIC_DEFAULT);
+	LocalDataChecksumVersion = ControlFile->data_checksum_version;
 }
 
 /*
@@ -4615,13 +4628,345 @@ GetMockAuthenticationNonce(void)
 }
 
 /*
- * Are checksums enabled for data pages?
+ * DataChecksumsNeedWrite
+ *		Returns whether data checksums must be written or not
+ *
+ * Returns true iff data checksums are enabled or are in the process of being
+ * enabled. During "inprogress-on" and "inprogress-off" states checksums must
+ * be written even though they are not verified (see datachecksumsworker.c for
+ * a longer discussion).
+ *
+ * This function is intended for callsites which are about to write a data page
+ * to storage, and need to know whether to re-calculate the checksum for the
+ * page header. Calling this function must be performed as close to the write
+ * operation as possible to keep the critical section short.
+ */
+bool
+DataChecksumsNeedWrite(void)
+{
+	return (LocalDataChecksumVersion == PG_DATA_CHECKSUM_VERSION ||
+			LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION ||
+			LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);
+}
+
+/*
+ * DataChecksumsNeedVerify
+ *		Returns whether data checksums must be verified or not
+ *
+ * Data checksums are only verified if they are fully enabled in the cluster.
+ * During the "inprogress-on" and "inprogress-off" states they are only
+ * updated, not verified (see datachecksumsworker.c for a longer discussion).
+ *
+ * This function is intended for callsites which have read data and are about
+ * to perform checksum validation based on the result of this.  Calling this
+ * function must be performed as close to the validation call as possible to
+ * keep the critical section short. This is in order to protect against time of
+ * check/time of use situations around data checksum validation.
+ */
+bool
+DataChecksumsNeedVerify(void)
+{
+	return (LocalDataChecksumVersion == PG_DATA_CHECKSUM_VERSION);
+}
+
+/*
+ * DataChecksumsOnInProgress
+ *		Returns whether data checksums are being enabled
+ *
+ * Most operations don't need to worry about the "inprogress" states, and
+ * should use DataChecksumsNeedVerify() or DataChecksumsNeedWrite(). The
+ * "inprogress-on" state for enabling checksums is used when the checksum
+ * worker is setting checksums on all pages, it can thus be used to check for
+ * aborted checksum processing which need to be restarted.
+ */
+inline bool
+DataChecksumsOnInProgress(void)
+{
+	return (LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
+}
+
+/*
+ * DataChecksumsOffInProgress
+ *		Returns whether data checksums are being disabled
+ *
+ * The "inprogress-off" state for disabling checksums is used for when the
+ * worker resets the catalog state.  DataChecksumsNeedVerify() or
+ * DataChecksumsNeedWrite() should be used for deciding whether to read/write
+ * checksums.
  */
 bool
-DataChecksumsEnabled(void)
+DataChecksumsOffInProgress(void)
+{
+	return (LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);
+}
+
+/*
+ * SetDataChecksumsOnInProgress
+ *		Sets the data checksum state to "inprogress-on" to enable checksums
+ *
+ * To start the process of enabling data checksums in a running cluster the
+ * data_checksum_version state must be changed to "inprogress-on". See
+ * SetDataChecksumsOn below for a description on how this state change works.
+ * This function blocks until all backends in the cluster have acknowledged the
+ * state transition.
+ */
+void
+SetDataChecksumsOnInProgress(void)
+{
+	uint64		barrier;
+
+	Assert(ControlFile != NULL);
+
+	/*
+	 * The state transition is performed in a critical section with
+	 * checkpoints held off to provide crash safety.
+	 */
+	MyProc->delayChkptFlags |= DELAY_CHKPT_START;
+	START_CRIT_SECTION();
+
+	XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
+
+	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+	ControlFile->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION;
+	LWLockRelease(ControlFileLock);
+
+	barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);
+
+	END_CRIT_SECTION();
+	MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
+
+	/*
+	 * Await state change in all backends to ensure that all backends are in
+	 * "inprogress-on". Once done we know that all backends are writing data
+	 * checksums.
+	 */
+	WaitForProcSignalBarrier(barrier);
+}
+
+/*
+ * SetDataChecksumsOn
+ *		Enables data checksums cluster-wide
+ *
+ * Enabling data checksums is performed using two barriers, the first one to
+ * set the checksums state to "inprogress-on" (which is performed by
+ * SetDataChecksumsOnInProgress()) and the second one to set the state to "on"
+ * (performed here).
+ *
+ * To start the process of enabling data checksums in a running cluster the
+ * data_checksum_version state must be changed to "inprogress-on".  This state
+ * requires data checksums to be written but not verified. This ensures that
+ * all data pages can be checksummed without the risk of false negatives in
+ * validation during the process.  When all existing pages are guaranteed to
+ * have checksums, and all new pages will be initiated with checksums, the
+ * state can be changed to "on". Once the state is "on" checksums will be both
+ * written and verified. See datachecksumsworker.c for a longer discussion on
+ * how data checksums can be enabled in a running cluster.
+ *
+ * This function blocks until all backends in the cluster have acknowledged the
+ * state transition.
+ */
+void
+SetDataChecksumsOn(void)
 {
+	uint64		barrier;
+
 	Assert(ControlFile != NULL);
-	return (ControlFile->data_checksum_version > 0);
+
+	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+
+	/*
+	 * The only allowed state transition to "on" is from "inprogress-on" since
+	 * that state ensures that all pages will have data checksums written.
+	 */
+	if (ControlFile->data_checksum_version != PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION)
+	{
+		LWLockRelease(ControlFileLock);
+		elog(ERROR, "checksums not in \"inprogress-on\" mode");
+	}
+
+	LWLockRelease(ControlFileLock);
+
+	MyProc->delayChkptFlags |= DELAY_CHKPT_START;
+	START_CRIT_SECTION();
+
+	XLogChecksums(PG_DATA_CHECKSUM_VERSION);
+
+	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+	ControlFile->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
+	LWLockRelease(ControlFileLock);
+
+	barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);
+
+	END_CRIT_SECTION();
+	MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
+
+	/*
+	 * Await state transition of "on" in all backends. When done we know that
+	 * data checksums are enabled in all backends and data checksums are both
+	 * written and verified.
+	 */
+	WaitForProcSignalBarrier(barrier);
+}
+
+/*
+ * SetDataChecksumsOff
+ *		Disables data checksums cluster-wide
+ *
+ * Disabling data checksums must be performed with two sets of barriers, each
+ * carrying a different state. The state is first set to "inprogress-off"
+ * during which checksums are still written but not verified. This ensures that
+ * backends which have yet to observe the state change from "on" won't get
+ * validation errors on concurrently modified pages. Once all backends have
+ * changed to "inprogress-off", the barrier for moving to "off" can be emitted.
+ * This function blocks until all backends in the cluster have acknowledged the
+ * state transition.
+ */
+void
+SetDataChecksumsOff(void)
+{
+	uint64		barrier;
+
+	Assert(ControlFile);
+
+	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+
+	/* If data checksums are already disabled there is nothing to do */
+	if (ControlFile->data_checksum_version == 0)
+	{
+		LWLockRelease(ControlFileLock);
+		return;
+	}
+
+	/*
+	 * If data checksums are currently enabled we first transition to the
+	 * "inprogress-off" state during which backends continue to write
+	 * checksums without verifying them. When all backends are in
+	 * "inprogress-off" the next transition to "off" can be performed, after
+	 * which all data checksum processing is disabled.
+	 */
+	if (ControlFile->data_checksum_version == PG_DATA_CHECKSUM_VERSION)
+	{
+		LWLockRelease(ControlFileLock);
+
+		MyProc->delayChkptFlags |= DELAY_CHKPT_START;
+		START_CRIT_SECTION();
+
+		XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);
+
+		LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+		ControlFile->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION;
+		LWLockRelease(ControlFileLock);
+
+		barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);
+
+		END_CRIT_SECTION();
+		MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
+
+
+		/*
+		 * Update local state in all backends to ensure that any backend in
+		 * "on" state is changed to "inprogress-off".
+		 */
+		WaitForProcSignalBarrier(barrier);
+
+		/*
+		 * At this point we know that no backends are verifying data checksums
+		 * during reading. Next, we can safely move to state "off" to also
+		 * stop writing checksums.
+		 */
+	}
+	else
+	{
+		/*
+		 * Ending up here implies that the checksums state is "inprogress-on"
+		 * or "inprogress-off" and we can transition directly to "off" from
+		 * there.
+		 */
+		LWLockRelease(ControlFileLock);
+	}
+
+	/*
+	 * Ensure that we don't incur a checkpoint during disabling checksums.
+	 */
+	MyProc->delayChkptFlags |= DELAY_CHKPT_START;
+	START_CRIT_SECTION();
+
+	XLogChecksums(0);
+
+	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+	ControlFile->data_checksum_version = 0;
+	LWLockRelease(ControlFileLock);
+
+	barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);
+
+	END_CRIT_SECTION();
+	MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
+
+	WaitForProcSignalBarrier(barrier);
+}
+
+/*
+ * ProcSignalBarrier absorption functions for enabling and disabling data
+ * checksums in a running cluster. The procsignalbarriers are emitted in the
+ * SetDataChecksums* functions.
+ */
+bool
+AbsorbChecksumsOnInProgressBarrier(void)
+{
+	LocalDataChecksumVersion = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION;
+	return true;
+}
+
+bool
+AbsorbChecksumsOnBarrier(void)
+{
+	Assert(LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
+	LocalDataChecksumVersion = PG_DATA_CHECKSUM_VERSION;
+	return true;
+}
+
+bool
+AbsorbChecksumsOffInProgressBarrier(void)
+{
+	LocalDataChecksumVersion = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION;
+	return true;
+}
+
+bool
+AbsorbChecksumsOffBarrier(void)
+{
+	LocalDataChecksumVersion = 0;
+	return true;
+}
+
+/*
+ * InitLocalControlData
+ *
+ * Set up backend local caches of controldata variables which may change at
+ * any point during runtime and thus require special cased locking. So far
+ * this only applies to data_checksum_version, but it's intended to be general
+ * purpose enough to handle future cases.
+ */
+void
+InitLocalControldata(void)
+{
+	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+	LocalDataChecksumVersion = ControlFile->data_checksum_version;
+	LWLockRelease(ControlFileLock);
+}
+
+/* guc hook */
+const char *
+show_data_checksums(void)
+{
+	if (LocalDataChecksumVersion == PG_DATA_CHECKSUM_VERSION)
+		return "on";
+	else if (LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION)
+		return "inprogress-on";
+	else if (LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION)
+		return "inprogress-off";
+	else
+		return "off";
 }
 
 /*
@@ -6194,6 +6539,33 @@ StartupXLOG(void)
 	 */
 	CompleteCommitTsInitialization();
 
+	/*
+	 * If we reach this point with checksums being enabled ("inprogress-on"
+	 * state), we notify the user that they need to manually restart the
+	 * process to enable checksums. This is because we cannot launch a dynamic
+	 * background worker directly from here, it has to be launched from a
+	 * regular backend.
+	 */
+	if (ControlFile->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION)
+		ereport(WARNING,
+				(errmsg("data checksums are being enabled, but no worker is running"),
+				 errhint("If checksums were being enabled during shutdown then processing must be manually restarted.")));
+
+	/*
+	 * If data checksums were being disabled when the cluster was shut down,
+	 * we know that we have a state where all backends have stopped validating
+	 * checksums and we can move to off instead of prompting the user to
+	 * perform any action.
+	 */
+	if (ControlFile->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION)
+	{
+		XLogChecksums(0);
+
+		LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+		ControlFile->data_checksum_version = 0;
+		LWLockRelease(ControlFileLock);
+	}
+
 	/*
 	 * All done with end-of-recovery actions.
 	 *
@@ -8201,6 +8573,24 @@ XLogReportParameters(void)
 	}
 }
 
+/*
+ * Log the new state of checksums
+ */
+static void
+XLogChecksums(ChecksumType new_type)
+{
+	xl_checksum_state xlrec;
+	XLogRecPtr	recptr;
+
+	xlrec.new_checksumtype = new_type;
+
+	XLogBeginInsert();
+	XLogRegisterData((char *) &xlrec, sizeof(xl_checksum_state));
+
+	recptr = XLogInsert(RM_XLOG_ID, XLOG_CHECKSUMS);
+	XLogFlush(recptr);
+}
+
 /*
  * Update full_page_writes in shared memory, and write an
  * XLOG_FPW_CHANGE record if necessary.
@@ -8629,6 +9019,46 @@ xlog_redo(XLogReaderState *record)
 	{
 		/* nothing to do here, just for informational purposes */
 	}
+	else if (info == XLOG_CHECKSUMS)
+	{
+		xl_checksum_state state;
+		uint64		barrier;
+
+		memcpy(&state, XLogRecGetData(record), sizeof(xl_checksum_state));
+
+		LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+		ControlFile->data_checksum_version = state.new_checksumtype;
+		LWLockRelease(ControlFileLock);
+
+		/*
+		 * Block on a procsignalbarrier to await all processes having seen the
+		 * change to checksum status. Once the barrier has been passed we can
+		 * initiate the corresponding processing.
+		 */
+		switch (state.new_checksumtype)
+		{
+			case PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION:
+				barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);
+				WaitForProcSignalBarrier(barrier);
+				break;
+
+			case PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION:
+				barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);
+				WaitForProcSignalBarrier(barrier);
+				break;
+
+			case PG_DATA_CHECKSUM_VERSION:
+				barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);
+				WaitForProcSignalBarrier(barrier);
+				break;
+
+			default:
+				Assert(state.new_checksumtype == 0);
+				barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);
+				WaitForProcSignalBarrier(barrier);
+				break;
+		}
+	}
 }
 
 /*
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 8c3090165f0..6b1c2ed85ea 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -26,6 +26,7 @@
 #include "funcapi.h"
 #include "miscadmin.h"
 #include "pgstat.h"
+#include "postmaster/datachecksumsworker.h"
 #include "replication/walreceiver.h"
 #include "storage/fd.h"
 #include "storage/latch.h"
@@ -748,3 +749,43 @@ pg_promote(PG_FUNCTION_ARGS)
 						   wait_seconds)));
 	PG_RETURN_BOOL(false);
 }
+
+/*
+ * Disables data checksums for the cluster, if applicable. Starts a background
+ * worker which turns off the data checksums.
+ */
+Datum
+disable_data_checksums(PG_FUNCTION_ARGS)
+{
+	if (!superuser())
+		ereport(ERROR, errmsg("must be superuser"));
+
+	StartDataChecksumsWorkerLauncher(false, 0, 0, false);
+	PG_RETURN_VOID();
+}
+
+/*
+ * Enables data checksums for the cluster, if applicable.  Supports vacuum-
+ * like cost based throttling to limit system load. Starts a background worker
+ * which updates data checksums on existing data.
+ */
+Datum
+enable_data_checksums(PG_FUNCTION_ARGS)
+{
+	int			cost_delay = PG_GETARG_INT32(0);
+	int			cost_limit = PG_GETARG_INT32(1);
+	bool		fast = PG_GETARG_BOOL(2);
+
+	if (!superuser())
+		ereport(ERROR, errmsg("must be superuser"));
+
+	if (cost_delay < 0)
+		ereport(ERROR, errmsg("cost delay cannot be a negative value"));
+
+	if (cost_limit <= 0)
+		ereport(ERROR, errmsg("cost limit must be greater than zero"));
+
+	StartDataChecksumsWorkerLauncher(true, cost_delay, cost_limit, fast);
+
+	PG_RETURN_VOID();
+}
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 3f8a3c55725..598dd41bae6 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -1613,7 +1613,8 @@ sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
 	 * enabled for this cluster, and if this is a relation file, then verify
 	 * the checksum.
 	 */
-	if (!noverify_checksums && DataChecksumsEnabled() &&
+	if (!noverify_checksums &&
+		DataChecksumsNeedWrite() &&
 		RelFileNumberIsValid(relfilenumber))
 		verify_checksum = true;
 
@@ -2006,6 +2007,9 @@ verify_page_checksum(Page page, XLogRecPtr start_lsn, BlockNumber blkno,
 	if (PageIsNew(page) || PageGetLSN(page) >= start_lsn)
 		return true;
 
+	if (!DataChecksumsNeedVerify())
+		return true;
+
 	/* Perform the actual checksum calculation. */
 	checksum = pg_checksum_page(page, blkno);
 
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 566f308e443..b18db2b5dde 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -650,6 +650,13 @@ LANGUAGE INTERNAL
 CALLED ON NULL INPUT VOLATILE PARALLEL SAFE
 AS 'pg_stat_reset_slru';
 
+CREATE OR REPLACE FUNCTION
+  pg_enable_data_checksums(cost_delay integer DEFAULT 0,
+                           cost_limit integer DEFAULT 100,
+						   fast boolean DEFAULT false)
+  RETURNS void STRICT VOLATILE LANGUAGE internal AS 'enable_data_checksums'
+  PARALLEL RESTRICTED;
+
 --
 -- The default permissions for functions mean that anyone can execute them.
 -- A number of functions shouldn't be executable by just anyone, but rather
@@ -775,6 +782,10 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
 
 REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
 
+REVOKE EXECUTE ON FUNCTION pg_enable_data_checksums(integer, integer, boolean) FROM public;
+
+REVOKE EXECUTE ON FUNCTION pg_disable_data_checksums() FROM public;
+
 --
 -- We also set up some things as accessible to standard roles.
 --
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index a4d2cfdcaf5..0d62976dc1f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1334,6 +1334,27 @@ CREATE VIEW pg_stat_progress_copy AS
     FROM pg_stat_get_progress_info('COPY') AS S
         LEFT JOIN pg_database D ON S.datid = D.oid;
 
+CREATE VIEW pg_stat_progress_data_checksums AS
+	SELECT
+		S.pid AS pid, S.datid AS datid, D.datname AS datname,
+		CASE S.param1 WHEN 0 THEN 'enabling'
+                      WHEN 1 THEN 'disabling'
+					  WHEN 2 THEN 'waiting'
+					  WHEN 3 THEN 'waiting on backends'
+					  WHEN 4 THEN 'waiting on temporary tables'
+					  WHEN 5 THEN 'done'
+					  END AS phase,
+		CASE S.param2 WHEN -1 THEN NULL ELSE S.param2 END AS databases_total,
+		CASE S.param3 WHEN -1 THEN NULL ELSE S.param3 END AS relations_total,
+		S.param4 AS databases_processed,
+		S.param5 AS relations_processed,
+		S.param6 AS databases_current,
+		S.param7 AS relation_current,
+		S.param8 AS relation_current_blocks,
+		S.param9 AS relation_current_blocks_processed
+	FROM pg_stat_get_progress_info('DATACHECKSUMS') AS S
+		LEFT JOIN pg_database D ON S.datid = D.oid;
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 0f4435d2d97..0c36765acfe 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -18,6 +18,7 @@ OBJS = \
 	bgworker.o \
 	bgwriter.o \
 	checkpointer.o \
+	datachecksumsworker.o \
 	fork_process.o \
 	interrupt.o \
 	launch_backend.o \
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index 116ddf7b835..5720f66b0fd 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -18,6 +18,7 @@
 #include "pgstat.h"
 #include "port/atomics.h"
 #include "postmaster/bgworker_internals.h"
+#include "postmaster/datachecksumsworker.h"
 #include "postmaster/postmaster.h"
 #include "replication/logicallauncher.h"
 #include "replication/logicalworker.h"
@@ -132,6 +133,12 @@ static const struct
 	},
 	{
 		"TablesyncWorkerMain", TablesyncWorkerMain
+	},
+	{
+		"DataChecksumsWorkerLauncherMain", DataChecksumsWorkerLauncherMain
+	},
+	{
+		"DataChecksumsWorkerMain", DataChecksumsWorkerMain
 	}
 };
 
diff --git a/src/backend/postmaster/datachecksumsworker.c b/src/backend/postmaster/datachecksumsworker.c
new file mode 100644
index 00000000000..de7a077f9c2
--- /dev/null
+++ b/src/backend/postmaster/datachecksumsworker.c
@@ -0,0 +1,1376 @@
+/*-------------------------------------------------------------------------
+ *
+ * datachecksumsworker.c
+ *	  Background worker for enabling or disabling data checksums online
+ *
+ * When enabling data checksums on a database at initdb time or when shut down
+ * with pg_checksums, no extra process is required as each page is checksummed,
+ * and verified, when accessed.  When enabling checksums on an already running
+ * cluster, this worker will ensure that all pages are checksummed before
+ * verification of the checksums is turned on. In the case of disabling
+ * checksums, the state transition is performed only in the control file, no
+ * changes are performed on the data pages.
+ *
+ * Checksums can be either enabled or disabled cluster-wide, with on/off being
+ * the end state for data_checksums.
+ *
+ * Enabling checksums
+ * ------------------
+ * When enabling checksums in an online cluster, data_checksums will be set to
+ * "inprogress-on" which signals that write operations MUST compute and write
+ * the checksum on the data page, but during reading the checksum SHALL NOT be
+ * verified. This ensures that all objects created during checksumming will
+ * have checksums set, but no reads will fail due to incorrect checksum. The
+ * DataChecksumsWorker will compile a list of databases which exist at the
+ * start of checksumming, and all of these which haven't been dropped during
+ * the processing MUST have been processed successfully in order for checksums
+ * to be enabled. Any new relation created during processing will see the
+ * in-progress state and will automatically be checksummed.
+ *
+ * For each database, all relations which have storage are read and every data
+ * page is marked dirty to force a write with the checksum. This will generate
+ * a lot of WAL as the entire database is read and written.
+ *
+ * If the processing is interrupted by a cluster restart, it will be restarted
+ * from the beginning again as state isn't persisted.
+ *
+ * Disabling checksums
+ * -------------------
+ * When disabling checksums, data_checksums will be set to "inprogress-off"
+ * which signals that checksums are written but no longer verified. This ensure
+ * that backends which have yet to move from the "on" state will still be able
+ * to process data checksum validation.
+ *
+ * Synchronization and Correctness
+ * -------------------------------
+ * The processes involved in enabling, or disabling, data checksums in an
+ * online cluster must be properly synchronized with the normal backends
+ * serving concurrent queries to ensure correctness. Correctness is defined
+ * as the following:
+ *
+ *    - Backends SHALL NOT violate local data_checksums state
+ *    - Data checksums SHALL NOT be considered enabled cluster-wide until all
+ *      currently connected backends have the local state "enabled"
+ *
+ * There are two levels of synchronization required for enabling data checksums
+ * in an online cluster: (i) changing state in the active backends ("on",
+ * "off", "inprogress-on" and "inprogress-off"), and (ii) ensuring no
+ * incompatible objects and processes are left in a database when workers end.
+ * The former deals with cluster-wide agreement on data checksum state and the
+ * latter with ensuring that any concurrent activity cannot break the data
+ * checksum contract during processing.
+ *
+ * Synchronizing the state change is done with procsignal barriers, where the
+ * WAL logging backend updating the global state in the controlfile will wait
+ * for all other backends to absorb the barrier. Barrier absorption will happen
+ * during interrupt processing, which means that connected backends will change
+ * state at different times. To prevent data checksum state changes when
+ * writing and verifying checksums, interrupts shall be held off before
+ * interrogating state and resumed when the IO operation has been performed.
+ *
+ *   When Enabling Data Checksums
+ *   ----------------------------
+ *   A process which fails to observe data checksums being enabled can induce
+ *   two types of errors: failing to write the checksum when modifying the page
+ *   and failing to validate the data checksum on the page when reading it.
+ *
+ *   When processing starts all backends belong to one of the below sets, with
+ *   one set being empty:
+ *
+ *   Bd: Backends in "off" state
+ *   Bi: Backends in "inprogress-on" state
+ *
+ *   If processing is started in an online cluster then all backends are in Bd.
+ *   If processing was halted by the cluster shutting down, the controlfile
+ *   state "inprogress-on" will be observed on system startup and all backends
+ *   will be in Bd. Backends transition Bd -> Bi via a procsignalbarrier.  When
+ *   the DataChecksumsWorker has finished writing checksums on all pages and
+ *   enables data checksums cluster-wide, there are four sets of backends where
+ *   Bd shall be an empty set:
+ *
+ *   Bg: Backend updating the global state and emitting the procsignalbarrier
+ *   Bd: Backends in "off" state
+ *   Be: Backends in "on" state
+ *   Bi: Backends in "inprogress-on" state
+ *
+ *   Backends in Bi and Be will write checksums when modifying a page, but only
+ *   backends in Be will verify the checksum during reading. The Bg backend is
+ *   blocked waiting for all backends in Bi to process interrupts and move to
+ *   Be. Any backend starting while Bg is waiting on the procsignalbarrier will
+ *   observe the global state being "on" and will thus automatically belong to
+ *   Be.  Checksums are enabled cluster-wide when Bi is an empty set. Bi and Be
+ *   are compatible sets while still operating based on their local state as
+ *   both write data checksums.
+ *
+ *   When Disabling Data Checksums
+ *   -----------------------------
+ *   A process which fails to observe that data checksums have been disabled
+ *   can induce two types of errors: writing the checksum when modifying the
+ *   page and validating a data checksum which is no longer correct due to
+ *   modifications to the page.
+ *
+ *   Bg: Backend updating the global state and emitting the procsignalbarrier
+ *   Bd: Backends in "off" state
+ *   Be: Backends in "on" state
+ *   Bo: Backends in "inprogress-off" state
+ *
+ *   Backends transition from the Be state to Bd like so: Be -> Bo -> Bd
+ *
+ *   The goal is to transition all backends to Bd making the others empty sets.
+ *   Backends in Bo write data checksums, but don't validate them, such that
+ *   backends still in Be can continue to validate pages until the barrier has
+ *   been absorbed such that they are in Bo. Once all backends are in Bo, the
+ *   barrier to transition to "off" can be raised and all backends can safely
+ *   stop writing data checksums as no backend is enforcing data checksum
+ *   validation any longer.
+ *
+ *
+ * Potential optimizations
+ * -----------------------
+ * Below are some potential optimizations and improvements which were brought
+ * up during reviews of this feature, but which weren't implemented in the
+ * initial version. These are ideas listed without any validation on their
+ * feasibility or potential payoff. More discussion on these can be found on
+ * the -hackers threads linked to in the commit message of this feature.
+ *
+ *   * Launching datachecksumsworker for resuming operation from the startup
+ *     process: Currently users have to restart processing manually after a
+ *     restart since dynamic background worker cannot be started from the
+ *     postmaster. Changing the startup process could make restarting the
+ *     processing automatic on cluster restart.
+ *   * Avoid dirtying the page when checksums already match: Iff the checksum
+ *     on the page happens to already match we still dirty the page. It should
+ *     be enough to only do the log_newpage_buffer() call in that case.
+ *   * Invent a lightweight WAL record that doesn't contain the full-page
+ *     image but just the block number: On replay, the redo routine would read
+ *     the page from disk.
+ *   * Teach pg_checksums to avoid checksummed pages when pg_checksums is used
+ *     to enable checksums on a cluster which is in inprogress-on state and
+ *     may have checksummed pages (make pg_checksums be able to resume an
+ *     online operation).
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ *	  src/backend/postmaster/datachecksumsworker.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/xact.h"
+#include "access/xloginsert.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_class.h"
+#include "catalog/pg_database.h"
+#include "commands/progress.h"
+#include "commands/vacuum.h"
+#include "common/relpath.h"
+#include "miscadmin.h"
+#include "pgstat.h"
+#include "postmaster/bgworker.h"
+#include "postmaster/bgwriter.h"
+#include "postmaster/datachecksumsworker.h"
+#include "storage/bufmgr.h"
+#include "storage/checksum.h"
+#include "storage/ipc.h"
+#include "storage/lmgr.h"
+#include "storage/lwlock.h"
+#include "storage/procarray.h"
+#include "storage/smgr.h"
+#include "tcop/tcopprot.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+#include "utils/ps_status.h"
+#include "utils/syscache.h"
+
+/*
+ * Number of times we retry to open a database before giving up and consider
+ * it to have failed processing.
+ */
+#define DATACHECKSUMSWORKER_MAX_DB_RETRIES 5
+
+typedef enum
+{
+	DATACHECKSUMSWORKER_SUCCESSFUL = 0,
+	DATACHECKSUMSWORKER_ABORTED,
+	DATACHECKSUMSWORKER_FAILED,
+	DATACHECKSUMSWORKER_RETRYDB,
+} DataChecksumsWorkerResult;
+
+/*
+ * Signaling between backends calling pg_enable/disable_data_checksums, the
+ * checksums launcher process, and the checksums worker process.
+ *
+ * This struct is protected by DataChecksumsWorkerLock
+ */
+typedef struct DataChecksumsWorkerShmemStruct
+{
+	/*
+	 * These are set by pg_enable/disable_data_checksums, to tell the launcher
+	 * what the target state is.
+	 */
+	bool		launch_enable_checksums;	/* True if checksums are being
+											 * enabled, else false */
+	int			launch_cost_delay;
+	int			launch_cost_limit;
+	bool		launch_fast;
+
+	/*
+	 * Is a launcher process is currently running?
+	 *
+	 * This is set by the launcher process, after it has read the above
+	 * launch_* parameters.
+	 */
+	bool		launcher_running;
+
+	/*
+	 * These fields indicate the target state that the launcher is currently
+	 * working towards. They can be different from the corresponding launch_*
+	 * fields, if a new pg_enable/disable_data_checksums() call was made while
+	 * the launcher/worker was already running.
+	 *
+	 * The below members are set when the launcher starts, and are only
+	 * accessed read-only by the single worker. Thus, we can access these
+	 * without a lock. If multiple workers, or dynamic cost parameters, are
+	 * supported at some point then this would need to be revisited.
+	 */
+	bool		enabling_checksums; /* True if checksums are being enabled,
+									 * else false */
+	int			cost_delay;
+	int			cost_limit;
+	bool		immediate_checkpoint;
+
+	/*
+	 * Signaling between the launcher and the worker process.
+	 *
+	 * As there is only a single worker, and the launcher won't read these
+	 * until the worker exits, they can be accessed without the need for a
+	 * lock. If multiple workers are supported then this will have to be
+	 * revisited.
+	 */
+
+	/* result, set by worker before exiting */
+	DataChecksumsWorkerResult success;
+
+	/*
+	 * tells the worker process whether it should also process the shared
+	 * catalogs
+	 */
+	bool		process_shared_catalogs;
+} DataChecksumsWorkerShmemStruct;
+
+/* Shared memory segment for datachecksumsworker */
+static DataChecksumsWorkerShmemStruct *DataChecksumsWorkerShmem;
+
+/* Bookkeeping for work to do */
+typedef struct DataChecksumsWorkerDatabase
+{
+	Oid			dboid;
+	char	   *dbname;
+} DataChecksumsWorkerDatabase;
+
+typedef struct DataChecksumsWorkerResultEntry
+{
+	Oid			dboid;
+	DataChecksumsWorkerResult result;
+	int			retries;
+} DataChecksumsWorkerResultEntry;
+
+
+/*
+ * Flag set by the interrupt handler
+ */
+static volatile sig_atomic_t abort_requested = false;
+
+/*
+ * Have we set the DataChecksumsWorkerShmemStruct->launcher_running flag?
+ * If we have, we need to clear it before exiting!
+ */
+static volatile sig_atomic_t launcher_running = false;
+
+/*
+ * Are we enabling data checksums, or disabling them?
+ */
+static bool enabling_checksums;
+
+/* Prototypes */
+static List *BuildDatabaseList(void);
+static List *BuildRelationList(bool temp_relations, bool include_shared);
+static void FreeDatabaseList(List *dblist);
+static DataChecksumsWorkerResult ProcessDatabase(DataChecksumsWorkerDatabase *db);
+static bool ProcessAllDatabases(bool immediate_checkpoint);
+static bool ProcessSingleRelationFork(Relation reln, ForkNumber forkNum, BufferAccessStrategy strategy);
+static void launcher_cancel_handler(SIGNAL_ARGS);
+static void WaitForAllTransactionsToFinish(void);
+
+/*
+ * StartDataChecksumsWorkerLauncher
+ *		Main entry point for datachecksumsworker launcher process
+ *
+ * The main entrypoint for starting data checksums processing for enabling as
+ * well as disabling.
+ */
+void
+StartDataChecksumsWorkerLauncher(bool enable_checksums,
+								 int cost_delay,
+								 int cost_limit,
+								 bool fast)
+{
+	BackgroundWorker bgw;
+	BackgroundWorkerHandle *bgw_handle;
+	bool		launcher_running;
+
+	/* the cost delay settings have no effect when disabling */
+	Assert(enable_checksums || cost_delay == 0);
+	Assert(enable_checksums || cost_limit == 0);
+
+	/* store the desired state in shared memory */
+	LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
+
+	DataChecksumsWorkerShmem->launch_enable_checksums = enable_checksums;
+	DataChecksumsWorkerShmem->launch_cost_delay = cost_delay;
+	DataChecksumsWorkerShmem->launch_cost_limit = cost_limit;
+	DataChecksumsWorkerShmem->launch_fast = fast;
+
+	/* is the launcher already running? */
+	launcher_running = DataChecksumsWorkerShmem->launcher_running;
+
+	LWLockRelease(DataChecksumsWorkerLock);
+
+	/*
+	 * Launch a new launcher process, if it's not running already.
+	 *
+	 * If the launcher is currently busy enabling the checksums, and we want
+	 * them disabled (or vice versa), the launcher will notice that at latest
+	 * when it's about to exit, and will loop back process the new request. So
+	 * if the launcher is already running, we don't need to do anything more
+	 * here to abort it.
+	 *
+	 * If you call pg_enable/disable_data_checksums() twice in a row, before
+	 * the launcher has had a chance to start up, we still end up launching it
+	 * twice.  That's OK, the second invocation will see that a launcher is
+	 * already running and exit quickly.
+	 *
+	 * TODO: We could optimize here and skip launching the launcher, if we are
+	 * already in the desired state, i.e. if the checksums are already enabled
+	 * and you call pg_enable_data_checksums().
+	 */
+	if (!launcher_running)
+	{
+		/*
+		 * Prepare the BackgroundWorker and launch it.
+		 */
+		memset(&bgw, 0, sizeof(bgw));
+		bgw.bgw_flags = BGWORKER_SHMEM_ACCESS | BGWORKER_BACKEND_DATABASE_CONNECTION;
+		bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+		snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+		snprintf(bgw.bgw_function_name, BGW_MAXLEN, "DataChecksumsWorkerLauncherMain");
+		snprintf(bgw.bgw_name, BGW_MAXLEN, "datachecksumsworker launcher");
+		snprintf(bgw.bgw_type, BGW_MAXLEN, "datachecksumsworker launcher");
+		bgw.bgw_restart_time = BGW_NEVER_RESTART;
+		bgw.bgw_notify_pid = MyProcPid;
+		bgw.bgw_main_arg = (Datum) 0;
+
+		if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle))
+			ereport(ERROR,
+					(errmsg("failed to start background worker to process data checksums")));
+	}
+}
+
+/*
+ * ProcessSingleRelationFork
+ *		Enable data checksums in a single relation/fork.
+ *
+ * Returns true if successful, and false if *aborted*. On error, an actual
+ * error is raised in the lower levels.
+ */
+static bool
+ProcessSingleRelationFork(Relation reln, ForkNumber forkNum, BufferAccessStrategy strategy)
+{
+	BlockNumber numblocks = RelationGetNumberOfBlocksInFork(reln, forkNum);
+	BlockNumber blknum;
+	char		activity[NAMEDATALEN * 2 + 128];
+	char	   *relns;
+
+	relns = get_namespace_name(RelationGetNamespace(reln));
+
+	if (!relns)
+		return false;
+
+	/* Report the current relation to pgstat_activity */
+	snprintf(activity, sizeof(activity) - 1, "processing: %s.%s (%s, %dblocks)",
+			 relns, RelationGetRelationName(reln), forkNames[forkNum], numblocks);
+	pgstat_report_activity(STATE_RUNNING, activity);
+
+	/*
+	 * We are looping over the blocks which existed at the time of process
+	 * start, which is safe since new blocks are created with checksums set
+	 * already due to the state being "inprogress-on".
+	 */
+	for (blknum = 0; blknum < numblocks; blknum++)
+	{
+		Buffer		buf = ReadBufferExtended(reln, forkNum, blknum, RBM_NORMAL, strategy);
+
+		/* Need to get an exclusive lock before we can flag as dirty */
+		LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+		/*
+		 * Mark the buffer as dirty and force a full page write.  We have to
+		 * re-write the page to WAL even if the checksum hasn't changed,
+		 * because if there is a replica it might have a slightly different
+		 * version of the page with an invalid checksum, caused by unlogged
+		 * changes (e.g. hintbits) on the master happening while checksums
+		 * were off. This can happen if there was a valid checksum on the page
+		 * at one point in the past, so only when checksums are first on, then
+		 * off, and then turned on again. Iff wal_level is set to "minimal",
+		 * this could be avoided iff the checksum is calculated to be correct.
+		 */
+		START_CRIT_SECTION();
+		MarkBufferDirty(buf);
+		log_newpage_buffer(buf, false);
+		END_CRIT_SECTION();
+
+		UnlockReleaseBuffer(buf);
+
+		/*
+		 * This is the only place where we check if we are asked to abort, the
+		 * abortion will bubble up from here. It's safe to check this without
+		 * a lock, because if we miss it being set, we will try again soon.
+		 */
+		Assert(enabling_checksums);
+		if (!DataChecksumsWorkerShmem->launch_enable_checksums)
+			abort_requested = true;
+		if (abort_requested)
+			return false;
+
+		vacuum_delay_point();
+	}
+
+	pfree(relns);
+	return true;
+}
+
+/*
+ * ProcessSingleRelationByOid
+ *		Process a single relation based on oid.
+ *
+ * Returns true if successful, and false if *aborted*. On error, an actual
+ * error is raised in the lower levels.
+ */
+static bool
+ProcessSingleRelationByOid(Oid relationId, BufferAccessStrategy strategy)
+{
+	Relation	rel;
+	ForkNumber	fnum;
+	bool		aborted = false;
+
+	StartTransactionCommand();
+
+	rel = try_relation_open(relationId, AccessShareLock);
+	if (rel == NULL)
+	{
+		/*
+		 * Relation no longer exists. We don't consider this an error since
+		 * there are no pages in it that need data checksums, and thus return
+		 * true. The worker operates off a list of relations generated at the
+		 * start of processing, so relations being dropped in the meantime is
+		 * to be expected.
+		 */
+		CommitTransactionCommand();
+		pgstat_report_activity(STATE_IDLE, NULL);
+		return true;
+	}
+	RelationGetSmgr(rel);
+
+	for (fnum = 0; fnum <= MAX_FORKNUM; fnum++)
+	{
+		if (smgrexists(rel->rd_smgr, fnum))
+		{
+			if (!ProcessSingleRelationFork(rel, fnum, strategy))
+			{
+				aborted = true;
+				break;
+			}
+		}
+	}
+	relation_close(rel, AccessShareLock);
+	elog(DEBUG2,
+		 "data checksum processing done for relation with OID %u: %s",
+		 relationId, (aborted ? "aborted" : "finished"));
+
+	CommitTransactionCommand();
+
+	pgstat_report_activity(STATE_IDLE, NULL);
+
+	return !aborted;
+}
+
+/*
+ * ProcessDatabase
+ *		Enable data checksums in a single database.
+ *
+ * We do this by launching a dynamic background worker into this database, and
+ * waiting for it to finish.  We have to do this in a separate worker, since
+ * each process can only be connected to one database during its lifetime.
+ */
+static DataChecksumsWorkerResult
+ProcessDatabase(DataChecksumsWorkerDatabase *db)
+{
+	BackgroundWorker bgw;
+	BackgroundWorkerHandle *bgw_handle;
+	BgwHandleStatus status;
+	pid_t		pid;
+	char		activity[NAMEDATALEN + 64];
+
+	DataChecksumsWorkerShmem->success = DATACHECKSUMSWORKER_FAILED;
+
+	memset(&bgw, 0, sizeof(bgw));
+	bgw.bgw_flags = BGWORKER_SHMEM_ACCESS | BGWORKER_BACKEND_DATABASE_CONNECTION;
+	bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+	snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+	snprintf(bgw.bgw_function_name, BGW_MAXLEN, "%s", "DataChecksumsWorkerMain");
+	snprintf(bgw.bgw_name, BGW_MAXLEN, "datachecksumsworker worker");
+	snprintf(bgw.bgw_type, BGW_MAXLEN, "datachecksumsworker worker");
+	bgw.bgw_restart_time = BGW_NEVER_RESTART;
+	bgw.bgw_notify_pid = MyProcPid;
+	bgw.bgw_main_arg = ObjectIdGetDatum(db->dboid);
+
+	/*
+	 * If there are no worker slots available, make sure we retry processing
+	 * this database. This will make the datachecksumsworker move on to the
+	 * next database and quite likely fail with the same problem. TODO: Maybe
+	 * we need a backoff to avoid running through all the databases here in
+	 * short order.
+	 */
+	if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle))
+	{
+		ereport(WARNING,
+				(errmsg("failed to start worker for enabling data checksums in database \"%s\", retrying",
+						db->dbname),
+				 errhint("The max_worker_processes setting might be too low.")));
+		return DATACHECKSUMSWORKER_RETRYDB;
+	}
+
+	status = WaitForBackgroundWorkerStartup(bgw_handle, &pid);
+	if (status == BGWH_STOPPED)
+	{
+		ereport(WARNING,
+				(errmsg("could not start background worker for enabling data checksums in database \"%s\"",
+						db->dbname),
+				 errhint("More details on the error might be found in the server log.")));
+		return DATACHECKSUMSWORKER_FAILED;
+	}
+
+	/*
+	 * If the postmaster crashed we cannot end up with a processed database so
+	 * we have no alternative other than exiting. When enabling checksums we
+	 * won't at this time have changed the pg_control version to enabled so
+	 * when the cluster comes back up processing will have to be restarted.
+	 * When disabling, the pg_control version will be set to off before this
+	 * so when the cluster comes up checksums will be off as expected.
+	 */
+	if (status == BGWH_POSTMASTER_DIED)
+		ereport(FATAL,
+				(errmsg("cannot enable data checksums without the postmaster process"),
+				 errhint("Restart the database and restart data checksum processing by calling pg_enable_data_checksums().")));
+
+	Assert(status == BGWH_STARTED);
+	ereport(DEBUG1,
+			(errmsg("initiating data checksum processing in database \"%s\"",
+					db->dbname)));
+
+	snprintf(activity, sizeof(activity) - 1,
+			 "Waiting for worker in database %s (pid %d)", db->dbname, pid);
+	pgstat_report_activity(STATE_RUNNING, activity);
+
+	status = WaitForBackgroundWorkerShutdown(bgw_handle);
+	if (status == BGWH_POSTMASTER_DIED)
+		ereport(FATAL,
+				(errmsg("postmaster exited during data checksum processing in \"%s\"",
+						db->dbname),
+				 errhint("Restart the database and restart data checksum processing by calling pg_enable_data_checksums().")));
+
+	if (DataChecksumsWorkerShmem->success == DATACHECKSUMSWORKER_ABORTED)
+		ereport(LOG,
+				(errmsg("data checksums processing was aborted in database \"%s\"",
+						db->dbname)));
+
+	pgstat_report_activity(STATE_IDLE, NULL);
+
+	return DataChecksumsWorkerShmem->success;
+}
+
+/*
+ * launcher_exit
+ *
+ * Internal routine for cleaning up state when the launcher process exits. We
+ * need to clean up the abort flag to ensure that processing can be restarted
+ * again after it was previously aborted.
+ */
+static void
+launcher_exit(int code, Datum arg)
+{
+	if (launcher_running)
+	{
+		LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
+		launcher_running = false;
+		DataChecksumsWorkerShmem->launcher_running = false;
+		LWLockRelease(DataChecksumsWorkerLock);
+	}
+}
+
+/*
+ * launcher_cancel_handler
+ *
+ * Internal routine for reacting to SIGINT and flagging the worker to abort.
+ * The worker won't be interrupted immediately but will check for abort flag
+ * between each block in a relation.
+ */
+static void
+launcher_cancel_handler(SIGNAL_ARGS)
+{
+	int			save_errno = errno;
+
+	abort_requested = true;
+
+	/*
+	 * There is no sleeping in the main loop, the flag will be checked
+	 * periodically in ProcessSingleRelationFork. The worker does however
+	 * sleep when waiting for concurrent transactions to end so we still need
+	 * to set the latch.
+	 */
+	SetLatch(MyLatch);
+
+	errno = save_errno;
+}
+
+/*
+ * WaitForAllTransactionsToFinish
+ *		Blocks awaiting all current transactions to finish
+ *
+ * Returns when all transactions which are active at the call of the function
+ * have ended, or if the postmaster dies while waiting. If the postmaster dies
+ * the abort flag will be set to indicate that the caller of this shouldn't
+ * proceed.
+ *
+ * NB: this will return early, if aborted by SIGINT or if the target state
+ * is changed while we're running.
+ */
+static void
+WaitForAllTransactionsToFinish(void)
+{
+	TransactionId waitforxid;
+
+	LWLockAcquire(XidGenLock, LW_SHARED);
+	waitforxid = XidFromFullTransactionId(TransamVariables->nextXid);
+	LWLockRelease(XidGenLock);
+
+	while (TransactionIdPrecedes(GetOldestActiveTransactionId(), waitforxid))
+	{
+		char		activity[64];
+		int			rc;
+
+		/* Oldest running xid is older than us, so wait */
+		snprintf(activity,
+				 sizeof(activity),
+				 "Waiting for current transactions to finish (waiting for %u)",
+				 waitforxid);
+		pgstat_report_activity(STATE_RUNNING, activity);
+
+		/* Retry every 5 seconds */
+		ResetLatch(MyLatch);
+		rc = WaitLatch(MyLatch,
+					   WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+					   5000,
+					   WAIT_EVENT_CHECKSUM_ENABLE_STARTCONDITION);
+
+		/*
+		 * If the postmaster died we won't be able to enable checksums
+		 * cluster-wide so abort and hope to continue when restarted.
+		 */
+		if (rc & WL_POSTMASTER_DEATH)
+			ereport(FATAL,
+					(errmsg("postmaster exited during data checksum processing"),
+					 errhint("Restart the database and restart data checksum processing by calling pg_enable_data_checksums().")));
+
+		LWLockAcquire(DataChecksumsWorkerLock, LW_SHARED);
+		if (DataChecksumsWorkerShmem->launch_enable_checksums != enabling_checksums)
+			abort_requested = true;
+		LWLockRelease(DataChecksumsWorkerLock);
+		if (abort_requested)
+			break;
+	}
+
+	pgstat_report_activity(STATE_IDLE, NULL);
+	return;
+}
+
+/*
+ * DataChecksumsWorkerLauncherMain
+ *
+ * Main function for launching dynamic background workers for processing data
+ * checksums in databases. This function has the bgworker management, with
+ * ProcessAllDatabases being responsible for looping over the databases and
+ * initiating processing.
+ */
+void
+DataChecksumsWorkerLauncherMain(Datum arg)
+{
+	on_shmem_exit(launcher_exit, 0);
+
+	ereport(DEBUG1,
+			errmsg("background worker \"datachecksumsworker\" launcher started"));
+
+	pqsignal(SIGTERM, die);
+	pqsignal(SIGINT, launcher_cancel_handler);
+
+	BackgroundWorkerUnblockSignals();
+
+	MyBackendType = B_DATACHECKSUMSWORKER_LAUNCHER;
+	init_ps_display(NULL);
+
+	LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
+
+	if (DataChecksumsWorkerShmem->launcher_running)
+	{
+		/* Launcher was already running, let it finish */
+		LWLockRelease(DataChecksumsWorkerLock);
+		return;
+	}
+
+	launcher_running = true;
+
+	/*
+	 * Initialize a connection to shared catalogs only.
+	 */
+	BackgroundWorkerInitializeConnectionByOid(InvalidOid, InvalidOid, 0);
+
+	/* Initialize backend status information */
+	pgstat_bestart();
+
+	enabling_checksums = DataChecksumsWorkerShmem->launch_enable_checksums;
+	DataChecksumsWorkerShmem->launcher_running = true;
+	DataChecksumsWorkerShmem->enabling_checksums = enabling_checksums;
+	DataChecksumsWorkerShmem->cost_delay = DataChecksumsWorkerShmem->launch_cost_delay;
+	DataChecksumsWorkerShmem->cost_limit = DataChecksumsWorkerShmem->launch_cost_limit;
+	DataChecksumsWorkerShmem->immediate_checkpoint = DataChecksumsWorkerShmem->launch_fast;
+	LWLockRelease(DataChecksumsWorkerLock);
+
+	/*
+	 * The target state can change while we are busy enabling/disabling
+	 * checksums, if the user calls pg_disable/enable_data_checksums() before
+	 * we are finished with the previous request. In that case, we will loop
+	 * back here, to process the new request.
+	 */
+again:
+
+	pgstat_progress_start_command(PROGRESS_COMMAND_DATACHECKSUMS,
+								  InvalidOid);
+
+	/*
+	 * If we're asked to enable checksums, we need to check if processing was
+	 * previously interrupted such that we should resume rather than start
+	 * from scratch.
+	 */
+	if (enabling_checksums)
+	{
+		/*
+		 * If we are asked to enable checksums in a cluster which already has
+		 * checksums enabled, exit immediately as there is nothing more to do.
+		 * Hold interrupts to make sure state doesn't change during checking.
+		 */
+		HOLD_INTERRUPTS();
+		if (DataChecksumsNeedVerify())
+		{
+			RESUME_INTERRUPTS();
+			goto done;
+		}
+		RESUME_INTERRUPTS();
+
+		/*
+		 * Initialize progress and indicate that we are waiting on the other
+		 * backends to clear the procsignalbarrier.
+		 */
+		pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,
+									 PROGRESS_DATACHECKSUMS_PHASE_WAITING_BACKENDS);
+
+		/*
+		 * Set the state to inprogress-on and wait on the procsignal barrier.
+		 */
+		pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,
+									 PROGRESS_DATACHECKSUMS_PHASE_ENABLING);
+		SetDataChecksumsOnInProgress();
+
+		if (!ProcessAllDatabases(DataChecksumsWorkerShmem->immediate_checkpoint))
+		{
+			/*
+			 * If the target state changed during processing then it's not a
+			 * failure, so restart processing instead.
+			 */
+			LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
+			if (DataChecksumsWorkerShmem->launch_enable_checksums != enabling_checksums)
+			{
+				LWLockRelease(DataChecksumsWorkerLock);
+				goto done;
+			}
+			LWLockRelease(DataChecksumsWorkerLock);
+			ereport(ERROR,
+					(errmsg("unable to enable data checksums in cluster")));
+		}
+
+		SetDataChecksumsOn();
+	}
+	else
+	{
+		pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,
+									 PROGRESS_DATACHECKSUMS_PHASE_DISABLING);
+		SetDataChecksumsOff();
+	}
+
+done:
+
+	/*
+	 * All done. But before we exit, check if the target state was changed
+	 * while we were running. In that case we will have to start all over
+	 * again.
+	 */
+	LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
+	if (DataChecksumsWorkerShmem->launch_enable_checksums != enabling_checksums)
+	{
+		DataChecksumsWorkerShmem->enabling_checksums = DataChecksumsWorkerShmem->launch_enable_checksums;
+		enabling_checksums = DataChecksumsWorkerShmem->launch_enable_checksums;
+		DataChecksumsWorkerShmem->cost_delay = DataChecksumsWorkerShmem->launch_cost_delay;
+		DataChecksumsWorkerShmem->cost_limit = DataChecksumsWorkerShmem->launch_cost_limit;
+		LWLockRelease(DataChecksumsWorkerLock);
+		goto again;
+	}
+
+	/* Shut down progress reporting as we are done */
+	pgstat_progress_end_command();
+
+	launcher_running = false;
+	DataChecksumsWorkerShmem->launcher_running = false;
+	LWLockRelease(DataChecksumsWorkerLock);
+}
+
+/*
+ * ProcessAllDatabases
+ *		Compute the list of all databases and process checksums in each
+ *
+ * This will repeatedly generate a list of databases to process for enabling
+ * checksums. Until no new databases are found, this will loop around computing
+ * a new list and comparing it to the already seen ones.
+ *
+ * If immediate_checkpoint is set to true then a CHECKPOINT_IMMEDIATE will be
+ * issued. This is useful for testing but should be avoided in production use
+ * as it may affect cluster performance drastically.
+ */
+static bool
+ProcessAllDatabases(bool immediate_checkpoint)
+{
+	List	   *DatabaseList;
+	HTAB	   *ProcessedDatabases = NULL;
+	HASHCTL		hash_ctl;
+	bool		found_failed = false;
+	int			flags;
+
+	/* Initialize a hash tracking all processed databases */
+	memset(&hash_ctl, 0, sizeof(hash_ctl));
+	hash_ctl.keysize = sizeof(Oid);
+	hash_ctl.entrysize = sizeof(DataChecksumsWorkerResultEntry);
+	ProcessedDatabases = hash_create("Processed databases",
+									 64,
+									 &hash_ctl,
+									 HASH_ELEM | HASH_BLOBS);
+
+	/*
+	 * Set up so first run processes shared catalogs, but not once in every
+	 * db.
+	 */
+	DataChecksumsWorkerShmem->process_shared_catalogs = true;
+
+	/*
+	 * Get a list of all databases to process. This may include databases that
+	 * were created during our runtime.  Since a database can be created as a
+	 * copy of any other database (which may not have existed in our last
+	 * run), we have to repeat this loop until no new databases show up in the
+	 * list.
+	 */
+	DatabaseList = BuildDatabaseList();
+
+	/*
+	 * Update progress reporting with the total number of databases we need to
+	 * process. This number should not be changed during processing, the
+	 * columns for processed databases is instead increased such that it can
+	 * be compared against the total.
+	 */
+	pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_TOTAL_DB,
+								 list_length(DatabaseList));
+
+	while (true)
+	{
+		int			processed_databases = 0;
+
+		foreach_ptr(DataChecksumsWorkerDatabase, db, DatabaseList)
+		{
+			DataChecksumsWorkerResult result;
+			DataChecksumsWorkerResultEntry *entry;
+			bool		found;
+
+			/*
+			 * Indicate which database is being processed set the number of
+			 * relations to -1 to clear field from previous values. -1 will
+			 * translate to NULL in the progress view.
+			 */
+			pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_CUR_DB, db->dboid);
+			pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_TOTAL_REL, -1);
+
+			/*
+			 * Check if this database has been processed already, and if so
+			 * whether it should be retried or skipped.
+			 */
+			entry = (DataChecksumsWorkerResultEntry *) hash_search(ProcessedDatabases, &db->dboid,
+																   HASH_FIND, NULL);
+
+			if (entry)
+			{
+				if (entry->result == DATACHECKSUMSWORKER_RETRYDB)
+				{
+					/*
+					 * Limit the number of retries to avoid infinite looping
+					 * in case there simply won't be enough workers in the
+					 * cluster to finish this operation.
+					 */
+					if (entry->retries > DATACHECKSUMSWORKER_MAX_DB_RETRIES)
+						entry->result = DATACHECKSUMSWORKER_FAILED;
+				}
+
+				/* Skip if this database has been processed already */
+				if (entry->result != DATACHECKSUMSWORKER_RETRYDB)
+					continue;
+			}
+
+			result = ProcessDatabase(db);
+			processed_databases++;
+
+			if (result == DATACHECKSUMSWORKER_SUCCESSFUL)
+			{
+				/*
+				 * If one database has completed shared catalogs, we don't
+				 * have to process them again.
+				 */
+				if (DataChecksumsWorkerShmem->process_shared_catalogs)
+					DataChecksumsWorkerShmem->process_shared_catalogs = false;
+			}
+			else if (result == DATACHECKSUMSWORKER_ABORTED)
+			{
+				/* Abort flag set, so exit the whole process */
+				return false;
+			}
+
+			entry = hash_search(ProcessedDatabases, &db->dboid, HASH_ENTER, &found);
+			entry->dboid = db->dboid;
+			entry->result = result;
+			if (!found)
+				entry->retries = 0;
+			else
+				entry->retries++;
+		}
+
+		elog(DEBUG1,
+			 "%i databases processed for data checksum enabling, %s",
+			 processed_databases,
+			 (processed_databases ? "process with restart" : "process completed"));
+
+		FreeDatabaseList(DatabaseList);
+
+		/*
+		 * If no databases were processed in this run of the loop, we have now
+		 * finished all databases and no concurrently created ones can exist.
+		 */
+		if (processed_databases == 0)
+			break;
+
+		/*
+		 * Re-generate the list of databases for another pass. Since we wait
+		 * for all pre-existing transactions finish, this way we can be
+		 * certain that there are no databases left without checksums.
+		 */
+		WaitForAllTransactionsToFinish();
+		DatabaseList = BuildDatabaseList();
+	}
+
+	/*
+	 * ProcessedDatabases now has all databases and the results of their
+	 * processing. Failure to enable checksums for a database can be because
+	 * they actually failed for some reason, or because the database was
+	 * dropped between us getting the database list and trying to process it.
+	 * Get a fresh list of databases to detect the second case where the
+	 * database was dropped before we had started processing it. If a database
+	 * still exists, but enabling checksums failed then we fail the entire
+	 * checksumming process and exit with an error.
+	 */
+	WaitForAllTransactionsToFinish();
+	DatabaseList = BuildDatabaseList();
+
+	foreach_ptr(DataChecksumsWorkerDatabase, db, DatabaseList)
+	{
+		DataChecksumsWorkerResult *entry;
+		bool		found;
+
+		entry = hash_search(ProcessedDatabases, (void *) &db->dboid,
+							HASH_FIND, &found);
+
+		/*
+		 * We are only interested in the databases where the failed database
+		 * still exists.
+		 */
+		if (found && *entry == DATACHECKSUMSWORKER_FAILED)
+		{
+			ereport(WARNING,
+					(errmsg("failed to enable data checksums in \"%s\"",
+							db->dbname)));
+			found_failed = found;
+			continue;
+		}
+	}
+
+	FreeDatabaseList(DatabaseList);
+
+	if (found_failed)
+	{
+		/* Disable checksums on cluster, because we failed */
+		SetDataChecksumsOff();
+		ereport(ERROR,
+				(errmsg("data checksums failed to get enabled in all databases, aborting"),
+				 errhint("The server log might have more information on the cause of the error.")));
+	}
+
+	/*
+	 * Force a checkpoint to get everything out to disk. The use of immediate
+	 * checkpoints is for running tests, as they would otherwise not execute
+	 * in such a way that they can reliably be placed under timeout control.
+	 */
+	flags = CHECKPOINT_FORCE | CHECKPOINT_WAIT;
+	if (immediate_checkpoint)
+		flags = flags | CHECKPOINT_IMMEDIATE;
+	RequestCheckpoint(flags);
+
+	return true;
+}
+
+/*
+ * DataChecksumsWorkerShmemSize
+ *		Compute required space for datachecksumsworker-related shared memory
+ */
+Size
+DataChecksumsWorkerShmemSize(void)
+{
+	Size		size;
+
+	size = sizeof(DataChecksumsWorkerShmemStruct);
+	size = MAXALIGN(size);
+
+	return size;
+}
+
+/*
+ * DataChecksumsWorkerShmemInit
+ *		Allocate and initialize datachecksumsworker-related shared memory
+ */
+void
+DataChecksumsWorkerShmemInit(void)
+{
+	bool		found;
+
+	DataChecksumsWorkerShmem = (DataChecksumsWorkerShmemStruct *)
+		ShmemInitStruct("DataChecksumsWorker Data",
+						DataChecksumsWorkerShmemSize(),
+						&found);
+
+	MemSet(DataChecksumsWorkerShmem, 0, DataChecksumsWorkerShmemSize());
+
+	/*
+	 * Even if this is a redundant assignment, we want to be explicit about
+	 * our intent for readability, since we want to be able to query this
+	 * state in case of restartability.
+	 */
+	DataChecksumsWorkerShmem->launch_enable_checksums = false;
+	DataChecksumsWorkerShmem->launcher_running = false;
+	DataChecksumsWorkerShmem->launch_fast = false;
+}
+
+/*
+ * BuildDatabaseList
+ *		Compile a list of all currently available databases in the cluster
+ *
+ * This creates the list of databases for the datachecksumsworker workers to
+ * add checksums to. If the caller wants to ensure that no concurrently
+ * running CREATE DATABASE calls exist, this needs to be preceded by a call
+ * to WaitForAllTransactionsToFinish().
+ */
+static List *
+BuildDatabaseList(void)
+{
+	List	   *DatabaseList = NIL;
+	Relation	rel;
+	TableScanDesc scan;
+	HeapTuple	tup;
+	MemoryContext ctx = CurrentMemoryContext;
+	MemoryContext oldctx;
+
+	StartTransactionCommand();
+
+	rel = table_open(DatabaseRelationId, AccessShareLock);
+	scan = table_beginscan_catalog(rel, 0, NULL);
+
+	while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+	{
+		Form_pg_database pgdb = (Form_pg_database) GETSTRUCT(tup);
+		DataChecksumsWorkerDatabase *db;
+
+		oldctx = MemoryContextSwitchTo(ctx);
+
+		db = (DataChecksumsWorkerDatabase *) palloc0(sizeof(DataChecksumsWorkerDatabase));
+
+		db->dboid = pgdb->oid;
+		db->dbname = pstrdup(NameStr(pgdb->datname));
+
+		DatabaseList = lappend(DatabaseList, db);
+
+		MemoryContextSwitchTo(oldctx);
+	}
+
+	table_endscan(scan);
+	table_close(rel, AccessShareLock);
+
+	CommitTransactionCommand();
+
+	return DatabaseList;
+}
+
+static void
+FreeDatabaseList(List *dblist)
+{
+	if (!dblist)
+		return;
+
+	foreach_ptr(DataChecksumsWorkerDatabase, db, dblist)
+	{
+		if (db->dbname != NULL)
+			pfree(db->dbname);
+	}
+
+	list_free_deep(dblist);
+}
+
+/*
+ * BuildRelationList
+ *		Compile a list of relations in the database
+ *
+ * Returns a list of OIDs for the request relation types. If temp_relations
+ * is True then only temporary relations are returned. If temp_relations is
+ * False then non-temporary relations which have data checksums are returned.
+ * If include_shared is True then shared relations are included as well in a
+ * non-temporary list. include_shared has no relevance when building a list of
+ * temporary relations.
+ */
+static List *
+BuildRelationList(bool temp_relations, bool include_shared)
+{
+	List	   *RelationList = NIL;
+	Relation	rel;
+	TableScanDesc scan;
+	HeapTuple	tup;
+	MemoryContext ctx = CurrentMemoryContext;
+	MemoryContext oldctx;
+
+	StartTransactionCommand();
+
+	rel = table_open(RelationRelationId, AccessShareLock);
+	scan = table_beginscan_catalog(rel, 0, NULL);
+
+	while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+	{
+		Form_pg_class pgc = (Form_pg_class) GETSTRUCT(tup);
+
+		/*
+		 * Only include temporary relations when asked for a temp relation
+		 * list.
+		 */
+		if (pgc->relpersistence == RELPERSISTENCE_TEMP)
+		{
+			if (!temp_relations)
+				continue;
+		}
+		else
+		{
+			/*
+			 * If we are only interested in temp relations then continue
+			 * immediately as the current relation isn't a temp relation.
+			 */
+			if (temp_relations)
+				continue;
+
+			if (!RELKIND_HAS_STORAGE(pgc->relkind))
+				continue;
+
+			if (pgc->relisshared && !include_shared)
+				continue;
+		}
+
+		oldctx = MemoryContextSwitchTo(ctx);
+		RelationList = lappend_oid(RelationList, pgc->oid);
+		MemoryContextSwitchTo(oldctx);
+	}
+
+	table_endscan(scan);
+	table_close(rel, AccessShareLock);
+
+	CommitTransactionCommand();
+
+	return RelationList;
+}
+
+/*
+ * DataChecksumsWorkerMain
+ *
+ * Main function for enabling checksums in a single database, This is the
+ * function set as the bgw_function_name in the dynamic background worker
+ * process initiated for each database by the worker launcher. After enabling
+ * data checksums in each applicable relation in the database, it will wait for
+ * all temporary relations that were present when the function started to
+ * disappear before returning. This is required since we cannot rewrite
+ * existing temporary relations with data checksums.
+ */
+void
+DataChecksumsWorkerMain(Datum arg)
+{
+	Oid			dboid = DatumGetObjectId(arg);
+	List	   *RelationList = NIL;
+	List	   *InitialTempTableList = NIL;
+	BufferAccessStrategy strategy;
+	bool		aborted = false;
+
+	enabling_checksums = true;
+
+	pqsignal(SIGTERM, die);
+
+	BackgroundWorkerUnblockSignals();
+
+	MyBackendType = B_DATACHECKSUMSWORKER_WORKER;
+	init_ps_display(NULL);
+
+	BackgroundWorkerInitializeConnectionByOid(dboid, InvalidOid,
+											  BGWORKER_BYPASS_ALLOWCONN);
+
+	/*
+	 * Get a list of all temp tables present as we start in this database. We
+	 * need to wait until they are all gone until we are done, since we cannot
+	 * access these relations and modify them.
+	 */
+	InitialTempTableList = BuildRelationList(true, false);
+
+	/*
+	 * Enable vacuum cost delay, if any.
+	 */
+	Assert(DataChecksumsWorkerShmem->enabling_checksums);
+	VacuumCostDelay = DataChecksumsWorkerShmem->cost_delay;
+	VacuumCostLimit = DataChecksumsWorkerShmem->cost_limit;
+	VacuumCostActive = (VacuumCostDelay > 0);
+	VacuumCostBalance = 0;
+	VacuumCostPageHit = 0;
+	VacuumCostPageMiss = 0;
+	VacuumCostPageDirty = 0;
+
+	/*
+	 * Create and set the vacuum strategy as our buffer strategy.
+	 */
+	strategy = GetAccessStrategy(BAS_VACUUM);
+
+	RelationList = BuildRelationList(false,
+									 DataChecksumsWorkerShmem->process_shared_catalogs);
+	foreach_oid(reloid, RelationList)
+	{
+		if (!ProcessSingleRelationByOid(reloid, strategy))
+		{
+			aborted = true;
+			break;
+		}
+	}
+	list_free(RelationList);
+
+	if (aborted)
+	{
+		DataChecksumsWorkerShmem->success = DATACHECKSUMSWORKER_ABORTED;
+		ereport(DEBUG1,
+				(errmsg("data checksum processing aborted in database OID %u",
+						dboid)));
+		return;
+	}
+
+	/*
+	 * Wait for all temp tables that existed when we started to go away. This
+	 * is necessary since we cannot "reach" them to enable checksums. Any temp
+	 * tables created after we started will already have checksums in them
+	 * (due to the "inprogress-on" state), so no need to wait for those.
+	 */
+	for (;;)
+	{
+		List	   *CurrentTempTables;
+		int			numleft;
+		char		activity[64];
+
+		CurrentTempTables = BuildRelationList(true, false);
+		numleft = 0;
+		foreach_oid(tmptbloid, InitialTempTableList)
+		{
+			if (list_member_oid(CurrentTempTables, tmptbloid))
+				numleft++;
+		}
+		list_free(CurrentTempTables);
+
+		if (numleft == 0)
+			break;
+
+		/*
+		 * At least one temp table is left to wait for, indicate in pgstat
+		 * activity and progress reporting.
+		 */
+		snprintf(activity,
+				 sizeof(activity),
+				 "Waiting for %d temp tables to be removed", numleft);
+		pgstat_report_activity(STATE_RUNNING, activity);
+
+		/* Retry every 5 seconds */
+		ResetLatch(MyLatch);
+		(void) WaitLatch(MyLatch,
+						 WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+						 5000,
+						 WAIT_EVENT_CHECKSUM_ENABLE_FINISHCONDITION);
+
+		LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE);
+		aborted = DataChecksumsWorkerShmem->launch_enable_checksums != enabling_checksums;
+		LWLockRelease(DataChecksumsWorkerLock);
+
+		if (aborted || abort_requested)
+		{
+			DataChecksumsWorkerShmem->success = DATACHECKSUMSWORKER_ABORTED;
+			ereport(DEBUG1,
+					(errmsg("data checksum processing aborted in database OID %u",
+							dboid)));
+			return;
+		}
+	}
+
+	list_free(InitialTempTableList);
+
+	DataChecksumsWorkerShmem->success = DATACHECKSUMSWORKER_SUCCESSFUL;
+}
diff --git a/src/backend/postmaster/meson.build b/src/backend/postmaster/meson.build
index 0008603cfee..ce10ef1059a 100644
--- a/src/backend/postmaster/meson.build
+++ b/src/backend/postmaster/meson.build
@@ -6,6 +6,7 @@ backend_sources += files(
   'bgworker.c',
   'bgwriter.c',
   'checkpointer.c',
+  'datachecksumsworker.c',
   'fork_process.c',
   'interrupt.c',
   'launch_backend.c',
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 24d88f368d8..00cce4c7673 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -186,6 +186,7 @@ xlog_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
 		case XLOG_FPW_CHANGE:
 		case XLOG_FPI_FOR_HINT:
 		case XLOG_FPI:
+		case XLOG_CHECKSUMS:
 		case XLOG_OVERWRITE_CONTRECORD:
 		case XLOG_CHECKPOINT_REDO:
 			break;
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 174eed70367..e7761a3ddb6 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -30,6 +30,8 @@
 #include "postmaster/autovacuum.h"
 #include "postmaster/bgworker_internals.h"
 #include "postmaster/bgwriter.h"
+#include "postmaster/datachecksumsworker.h"
+#include "postmaster/postmaster.h"
 #include "postmaster/walsummarizer.h"
 #include "replication/logicallauncher.h"
 #include "replication/origin.h"
@@ -148,6 +150,7 @@ CalculateShmemSize(int *num_semaphores)
 	size = add_size(size, WaitEventCustomShmemSize());
 	size = add_size(size, InjectionPointShmemSize());
 	size = add_size(size, SlotSyncShmemSize());
+	size = add_size(size, DataChecksumsWorkerShmemSize());
 
 	/* include additional requested shmem from preload libraries */
 	size = add_size(size, total_addin_request);
@@ -330,6 +333,7 @@ CreateOrAttachShmemStructs(void)
 	PgArchShmemInit();
 	ApplyLauncherShmemInit();
 	SlotSyncShmemInit();
+	DataChecksumsWorkerShmemInit();
 
 	/*
 	 * Set up other modules that need some shared memory space
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 7d201965503..2b13a8cd260 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -18,6 +18,7 @@
 #include <unistd.h>
 
 #include "access/parallel.h"
+#include "access/xlog.h"
 #include "commands/async.h"
 #include "miscadmin.h"
 #include "pgstat.h"
@@ -575,6 +576,18 @@ ProcessProcSignalBarrier(void)
 					case PROCSIGNAL_BARRIER_SMGRRELEASE:
 						processed = ProcessBarrierSmgrRelease();
 						break;
+					case PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON:
+						processed = AbsorbChecksumsOnInProgressBarrier();
+						break;
+					case PROCSIGNAL_BARRIER_CHECKSUM_ON:
+						processed = AbsorbChecksumsOnBarrier();
+						break;
+					case PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF:
+						processed = AbsorbChecksumsOffInProgressBarrier();
+						break;
+					case PROCSIGNAL_BARRIER_CHECKSUM_OFF:
+						processed = AbsorbChecksumsOffBarrier();
+						break;
 				}
 
 				/*
diff --git a/src/backend/storage/page/README b/src/backend/storage/page/README
index e30d7ac59ad..73c36a63908 100644
--- a/src/backend/storage/page/README
+++ b/src/backend/storage/page/README
@@ -10,7 +10,9 @@ http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf, discussed
 2010/12/22 on -hackers list.
 
 Current implementation requires this be enabled system-wide at initdb time, or
-by using the pg_checksums tool on an offline cluster.
+by using the pg_checksums tool on an offline cluster.  Checksums can also be
+enabled at runtime using pg_enable_data_checksums(), and disabled by using
+pg_disable_data_checksums().
 
 The checksum is not valid at all times on a data page!!
 The checksum is valid when the page leaves the shared pool and is checked
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index ecc81aacfc3..8bd5fed8c85 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -98,7 +98,7 @@ PageIsVerifiedExtended(PageData *page, BlockNumber blkno, int flags)
 	 */
 	if (!PageIsNew(page))
 	{
-		if (DataChecksumsEnabled())
+		if (DataChecksumsNeedVerify())
 		{
 			checksum = pg_checksum_page(page, blkno);
 
@@ -1501,7 +1501,7 @@ PageSetChecksumCopy(Page page, BlockNumber blkno)
 	static char *pageCopy = NULL;
 
 	/* If we don't need a checksum, just return the passed-in data */
-	if (PageIsNew(page) || !DataChecksumsEnabled())
+	if (PageIsNew(page) || !DataChecksumsNeedWrite())
 		return page;
 
 	/*
@@ -1531,7 +1531,7 @@ void
 PageSetChecksumInplace(Page page, BlockNumber blkno)
 {
 	/* If we don't need a checksum, just return */
-	if (PageIsNew(page) || !DataChecksumsEnabled())
+	if (PageIsNew(page) || !DataChecksumsNeedWrite())
 		return;
 
 	((PageHeader) page)->pd_checksum = pg_checksum_page(page, blkno);
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index eb575025596..e7b418a000e 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -370,6 +370,8 @@ pgstat_tracks_io_bktype(BackendType bktype)
 		case B_LOGGER:
 			return false;
 
+		case B_DATACHECKSUMSWORKER_LAUNCHER:
+		case B_DATACHECKSUMSWORKER_WORKER:
 		case B_AUTOVAC_LAUNCHER:
 		case B_AUTOVAC_WORKER:
 		case B_BACKEND:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index e199f071628..c7cd48e8d33 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -115,6 +115,8 @@ CHECKPOINT_DELAY_COMPLETE	"Waiting for a backend that blocks a checkpoint from c
 CHECKPOINT_DELAY_START	"Waiting for a backend that blocks a checkpoint from starting."
 CHECKPOINT_DONE	"Waiting for a checkpoint to complete."
 CHECKPOINT_START	"Waiting for a checkpoint to start."
+CHECKSUM_ENABLE_STARTCONDITION	"Waiting for data checksums enabling to start."
+CHECKSUM_ENABLE_FINISHCONDITION	"Waiting for data checksums to be enabled."
 EXECUTE_GATHER	"Waiting for activity from a child process while executing a <literal>Gather</literal> plan node."
 HASH_BATCH_ALLOCATE	"Waiting for an elected Parallel Hash participant to allocate a hash table."
 HASH_BATCH_ELECT	"Waiting to elect a Parallel Hash participant to allocate a hash table."
@@ -346,6 +348,7 @@ WALSummarizer	"Waiting to read or update WAL summarization state."
 DSMRegistry	"Waiting to read or update the dynamic shared memory registry."
 InjectionPoint	"Waiting to read or update information related to injection points."
 SerialControl	"Waiting to read or update shared <filename>pg_serial</filename> state."
+DataChecksumsWorker	"Waiting for data checksumsworker."
 
 #
 # END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 9172e1cb9d2..f6bd1e9f0ca 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -274,6 +274,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
 		cmdtype = PROGRESS_COMMAND_BASEBACKUP;
 	else if (pg_strcasecmp(cmd, "COPY") == 0)
 		cmdtype = PROGRESS_COMMAND_COPY;
+	else if (pg_strcasecmp(cmd, "DATACHECKSUMS") == 0)
+		cmdtype = PROGRESS_COMMAND_DATACHECKSUMS;
 	else
 		ereport(ERROR,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
@@ -1146,9 +1148,6 @@ pg_stat_get_db_checksum_failures(PG_FUNCTION_ARGS)
 	int64		result;
 	PgStat_StatDBEntry *dbentry;
 
-	if (!DataChecksumsEnabled())
-		PG_RETURN_NULL();
-
 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
 		result = 0;
 	else
@@ -1164,9 +1163,6 @@ pg_stat_get_db_checksum_last_failure(PG_FUNCTION_ARGS)
 	TimestampTz result;
 	PgStat_StatDBEntry *dbentry;
 
-	if (!DataChecksumsEnabled())
-		PG_RETURN_NULL();
-
 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
 		result = 0;
 	else
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index dc3521457c7..a071ba6f455 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -293,6 +293,12 @@ GetBackendTypeDesc(BackendType backendType)
 		case B_CHECKPOINTER:
 			backendDesc = gettext_noop("checkpointer");
 			break;
+		case B_DATACHECKSUMSWORKER_LAUNCHER:
+			backendDesc = "datachecksumsworker launcher";
+			break;
+		case B_DATACHECKSUMSWORKER_WORKER:
+			backendDesc = "datachecksumsworker worker";
+			break;
 		case B_LOGGER:
 			backendDesc = gettext_noop("logger");
 			break;
@@ -892,7 +898,8 @@ InitializeSessionUserIdStandalone(void)
 	 * workers, in slot sync worker and in background workers.
 	 */
 	Assert(!IsUnderPostmaster || AmAutoVacuumWorkerProcess() ||
-		   AmLogicalSlotSyncWorkerProcess() || AmBackgroundWorkerProcess());
+		   AmLogicalSlotSyncWorkerProcess() || AmBackgroundWorkerProcess() ||
+		   AmDataChecksumsWorkerProcess());
 
 	/* call only once */
 	Assert(!OidIsValid(AuthenticatedUserId));
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index ee1a9d5d98b..70899e6ef2d 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -739,6 +739,11 @@ InitPostgres(const char *in_dbname, Oid dboid,
 	 */
 	SharedInvalBackendInit(false);
 
+	/*
+	 * Set up backend local cache of Controldata values.
+	 */
+	InitLocalControldata();
+
 	ProcSignalInit(MyCancelKeyValid, MyCancelKey);
 
 	/*
@@ -869,7 +874,7 @@ InitPostgres(const char *in_dbname, Oid dboid,
 					 errhint("You should immediately run CREATE USER \"%s\" SUPERUSER;.",
 							 username != NULL ? username : "postgres")));
 	}
-	else if (AmBackgroundWorkerProcess())
+	else if (AmBackgroundWorkerProcess() || AmDataChecksumsWorkerProcess())
 	{
 		if (username == NULL && !OidIsValid(useroid))
 		{
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ad25cbb39c5..05cba3a02e3 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -476,6 +476,14 @@ static const struct config_enum_entry wal_compression_options[] = {
 	{NULL, 0, false}
 };
 
+static const struct config_enum_entry data_checksums_options[] = {
+	{"on", DATA_CHECKSUMS_ON, true},
+	{"off", DATA_CHECKSUMS_OFF, true},
+	{"inprogress-on", DATA_CHECKSUMS_INPROGRESS_ON, true},
+	{"inprogress-off", DATA_CHECKSUMS_INPROGRESS_OFF, true},
+	{NULL, 0, false}
+};
+
 /*
  * Options for enum values stored in other modules
  */
@@ -601,7 +609,7 @@ static int	shared_memory_size_mb;
 static int	shared_memory_size_in_huge_pages;
 static int	wal_block_size;
 static int	num_os_semaphores;
-static bool data_checksums;
+static int	data_checksums;
 static bool integer_datetimes;
 
 #ifdef USE_ASSERT_CHECKING
@@ -1952,17 +1960,6 @@ struct config_bool ConfigureNamesBool[] =
 		NULL, NULL, NULL
 	},
 
-	{
-		{"data_checksums", PGC_INTERNAL, PRESET_OPTIONS,
-			gettext_noop("Shows whether data checksums are turned on for this cluster."),
-			NULL,
-			GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_RUNTIME_COMPUTED
-		},
-		&data_checksums,
-		false,
-		NULL, NULL, NULL
-	},
-
 	{
 		{"syslog_sequence_numbers", PGC_SIGHUP, LOGGING_WHERE,
 			gettext_noop("Add sequence number to syslog messages to avoid duplicate suppression."),
@@ -5299,6 +5296,17 @@ struct config_enum ConfigureNamesEnum[] =
 		NULL, NULL, NULL
 	},
 
+	{
+		{"data_checksums", PGC_INTERNAL, PRESET_OPTIONS,
+			gettext_noop("Shows whether data checksums are turned on for this cluster."),
+			NULL,
+			GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+		},
+		&data_checksums,
+		DATA_CHECKSUMS_OFF, data_checksums_options,
+		NULL, NULL, show_data_checksums
+	},
+
 	/* End-of-list marker */
 	{
 		{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/bin/pg_checksums/pg_checksums.c b/src/bin/pg_checksums/pg_checksums.c
index 867aeddc601..79c3f86357e 100644
--- a/src/bin/pg_checksums/pg_checksums.c
+++ b/src/bin/pg_checksums/pg_checksums.c
@@ -576,7 +576,7 @@ main(int argc, char *argv[])
 		mode == PG_MODE_DISABLE)
 		pg_fatal("data checksums are already disabled in cluster");
 
-	if (ControlFile->data_checksum_version > 0 &&
+	if (ControlFile->data_checksum_version == DATA_CHECKSUMS_ON &&
 		mode == PG_MODE_ENABLE)
 		pg_fatal("data checksums are already enabled in cluster");
 
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index bd49ea867bf..f48936d3b00 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -14,6 +14,7 @@
 
 #include "common/string.h"
 #include "pg_upgrade.h"
+#include "storage/bufpage.h"

 /*
@@ -735,6 +736,14 @@ check_control_data(ControlData *oldctrl,
 	 * check_for_isn_and_int8_passing_mismatch().
 	 */
 
+	/*
+	 * If data checksums are in any in-progress state then disallow the
+	 * upgrade. The user should either let the process finish, or turn off
+	 * data checksums, before retrying.
+	 */
+	if (oldctrl->data_checksum_version > PG_DATA_CHECKSUM_VERSION)
+		pg_fatal("checksums are being enabled in the old cluster");
+
 	/*
 	 * We might eventually allow upgrades from checksum to no-checksum
 	 * clusters.
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d313099c027..0bb32c9c0fc 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -117,7 +117,7 @@ extern PGDLLIMPORT int wal_level;
  * of the bits make it to disk, but the checksum wouldn't match.  Also WAL-log
  * them if forced by wal_log_hints=on.
  */
-#define XLogHintBitIsNeeded() (DataChecksumsEnabled() || wal_log_hints)
+#define XLogHintBitIsNeeded() (wal_log_hints || DataChecksumsNeedWrite())
 
 /* Do we need to WAL-log information required only for Hot Standby and logical replication? */
 #define XLogStandbyInfoActive() (wal_level >= WAL_LEVEL_REPLICA)
@@ -230,7 +230,19 @@ extern XLogRecPtr GetXLogWriteRecPtr(void);
 
 extern uint64 GetSystemIdentifier(void);
 extern char *GetMockAuthenticationNonce(void);
-extern bool DataChecksumsEnabled(void);
+extern bool DataChecksumsNeedWrite(void);
+extern bool DataChecksumsNeedVerify(void);
+extern bool DataChecksumsOnInProgress(void);
+extern bool DataChecksumsOffInProgress(void);
+extern void SetDataChecksumsOnInProgress(void);
+extern void SetDataChecksumsOn(void);
+extern void SetDataChecksumsOff(void);
+extern bool AbsorbChecksumsOnInProgressBarrier(void);
+extern bool AbsorbChecksumsOffInProgressBarrier(void);
+extern bool AbsorbChecksumsOnBarrier(void);
+extern bool AbsorbChecksumsOffBarrier(void);
+extern const char *show_data_checksums(void);
+extern void InitLocalControldata(void);
 extern bool GetDefaultCharSignedness(void);
 extern XLogRecPtr GetFakeLSNForUnloggedRel(void);
 extern Size XLOGShmemSize(void);
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 2cf8d55d706..ee82d4ce73d 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -25,6 +25,7 @@
 #include "lib/stringinfo.h"
 #include "pgtime.h"
 #include "storage/block.h"
+#include "storage/checksum.h"
 #include "storage/relfilelocator.h"

@@ -289,6 +290,12 @@ typedef struct xl_restore_point
 	char		rp_name[MAXFNAMELEN];
 } xl_restore_point;
 
+/* Information logged when data checksum level is changed */
+typedef struct xl_checksum_state
+{
+	ChecksumType new_checksumtype;
+} xl_checksum_state;
+
 /* Overwrite of prior contrecord */
 typedef struct xl_overwrite_contrecord
 {
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 63e834a6ce4..9b5a50adf54 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -80,6 +80,7 @@ typedef struct CheckPoint
 /* 0xC0 is used in Postgres 9.5-11 */
 #define XLOG_OVERWRITE_CONTRECORD		0xD0
 #define XLOG_CHECKPOINT_REDO			0xE0
+#define XLOG_CHECKSUMS					0xF0

 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index cede992b6e2..fb0e062b984 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12246,6 +12246,23 @@
   proname => 'jsonb_subscript_handler', prorettype => 'internal',
   proargtypes => 'internal', prosrc => 'jsonb_subscript_handler' },
 
+# data checksum management functions
+{ oid => '9258',
+  descr => 'disable data checksums',
+  proname => 'pg_disable_data_checksums', provolatile => 'v', prorettype => 'void',
+  proparallel => 'r',
+  proargtypes => '',
+  prosrc => 'disable_data_checksums' },
+
+{ oid => '9257',
+  descr => 'enable data checksums',
+  proname => 'pg_enable_data_checksums', provolatile => 'v', prorettype => 'void',
+  proparallel => 'r',
+  proargtypes => 'int4 int4 bool', proallargtypes => '{int4,int4,bool}',
+  proargmodes => '{i,i,i}',
+  proargnames => '{cost_delay,cost_limit,fast}',
+  prosrc => 'enable_data_checksums' },
+
 # collation management functions
 { oid => '3445', descr => 'import collations from operating system',
   proname => 'pg_import_system_collations', procost => '100',
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 7c736e7b03b..1ebd0c792b4 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -157,4 +157,22 @@
 #define PROGRESS_COPY_TYPE_PIPE 3
 #define PROGRESS_COPY_TYPE_CALLBACK 4
 
+/* Progress parameters for PROGRESS_DATACHECKSUMS */
+#define PROGRESS_DATACHECKSUMS_PHASE 0
+#define PROGRESS_DATACHECKSUMS_TOTAL_DB 1
+#define PROGRESS_DATACHECKSUMS_TOTAL_REL 2
+#define PROGRESS_DATACHECKSUMS_PROCESSED_DB 3
+#define PROGRESS_DATACHECKSUMS_PROCESSED_REL 4
+#define PROGRESS_DATACHECKSUMS_CUR_DB 5
+#define PROGRESS_DATACHECKSUMS_CUR_REL 6
+#define PROGRESS_DATACHECKSUMS_CUR_REL_TOTAL_BLOCKS 7
+#define PROGRESS_DATACHECKSUMS_CUR_REL_PROCESSED_BLOCKS 8
+
+/* Phases of datachecksumsworker operation */
+#define PROGRESS_DATACHECKSUMS_PHASE_ENABLING 0
+#define PROGRESS_DATACHECKSUMS_PHASE_DISABLING 1
+#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_BACKENDS 2
+#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_TEMPREL 3
+#define PROGRESS_DATACHECKSUMS_DONE 4
+
 #endif
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..9923c7f518d 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -365,6 +365,9 @@ typedef enum BackendType
 	B_WAL_SUMMARIZER,
 	B_WAL_WRITER,
 
+	B_DATACHECKSUMSWORKER_LAUNCHER,
+	B_DATACHECKSUMSWORKER_WORKER,
+
 	/*
 	 * Logger is not connected to shared memory and does not have a PGPROC
 	 * entry.
@@ -389,6 +392,7 @@ extern PGDLLIMPORT BackendType MyBackendType;
 #define AmWalReceiverProcess()		(MyBackendType == B_WAL_RECEIVER)
 #define AmWalSummarizerProcess()	(MyBackendType == B_WAL_SUMMARIZER)
 #define AmWalWriterProcess()		(MyBackendType == B_WAL_WRITER)
+#define AmDataChecksumsWorkerProcess()	(MyBackendType == B_DATACHECKSUMSWORKER_LAUNCHER || MyBackendType == B_DATACHECKSUMSWORKER_WORKER)
 
 #define AmSpecialWorkerProcess() \
 	(AmAutoVacuumLauncherProcess() || \
diff --git a/src/include/postmaster/datachecksumsworker.h b/src/include/postmaster/datachecksumsworker.h
new file mode 100644
index 00000000000..59c9000d646
--- /dev/null
+++ b/src/include/postmaster/datachecksumsworker.h
@@ -0,0 +1,31 @@
+/*-------------------------------------------------------------------------
+ *
+ * datachecksumsworker.h
+ *	  header file for checksum helper background worker
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/datachecksumsworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef DATACHECKSUMSWORKER_H
+#define DATACHECKSUMSWORKER_H
+
+/* Shared memory */
+extern Size DataChecksumsWorkerShmemSize(void);
+extern void DataChecksumsWorkerShmemInit(void);
+
+/* Start the background processes for enabling or disabling checksums */
+void		StartDataChecksumsWorkerLauncher(bool enable_checksums,
+											 int cost_delay,
+											 int cost_limit,
+											 bool fast);
+
+/* Background worker entrypoints */
+void		DataChecksumsWorkerLauncherMain(Datum arg);
+void		DataChecksumsWorkerMain(Datum arg);
+
+#endif							/* DATACHECKSUMSWORKER_H */
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6646b6f6371..5326782171f 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -205,7 +205,17 @@ typedef PageHeaderData *PageHeader;
  * handling pages.
  */
 #define PG_PAGE_LAYOUT_VERSION		4
+
+/*
+ * Checksum version 0 is used for when data checksums are disabled.
+ * PG_DATA_CHECKSUM_VERSION defines that data checksums are enabled in the
+ * cluster and PG_DATA_CHECKSUM_INPROGRESS_{ON|OFF}_VERSION defines that data
+ * checksums are either currently being enabled or disabled.
+ */
 #define PG_DATA_CHECKSUM_VERSION	1
+#define PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION	2
+#define PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION	3
+
 
 /* ----------------------------------------------------------------
  *						page support functions
diff --git a/src/include/storage/checksum.h b/src/include/storage/checksum.h
index 25d13a798d1..6faff962ef0 100644
--- a/src/include/storage/checksum.h
+++ b/src/include/storage/checksum.h
@@ -15,6 +15,14 @@
 
 #include "storage/block.h"
 
+typedef enum ChecksumType
+{
+	DATA_CHECKSUMS_OFF = 0,
+	DATA_CHECKSUMS_ON,
+	DATA_CHECKSUMS_INPROGRESS_ON,
+	DATA_CHECKSUMS_INPROGRESS_OFF
+} ChecksumType;
+
 /*
  * Compute the checksum for a Postgres page.  The page must be aligned on a
  * 4-byte boundary.
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index cf565452382..afe39db753c 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -83,3 +83,4 @@ PG_LWLOCK(49, WALSummarizer)
 PG_LWLOCK(50, DSMRegistry)
 PG_LWLOCK(51, InjectionPoint)
 PG_LWLOCK(52, SerialControl)
+PG_LWLOCK(53, DataChecksumsWorker)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 114eb1f8f76..61a3e3af9d4 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -447,9 +447,10 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
  * Background writer, checkpointer, WAL writer, WAL summarizer, and archiver
  * run during normal operation.  Startup process and WAL receiver also consume
  * 2 slots, but WAL writer is launched only after startup has exited, so we
- * only need 6 slots.
+ * only need 6 slots to cover these. The DataChecksums worker and launcher
+ * can consume 2 slots when data checksums are enabled or disabled.
  */
-#define NUM_AUXILIARY_PROCS		6
+#define NUM_AUXILIARY_PROCS		8
 
 /* configurable options */
 extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 022fd8ed933..8937fa6ed3d 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -54,6 +54,11 @@ typedef enum
 typedef enum
 {
 	PROCSIGNAL_BARRIER_SMGRRELEASE, /* ask smgr to close files */
+
+	PROCSIGNAL_BARRIER_CHECKSUM_OFF,
+	PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON,
+	PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF,
+	PROCSIGNAL_BARRIER_CHECKSUM_ON,
 } ProcSignalBarrierType;
 
 /*
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index dda813ab407..c664e92dbfe 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -28,6 +28,7 @@ typedef enum ProgressCommandType
 	PROGRESS_COMMAND_CREATE_INDEX,
 	PROGRESS_COMMAND_BASEBACKUP,
 	PROGRESS_COMMAND_COPY,
+	PROGRESS_COMMAND_DATACHECKSUMS,
 } ProgressCommandType;
 
 #define PGSTAT_NUM_PROGRESS_PARAM	20
diff --git a/src/test/Makefile b/src/test/Makefile
index 511a72e6238..278ce3e8a86 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -12,7 +12,16 @@ subdir = src/test
 top_builddir = ../..
 include $(top_builddir)/src/Makefile.global
 
-SUBDIRS = perl postmaster regress isolation modules authentication recovery subscription
+SUBDIRS = \
+		perl \
+		postmaster \
+		regress \
+		isolation \
+		modules \
+		authentication \
+		recovery \
+		subscription \
+		checksum
 
 ifeq ($(with_icu),yes)
 SUBDIRS += icu
diff --git a/src/test/checksum/.gitignore b/src/test/checksum/.gitignore
new file mode 100644
index 00000000000..871e943d50e
--- /dev/null
+++ b/src/test/checksum/.gitignore
@@ -0,0 +1,2 @@
+# Generated by test suite
+/tmp_check/
diff --git a/src/test/checksum/Makefile b/src/test/checksum/Makefile
new file mode 100644
index 00000000000..fd03bf73df4
--- /dev/null
+++ b/src/test/checksum/Makefile
@@ -0,0 +1,23 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/checksum
+#
+# Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/checksum/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/checksum
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
+
+installcheck:
+	$(prove_installcheck)
+
+clean distclean maintainer-clean:
+	rm -rf tmp_check
diff --git a/src/test/checksum/README b/src/test/checksum/README
new file mode 100644
index 00000000000..0f0317060b3
--- /dev/null
+++ b/src/test/checksum/README
@@ -0,0 +1,22 @@
+src/test/checksum/README
+
+Regression tests for data checksums
+===================================
+
+This directory contains a test suite for enabling data checksums
+in a running cluster.
+
+Running the tests
+=================
+
+    make check
+
+or
+
+    make installcheck
+
+NOTE: This creates a temporary installation (in the case of "check"),
+with multiple nodes, be they master or standby(s) for the purpose of
+the tests.
+
+NOTE: This requires the --enable-tap-tests argument to configure.
diff --git a/src/test/checksum/meson.build b/src/test/checksum/meson.build
new file mode 100644
index 00000000000..5f96b5c246d
--- /dev/null
+++ b/src/test/checksum/meson.build
@@ -0,0 +1,15 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+tests += {
+  'name': 'checksums',
+  'sd': meson.current_source_dir(),
+  'bd': meson.current_build_dir(),
+  'tap': {
+    'tests': [
+      't/001_basic.pl',
+      't/002_restarts.pl',
+      't/003_standby_restarts.pl',
+      't/004_offline.pl',
+    ],
+  },
+}
diff --git a/src/test/checksum/t/001_basic.pl b/src/test/checksum/t/001_basic.pl
new file mode 100644
index 00000000000..4c64f6a14fc
--- /dev/null
+++ b/src/test/checksum/t/001_basic.pl
@@ -0,0 +1,88 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test suite for testing enabling data checksums in an online cluster
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# pg_enable_checksums take three params: cost_delay, cost_limit and fast. For
+# testing we always want to override the default value for 'fast' with True
+# which will cause immediate checkpoints. 0 and 100 are the defaults for
+# cost_delay and cost_limit which are fine to use for testing so let's keep
+# them.
+my $enable_params = '0, 100, true';
+
+# Initialize node with checksums disabled.
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init(no_data_checksums => 1);
+$node->start;
+
+# Create some content to have un-checksummed data in the cluster
+$node->safe_psql('postgres',
+	"CREATE TABLE t AS SELECT generate_series(1,10000) AS a;");
+
+# Ensure that checksums are turned off
+my $result = $node->safe_psql('postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
+);
+is($result, 'off', 'ensure checksums are disabled');
+
+# Enable data checksums
+$node->safe_psql('postgres',
+	"SELECT pg_enable_data_checksums($enable_params);");
+
+# Wait for checksums to become enabled
+$result = $node->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'on');
+is($result, 1, 'ensure checksums are enabled');
+
+# Run a dummy query just to make sure we can read back some data
+$result = $node->safe_psql('postgres', "SELECT count(*) FROM t");
+is($result, '10000', 'ensure checksummed pages can be read back');
+
+# Enable data checksums again which should be a no-op..
+$node->safe_psql('postgres',
+	"SELECT pg_enable_data_checksums($enable_params);");
+# ..and make sure we can still read/write data
+$node->safe_psql('postgres', "UPDATE t SET a = a + 1;");
+$result = $node->safe_psql('postgres', "SELECT count(*) FROM t");
+is($result, '10000', 'ensure checksummed pages can be read back');
+
+# Disable checksums again
+$node->safe_psql('postgres', "SELECT pg_disable_data_checksums();");
+
+$result = $node->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'off');
+is($result, 1, 'ensure checksums are disabled');
+
+# Test reading again
+$result = $node->safe_psql('postgres', "SELECT count(*) FROM t");
+is($result, '10000', 'ensure previously checksummed pages can be read back');
+
+# Re-enable checksums and make sure that the underlying data has changed to
+# ensure that checksums will be different.
+$node->safe_psql('postgres', "UPDATE t SET a = a + 1;");
+
+$node->safe_psql('postgres',
+	"SELECT pg_enable_data_checksums($enable_params);");
+$result = $node->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'on');
+is($result, 1, 'ensure checksums are enabled');
+
+# Run a dummy query just to make sure we can read back the data
+$result = $node->safe_psql('postgres', "SELECT count(*) FROM t");
+is($result, '10000', 'ensure checksummed pages can be read back');
+
+$node->stop;
+
+done_testing();
diff --git a/src/test/checksum/t/002_restarts.pl b/src/test/checksum/t/002_restarts.pl
new file mode 100644
index 00000000000..2697b722257
--- /dev/null
+++ b/src/test/checksum/t/002_restarts.pl
@@ -0,0 +1,92 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test suite for testing enabling data checksums in an online cluster with
+# restarting the processing
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# pg_enable_checksums take three params: cost_delay, cost_limit and fast. For
+# testing we always want to override the default value for 'fast' with True
+# which will cause immediate checkpoints. 0 and 100 are the defaults for
+# cost_delay and cost_limit which are fine to use for testing so let's keep
+# them.
+my $enable_params = '0, 100, true';
+
+# Initialize node with checksums disabled.
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init(no_data_checksums => 1);
+$node->start;
+
+# Create some content to have un-checksummed data in the cluster
+$node->safe_psql('postgres',
+	"CREATE TABLE t AS SELECT generate_series(1,10000) AS a;");
+
+# Ensure that checksums are disabled
+my $result = $node->safe_psql('postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
+);
+is($result, 'off', 'ensure checksums are disabled');
+
+# Create a barrier for checksumming to block on, in this case a pre-existing
+# temporary table which is kept open while processing is started. We can
+# accomplish this by setting up an interactive psql process which keeps the
+# temporary table created as we enable checksums in another psql process.
+
+my $bsession = $node->background_psql('postgres');
+$bsession->query_safe('CREATE TEMPORARY TABLE tt (a integer);');
+
+# In another session, make sure we can see the blocking temp table but start
+# processing anyways and check that we are blocked with a proper wait event.
+$result = $node->safe_psql('postgres',
+	"SELECT relpersistence FROM pg_catalog.pg_class WHERE relname = 'tt';");
+is($result, 't', 'ensure we can see the temporary table');
+
+$node->safe_psql('postgres',
+	"SELECT pg_enable_data_checksums($enable_params);");
+
+$result = $node->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'inprogress-on');
+is($result, '1', "ensure checksums aren't enabled yet");
+
+$bsession->quit;
+$node->stop;
+$node->start;
+
+$result = $node->safe_psql('postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
+);
+is($result, 'inprogress-on', "ensure checksums aren't enabled yet");
+
+$node->safe_psql('postgres',
+	"SELECT pg_enable_data_checksums($enable_params);");
+
+$result = $node->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'on');
+is($result, 1, 'ensure checksums are turned on');
+
+$result = $node->safe_psql('postgres', "SELECT count(*) FROM t");
+is($result, '10000', 'ensure checksummed pages can be read back');
+
+$result = $node->poll_query_until(
+	'postgres',
+	"SELECT count(*) FROM pg_stat_activity WHERE backend_type LIKE 'datachecksumsworker%';",
+	'0');
+is($result, 1, 'await datachecksums worker/launcher termination');
+
+$result = $node->safe_psql('postgres', "SELECT pg_disable_data_checksums();");
+$result = $node->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'off');
+is($result, 1, 'ensure checksums are turned off');
+
+done_testing();
diff --git a/src/test/checksum/t/003_standby_restarts.pl b/src/test/checksum/t/003_standby_restarts.pl
new file mode 100644
index 00000000000..6c0fe8f3bf8
--- /dev/null
+++ b/src/test/checksum/t/003_standby_restarts.pl
@@ -0,0 +1,131 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test suite for testing enabling data checksums in an online cluster with
+# streaming replication
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# pg_enable_checksums take three params: cost_delay, cost_limit and fast. For
+# testing we always want to override the default value for 'fast' with True
+# which will cause immediate checkpoints. 0 and 100 are the defaults for
+# cost_delay and cost_limit which are fine to use for testing so let's keep
+# them.
+my $enable_params = '0, 100, true';
+
+# Initialize primary node
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 1, no_data_checksums => 1);
+$node_primary->start;
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = PostgreSQL::Test::Cluster->new('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+	has_streaming => 1);
+$node_standby_1->start;
+
+# Create some content on the primary to have un-checksummed data in the cluster
+$node_primary->safe_psql('postgres',
+	"CREATE TABLE t AS SELECT generate_series(1,10000) AS a;");
+
+# Wait for standbys to catch up
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+	$node_primary->lsn('insert'));
+
+# Check that checksums are turned off on all nodes
+my $result = $node_primary->safe_psql('postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
+);
+is($result, "off", 'ensure checksums are turned off on primary');
+
+$result = $node_standby_1->safe_psql('postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
+);
+is($result, "off", 'ensure checksums are turned off on standby_1');
+
+# Enable checksums for the cluster
+$node_primary->safe_psql('postgres',
+	"SELECT pg_enable_data_checksums($enable_params);");
+
+# Ensure that the primary switches to "inprogress-on"
+$result = $node_primary->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	"inprogress-on");
+is($result, 1, 'ensure checksums are in progress on primary');
+
+# Wait for checksum enable to be replayed
+$node_primary->wait_for_catchup($node_standby_1, 'replay');
+
+# Ensure that the standby has switched to "inprogress-on" or "on".  Normally it
+# would be "inprogress-on", but it is theoretically possible for the primary to
+# complete the checksum enabling *and* have the standby replay that record
+# before we reach the check below.
+$result = $node_standby_1->poll_query_until(
+	'postgres',
+	"SELECT setting = 'off' FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'f');
+is($result, 1, 'ensure standby has absorbed the inprogress-on barrier');
+$result = $node_standby_1->safe_psql('postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
+);
+
+is(($result eq 'inprogress-on' || $result eq 'on'),
+	1, 'ensure checksums are on, or in progress, on standby_1');
+
+# Insert some more data which should be checksummed on INSERT
+$node_primary->safe_psql('postgres',
+	"INSERT INTO t VALUES (generate_series(1, 10000));");
+
+# Wait for checksums enabled on the primary
+$result = $node_primary->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'on');
+is($result, 1, 'ensure checksums are enabled on the primary');
+
+# Wait for checksums enabled on the standby
+$result = $node_standby_1->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'on');
+is($result, 1, 'ensure checksums are enabled on the standby');
+
+$result = $node_primary->safe_psql('postgres', "SELECT count(a) FROM t");
+is($result, '20000', 'ensure we can safely read all data with checksums');
+
+$result = $node_primary->poll_query_until(
+	'postgres',
+	"SELECT count(*) FROM pg_stat_activity WHERE backend_type LIKE 'datachecksumsworker%';",
+	'0');
+is($result, 1, 'await datachecksums worker/launcher termination');
+
+# Disable checksums and ensure it's propagated to standby and that we can
+# still read all data
+$node_primary->safe_psql('postgres', "SELECT pg_disable_data_checksums();");
+# Wait for checksum disable to be replayed
+$node_primary->wait_for_catchup($node_standby_1, 'replay');
+$result = $node_primary->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'off');
+is($result, 1, 'ensure data checksums are disabled on the primary 2');
+
+# Ensure that the standby has switched to off
+$result = $node_standby_1->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'off');
+is($result, 1, 'ensure checksums are off on standby_1');
+
+$result = $node_primary->safe_psql('postgres', "SELECT count(a) FROM t");
+is($result, "20000", 'ensure we can safely read all data without checksums');
+
+done_testing();
diff --git a/src/test/checksum/t/004_offline.pl b/src/test/checksum/t/004_offline.pl
new file mode 100644
index 00000000000..9cee62c9b52
--- /dev/null
+++ b/src/test/checksum/t/004_offline.pl
@@ -0,0 +1,101 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test suite for testing enabling data checksums offline from various states
+# of checksum processing
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# pg_enable_checksums take three params: cost_delay, cost_limit and fast. For
+# testing we always want to override the default value for 'fast' with True
+# which will cause immediate checkpoints. 0 and 100 are the defaults for
+# cost_delay and cost_limit which are fine to use for testing so let's keep
+# them.
+my $enable_params = '0, 100, true';
+
+# Initialize node with checksums disabled.
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init(no_data_checksums => 1);
+$node->start;
+
+# Create some content to have un-checksummed data in the cluster
+$node->safe_psql('postgres',
+	"CREATE TABLE t AS SELECT generate_series(1,10000) AS a;");
+
+# Ensure that checksums are disabled
+my $result = $node->safe_psql('postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
+);
+is($result, 'off', 'ensure checksums are disabled');
+
+# Enable checksums offline using pg_checksums
+$node->stop;
+$node->checksum_enable_offline;
+$node->start;
+
+# Ensure that checksums are enabled
+$result = $node->safe_psql('postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
+);
+is($result, 'on', 'ensure checksums are enabled');
+
+# Run a dummy query just to make sure we can read back some data
+$result = $node->safe_psql('postgres', "SELECT count(*) FROM t");
+is($result, '10000', 'ensure checksummed pages can be read back');
+
+# Disable checksums offline again using pg_checksums
+$node->stop;
+$node->checksum_disable_offline;
+$node->start;
+
+# Ensure that checksums are disabled
+$result = $node->safe_psql('postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
+);
+is($result, 'off', 'ensure checksums are disabled');
+
+# Create a barrier for checksumming to block on, in this case a pre-existing
+# temporary table which is kept open while processing is started. We can
+# accomplish this by setting up an interactive psql process which keeps the
+# temporary table created as we enable checksums in another psql process.
+
+my $bsession = $node->background_psql('postgres');
+$bsession->query_safe('CREATE TEMPORARY TABLE tt (a integer);');
+
+# In another session, make sure we can see the blocking temp table but start
+# processing anyways and check that we are blocked with a proper wait event.
+$result = $node->safe_psql('postgres',
+	"SELECT relpersistence FROM pg_catalog.pg_class WHERE relname = 'tt';");
+is($result, 't', 'ensure we can see the temporary table');
+
+$node->safe_psql('postgres',
+	"SELECT pg_enable_data_checksums($enable_params);");
+
+$result = $node->poll_query_until(
+	'postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
+	'inprogress-on');
+is($result, 1, 'ensure checksums are in the process of being enabled');
+
+# Turn the cluster off and enable checksums offline, then start back up
+$bsession->quit;
+$node->stop;
+$node->checksum_enable_offline;
+$node->start;
+
+# Ensure that checksums are now enabled even though processing wasn't
+# restarted
+$result = $node->safe_psql('postgres',
+	"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
+);
+is($result, 'on', 'ensure checksums are enabled');
+
+# Run a dummy query just to make sure we can read back some data
+$result = $node->safe_psql('postgres', "SELECT count(*) FROM t");
+is($result, '10000', 'ensure checksummed pages can be read back');
+
+done_testing();
diff --git a/src/test/meson.build b/src/test/meson.build
index ccc31d6a86a..c4bfef2c0e2 100644
--- a/src/test/meson.build
+++ b/src/test/meson.build
@@ -8,6 +8,7 @@ subdir('postmaster')
 subdir('recovery')
 subdir('subscription')
 subdir('modules')
+subdir('checksum')
 
 if ssl.found()
   subdir('ssl')
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index b105cba05a6..666bd2a2d4c 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -3748,6 +3748,40 @@ sub advance_wal
 	}
 }
 
+=item $node->checksum_enable_offline()
+
+Enable data page checksums in an offline cluster with B<pg_checksums>. The
+caller is responsible for ensuring that the cluster is in the right state for
+this operation.
+
+=cut
+
+sub checksum_enable_offline
+{
+	my ($self) = @_;
+
+	print "### Enabling checksums in \"$self->data_dir\"\n";
+	PostgreSQL::Test::Utils::system_or_bail('pg_checksums', '-D', $self->data_dir, '-e');
+	return;
+}
+
+=item checksum_disable_offline
+
+Disable data page checksums in an offline cluster with B<pg_checksums>. The
+caller is responsible for ensuring that the cluster is in the right state for
+this operation.
+
+=cut
+
+sub checksum_disable_offline
+{
+	my ($self) = @_;
+
+	print "### Disabling checksums in \"$self->data_dir\"\n";
+	PostgreSQL::Test::Utils::system_or_bail('pg_checksums', '-D', $self->data_dir, '-d');
+	return;
+}
+
 =pod
 
 =back
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62f69ac20b2..36bed4b168d 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2041,6 +2041,34 @@ pg_stat_progress_create_index| SELECT s.pid,
     s.param15 AS partitions_done
    FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
      LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_data_checksums| SELECT s.pid,
+    s.datid,
+    d.datname,
+        CASE s.param1
+            WHEN 0 THEN 'enabling'::text
+            WHEN 1 THEN 'disabling'::text
+            WHEN 2 THEN 'waiting'::text
+            WHEN 3 THEN 'waiting on backends'::text
+            WHEN 4 THEN 'waiting on temporary tables'::text
+            WHEN 5 THEN 'done'::text
+            ELSE NULL::text
+        END AS phase,
+        CASE s.param2
+            WHEN '-1'::integer THEN NULL::bigint
+            ELSE s.param2
+        END AS databases_total,
+        CASE s.param3
+            WHEN '-1'::integer THEN NULL::bigint
+            ELSE s.param3
+        END AS relations_total,
+    s.param4 AS databases_processed,
+    s.param5 AS relations_processed,
+    s.param6 AS databases_current,
+    s.param7 AS relation_current,
+    s.param8 AS relation_current_blocks,
+    s.param9 AS relation_current_blocks_processed
+   FROM (pg_stat_get_progress_info('DATACHECKSUMS'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+     LEFT JOIN pg_database d ON ((s.datid = d.oid)));
 pg_stat_progress_vacuum| SELECT s.pid,
     s.datid,
     d.datname,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9840060997f..86e4057f61d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -403,6 +403,7 @@ CheckPointStmt
 CheckpointStatsData
 CheckpointerRequest
 CheckpointerShmemStruct
+ChecksumType
 Chromosome
 CkptSortItem
 CkptTsStatus
@@ -593,6 +594,10 @@ DataPageDeleteStack
 DataTypesUsageChecks
 DataTypesUsageVersionCheck
 DatabaseInfo
+DataChecksumsWorkerDatabase
+DataChecksumsWorkerResult
+DataChecksumsWorkerResultEntry
+DataChecksumsWorkerShmemStruct
 DateADT
 DateTimeErrorExtra
 Datum
@@ -4141,6 +4146,7 @@ xl_btree_split
 xl_btree_unlink_page
 xl_btree_update
 xl_btree_vacuum
+xl_checksum_state
 xl_clog_truncate
 xl_commit_ts_truncate
 xl_dbase_create_file_copy_rec
-- 
2.48.1

From b72c6ed49314783a421239d350e1a05057b01da1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 20:19:29 +0100
Subject: [PATCH v20250309 2/5] simple post-rebase fixes

- Update checks in PostmasterStateMachine to account for datachecksum
  workers, etc.

- Remove pgstat_bestart() call - it would need to be _initial(), but I
  don't think it's needed.

- Update vacuum_delay_point() call.
---
 src/backend/postmaster/datachecksumsworker.c | 5 +----
 src/backend/postmaster/postmaster.c          | 5 +++++
 src/backend/utils/activity/pgstat_backend.c  | 2 ++
 3 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/src/backend/postmaster/datachecksumsworker.c b/src/backend/postmaster/datachecksumsworker.c
index de7a077f9c2..b9d003423c0 100644
--- a/src/backend/postmaster/datachecksumsworker.c
+++ b/src/backend/postmaster/datachecksumsworker.c
@@ -450,7 +450,7 @@ ProcessSingleRelationFork(Relation reln, ForkNumber forkNum, BufferAccessStrateg
 		if (abort_requested)
 			return false;
 
-		vacuum_delay_point();
+		vacuum_delay_point(false);
 	}
 
 	pfree(relns);
@@ -752,9 +752,6 @@ DataChecksumsWorkerLauncherMain(Datum arg)
 	 */
 	BackgroundWorkerInitializeConnectionByOid(InvalidOid, InvalidOid, 0);
 
-	/* Initialize backend status information */
-	pgstat_bestart();
-
 	enabling_checksums = DataChecksumsWorkerShmem->launch_enable_checksums;
 	DataChecksumsWorkerShmem->launcher_running = true;
 	DataChecksumsWorkerShmem->enabling_checksums = enabling_checksums;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index d2a7a7add6f..2fc438987b5 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2947,6 +2947,11 @@ PostmasterStateMachine(void)
 									B_INVALID,
 									B_STANDALONE_BACKEND);
 
+			/* also add checksumming processes */
+			remainMask = btmask_add(remainMask,
+									B_DATACHECKSUMSWORKER_LAUNCHER,
+									B_DATACHECKSUMSWORKER_WORKER);
+
 			/* All types should be included in targetMask or remainMask */
 			Assert((remainMask.mask | targetMask.mask) == BTYPE_MASK_ALL.mask);
 		}
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index a9343b7b59e..f4b2f3b91d8 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -292,6 +292,8 @@ pgstat_tracks_backend_bktype(BackendType bktype)
 		case B_BG_WRITER:
 		case B_CHECKPOINTER:
 		case B_STARTUP:
+		case B_DATACHECKSUMSWORKER_LAUNCHER:
+		case B_DATACHECKSUMSWORKER_WORKER:
 			return false;
 
 		case B_AUTOVAC_WORKER:
-- 
2.48.1

From 9257d742294b27a4ee559fa31459fa0afa502b6b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 23:07:44 +0100
Subject: [PATCH v20250309 3/5] sync the data_checksums GUC with the local
 variable

We now have three places that in some way express the state of data
checksums in the instance.

- control file / data_checksum_version

- LocalDataChecksumVersion as a local cache to reduce locking

- data_checksums backing the GUC

We need to keep the GUC variable in sync, to ensure we get the correct
value even early during startup (e.g. with -C).

Introduces a new SetLocalDataChecksumVersion() which sets the local
variable, and also updates the data_checksum GUC at the same time.
---
 src/backend/access/transam/xlog.c   | 51 +++++++++++++++++++++++++----
 src/backend/utils/misc/guc_tables.c |  3 +-
 src/include/access/xlog.h           |  1 +
 3 files changed, 47 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index b5190a7e104..f137cdc6d42 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -657,6 +657,14 @@ static bool updateMinRecoveryPoint = true;
  */
 static uint32 LocalDataChecksumVersion = 0;
 
+/*
+ * Variable backing the GUC, keep it in sync with LocalDataChecksumVersion.
+ * See SetLocalDataChecksumVersion().
+ */
+int			data_checksums = 0;
+
+static void SetLocalDataChecksumVersion(uint32 data_checksum_version);
+
 /* For WALInsertLockAcquire/Release functions */
 static int	MyLockNo = 0;
 static bool holdingAllLocks = false;
@@ -4594,7 +4602,7 @@ ReadControlFile(void)
 
 	CalculateCheckpointSegments();
 
-	LocalDataChecksumVersion = ControlFile->data_checksum_version;
+	SetLocalDataChecksumVersion(ControlFile->data_checksum_version);
 }
 
 /*
@@ -4913,7 +4921,7 @@ SetDataChecksumsOff(void)
 bool
 AbsorbChecksumsOnInProgressBarrier(void)
 {
-	LocalDataChecksumVersion = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION;
+	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
 	return true;
 }
 
@@ -4921,21 +4929,21 @@ bool
 AbsorbChecksumsOnBarrier(void)
 {
 	Assert(LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
-	LocalDataChecksumVersion = PG_DATA_CHECKSUM_VERSION;
+	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_VERSION);
 	return true;
 }
 
 bool
 AbsorbChecksumsOffInProgressBarrier(void)
 {
-	LocalDataChecksumVersion = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION;
+	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);
 	return true;
 }
 
 bool
 AbsorbChecksumsOffBarrier(void)
 {
-	LocalDataChecksumVersion = 0;
+	SetLocalDataChecksumVersion(0);
 	return true;
 }
 
@@ -4951,10 +4959,41 @@ void
 InitLocalControldata(void)
 {
 	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
-	LocalDataChecksumVersion = ControlFile->data_checksum_version;
+	SetLocalDataChecksumVersion(ControlFile->data_checksum_version);
 	LWLockRelease(ControlFileLock);
 }
 
+/*
+ * XXX probably should be called in all places that modify the value of
+ * LocalDataChecksumVersion (to make sure data_checksums GUC is in sync)
+ *
+ * XXX aren't PG_DATA_ and DATA_ constants the same? why do we need both?
+ */
+void
+SetLocalDataChecksumVersion(uint32 data_checksum_version)
+{
+	LocalDataChecksumVersion = data_checksum_version;
+
+	switch (LocalDataChecksumVersion)
+	{
+		case PG_DATA_CHECKSUM_VERSION:
+			data_checksums = DATA_CHECKSUMS_ON;
+			break;
+
+		case PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION:
+			data_checksums = DATA_CHECKSUMS_INPROGRESS_ON;
+			break;
+
+		case PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION:
+			data_checksums = DATA_CHECKSUMS_INPROGRESS_OFF;
+			break;
+
+		default:
+			data_checksums = DATA_CHECKSUMS_OFF;
+			break;
+	}
+}
+
 /* guc hook */
 const char *
 show_data_checksums(void)
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 05cba3a02e3..66abf056159 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -609,7 +609,6 @@ static int	shared_memory_size_mb;
 static int	shared_memory_size_in_huge_pages;
 static int	wal_block_size;
 static int	num_os_semaphores;
-static int	data_checksums;
 static bool integer_datetimes;
 
 #ifdef USE_ASSERT_CHECKING
@@ -5300,7 +5299,7 @@ struct config_enum ConfigureNamesEnum[] =
 		{"data_checksums", PGC_INTERNAL, PRESET_OPTIONS,
 			gettext_noop("Shows whether data checksums are turned on for this cluster."),
 			NULL,
-			GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+			GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_RUNTIME_COMPUTED
 		},
 		&data_checksums,
 		DATA_CHECKSUMS_OFF, data_checksums_options,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 0bb32c9c0fc..aec3ea0bc63 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -56,6 +56,7 @@ extern PGDLLIMPORT int CommitDelay;
 extern PGDLLIMPORT int CommitSiblings;
 extern PGDLLIMPORT bool track_wal_io_timing;
 extern PGDLLIMPORT int wal_decode_buffer_size;
+extern PGDLLIMPORT int data_checksums;
 
 extern PGDLLIMPORT int CheckPointSegments;
 
-- 
2.48.1

From aa8a3867b3316a1fa379b1e70553057d5d96fedc Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Sat, 8 Mar 2025 19:34:50 +0100
Subject: [PATCH v20250309 4/5] make progress reporting work

- Splits the progress status by worker type - one row for launcher,
  one row for checksum worker (might be more with parallel workers).

- The launcher only updates database counters, the workers only set
  relation/block counters.

- Also reworks the columns in the system view a bit, discards the
  "current" fields (we still know the database for each worker).

- Issue: Not sure what to do about relation forks, at this point it
  tracks only blocks for MAIN fork.
---
 src/backend/catalog/system_views.sql         | 36 ++++----
 src/backend/postmaster/datachecksumsworker.c | 91 +++++++++++++++++---
 src/include/commands/progress.h              | 27 +++---
 3 files changed, 112 insertions(+), 42 deletions(-)

diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 0d62976dc1f..4330d0ad656 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1335,25 +1335,25 @@ CREATE VIEW pg_stat_progress_copy AS
         LEFT JOIN pg_database D ON S.datid = D.oid;
 
 CREATE VIEW pg_stat_progress_data_checksums AS
-	SELECT
-		S.pid AS pid, S.datid AS datid, D.datname AS datname,
-		CASE S.param1 WHEN 0 THEN 'enabling'
+    SELECT
+        S.pid AS pid, S.datid, D.datname AS datname,
+        CASE S.param1 WHEN 0 THEN 'enabling'
                       WHEN 1 THEN 'disabling'
-					  WHEN 2 THEN 'waiting'
-					  WHEN 3 THEN 'waiting on backends'
-					  WHEN 4 THEN 'waiting on temporary tables'
-					  WHEN 5 THEN 'done'
-					  END AS phase,
-		CASE S.param2 WHEN -1 THEN NULL ELSE S.param2 END AS databases_total,
-		CASE S.param3 WHEN -1 THEN NULL ELSE S.param3 END AS relations_total,
-		S.param4 AS databases_processed,
-		S.param5 AS relations_processed,
-		S.param6 AS databases_current,
-		S.param7 AS relation_current,
-		S.param8 AS relation_current_blocks,
-		S.param9 AS relation_current_blocks_processed
-	FROM pg_stat_get_progress_info('DATACHECKSUMS') AS S
-		LEFT JOIN pg_database D ON S.datid = D.oid;
+                      WHEN 2 THEN 'waiting'
+                      WHEN 3 THEN 'waiting on backends'
+                      WHEN 4 THEN 'waiting on temporary tables'
+                      WHEN 5 THEN 'waiting on checkpoint'
+                      WHEN 6 THEN 'done'
+                      END AS phase,
+        CASE S.param2 WHEN -1 THEN NULL ELSE S.param2 END AS databases_total,
+        S.param3 AS databases_done,
+        CASE S.param4 WHEN -1 THEN NULL ELSE S.param4 END AS relations_total,
+        CASE S.param5 WHEN -1 THEN NULL ELSE S.param5 END AS relations_done,
+        CASE S.param6 WHEN -1 THEN NULL ELSE S.param6 END AS blocks_total,
+        CASE S.param7 WHEN -1 THEN NULL ELSE S.param7 END AS blocks_done
+    FROM pg_stat_get_progress_info('DATACHECKSUMS') AS S
+        LEFT JOIN pg_database D ON S.datid = D.oid
+    ORDER BY S.datid; -- return the launcher process first
 
 CREATE VIEW pg_user_mappings AS
     SELECT
diff --git a/src/backend/postmaster/datachecksumsworker.c b/src/backend/postmaster/datachecksumsworker.c
index b9d003423c0..f79dc290b2b 100644
--- a/src/backend/postmaster/datachecksumsworker.c
+++ b/src/backend/postmaster/datachecksumsworker.c
@@ -409,6 +409,11 @@ ProcessSingleRelationFork(Relation reln, ForkNumber forkNum, BufferAccessStrateg
 			 relns, RelationGetRelationName(reln), forkNames[forkNum], numblocks);
 	pgstat_report_activity(STATE_RUNNING, activity);
 
+	/* XXX only do this for main forks, maybe we should do it for all? */
+	if (forkNum == MAIN_FORKNUM)
+		pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_BLOCKS_TOTAL,
+									 numblocks);
+
 	/*
 	 * We are looping over the blocks which existed at the time of process
 	 * start, which is safe since new blocks are created with checksums set
@@ -450,6 +455,11 @@ ProcessSingleRelationFork(Relation reln, ForkNumber forkNum, BufferAccessStrateg
 		if (abort_requested)
 			return false;
 
+		/* XXX only do this for main forks, maybe we should do it for all? */
+		if (forkNum == MAIN_FORKNUM)
+			pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_BLOCKS_DONE,
+										 (blknum + 1));
+
 		vacuum_delay_point(false);
 	}
 
@@ -798,6 +808,8 @@ again:
 		pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,
 									 PROGRESS_DATACHECKSUMS_PHASE_WAITING_BACKENDS);
 
+		/* XXX isn't it weird there's no wait between the phase updates? */
+
 		/*
 		 * Set the state to inprogress-on and wait on the procsignal barrier.
 		 */
@@ -908,8 +920,29 @@ ProcessAllDatabases(bool immediate_checkpoint)
 	 * columns for processed databases is instead increased such that it can
 	 * be compared against the total.
 	 */
-	pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_TOTAL_DB,
-								 list_length(DatabaseList));
+	{
+		const int	index[] = {
+			PROGRESS_DATACHECKSUMS_DBS_TOTAL,
+			PROGRESS_DATACHECKSUMS_DBS_DONE,
+			PROGRESS_DATACHECKSUMS_RELS_TOTAL,
+			PROGRESS_DATACHECKSUMS_RELS_DONE,
+			PROGRESS_DATACHECKSUMS_BLOCKS_TOTAL,
+			PROGRESS_DATACHECKSUMS_BLOCKS_DONE,
+		};
+
+		int64	vals[6];
+
+		vals[0] = list_length(DatabaseList);
+		vals[1] = 0;
+
+		/* translated to NULL */
+		vals[2] = -1;
+		vals[3] = -1;
+		vals[4] = -1;
+		vals[5] = -1;
+
+		pgstat_progress_update_multi_param(6, index, vals);
+	}
 
 	while (true)
 	{
@@ -921,14 +954,6 @@ ProcessAllDatabases(bool immediate_checkpoint)
 			DataChecksumsWorkerResultEntry *entry;
 			bool		found;
 
-			/*
-			 * Indicate which database is being processed set the number of
-			 * relations to -1 to clear field from previous values. -1 will
-			 * translate to NULL in the progress view.
-			 */
-			pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_CUR_DB, db->dboid);
-			pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_TOTAL_REL, -1);
-
 			/*
 			 * Check if this database has been processed already, and if so
 			 * whether it should be retried or skipped.
@@ -957,6 +982,12 @@ ProcessAllDatabases(bool immediate_checkpoint)
 			result = ProcessDatabase(db);
 			processed_databases++;
 
+			/*
+			 * Update the number of processed databases in the progress report.
+			 */
+			pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_DBS_DONE,
+										 processed_databases);
+
 			if (result == DATACHECKSUMSWORKER_SUCCESSFUL)
 			{
 				/*
@@ -1050,6 +1081,13 @@ ProcessAllDatabases(bool immediate_checkpoint)
 				 errhint("The server log might have more information on the cause of the error.")));
 	}
 
+	/*
+	 * When enabling checksums, we have to wait for a checkpoint for the
+	 * checksums to e.
+	 */
+	pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,
+								 PROGRESS_DATACHECKSUMS_PHASE_WAITING_CHECKPOINT);
+
 	/*
 	 * Force a checkpoint to get everything out to disk. The use of immediate
 	 * checkpoints is for running tests, as they would otherwise not execute
@@ -1255,6 +1293,7 @@ DataChecksumsWorkerMain(Datum arg)
 	List	   *InitialTempTableList = NIL;
 	BufferAccessStrategy strategy;
 	bool		aborted = false;
+	int64		rels_done;
 
 	enabling_checksums = true;
 
@@ -1268,6 +1307,10 @@ DataChecksumsWorkerMain(Datum arg)
 	BackgroundWorkerInitializeConnectionByOid(dboid, InvalidOid,
 											  BGWORKER_BYPASS_ALLOWCONN);
 
+	/* worker will have a separate entry in pg_stat_progress_data_checksums */
+	pgstat_progress_start_command(PROGRESS_COMMAND_DATACHECKSUMS,
+								  InvalidOid);
+
 	/*
 	 * Get a list of all temp tables present as we start in this database. We
 	 * need to wait until they are all gone until we are done, since we cannot
@@ -1294,6 +1337,24 @@ DataChecksumsWorkerMain(Datum arg)
 
 	RelationList = BuildRelationList(false,
 									 DataChecksumsWorkerShmem->process_shared_catalogs);
+
+	/* Update the total number of relations to be processed in this DB. */
+	{
+		const int	index[] = {
+			PROGRESS_DATACHECKSUMS_RELS_TOTAL,
+			PROGRESS_DATACHECKSUMS_RELS_DONE
+		};
+
+		int64	vals[2];
+
+		vals[0] = list_length(RelationList);
+		vals[1] = 0;
+
+		pgstat_progress_update_multi_param(2, index, vals);
+	}
+
+	/* process the relations */
+	rels_done = 0;
 	foreach_oid(reloid, RelationList)
 	{
 		if (!ProcessSingleRelationByOid(reloid, strategy))
@@ -1301,6 +1362,9 @@ DataChecksumsWorkerMain(Datum arg)
 			aborted = true;
 			break;
 		}
+
+		pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_RELS_DONE,
+									 ++rels_done);
 	}
 	list_free(RelationList);
 
@@ -1313,6 +1377,10 @@ DataChecksumsWorkerMain(Datum arg)
 		return;
 	}
 
+	/* The worker is about to wait for temporary tables to go away. */
+	pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,
+								 PROGRESS_DATACHECKSUMS_PHASE_WAITING_TEMPREL);
+
 	/*
 	 * Wait for all temp tables that existed when we started to go away. This
 	 * is necessary since we cannot "reach" them to enable checksums. Any temp
@@ -1369,5 +1437,8 @@ DataChecksumsWorkerMain(Datum arg)
 
 	list_free(InitialTempTableList);
 
+	/* worker done */
+	pgstat_progress_end_command();
+
 	DataChecksumsWorkerShmem->success = DATACHECKSUMSWORKER_SUCCESSFUL;
 }
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 1ebd0c792b4..94b478a6cc9 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -158,21 +158,20 @@
 #define PROGRESS_COPY_TYPE_CALLBACK 4
 
 /* Progress parameters for PROGRESS_DATACHECKSUMS */
-#define PROGRESS_DATACHECKSUMS_PHASE 0
-#define PROGRESS_DATACHECKSUMS_TOTAL_DB 1
-#define PROGRESS_DATACHECKSUMS_TOTAL_REL 2
-#define PROGRESS_DATACHECKSUMS_PROCESSED_DB 3
-#define PROGRESS_DATACHECKSUMS_PROCESSED_REL 4
-#define PROGRESS_DATACHECKSUMS_CUR_DB 5
-#define PROGRESS_DATACHECKSUMS_CUR_REL 6
-#define PROGRESS_DATACHECKSUMS_CUR_REL_TOTAL_BLOCKS 7
-#define PROGRESS_DATACHECKSUMS_CUR_REL_PROCESSED_BLOCKS 8
+#define PROGRESS_DATACHECKSUMS_PHASE		0
+#define PROGRESS_DATACHECKSUMS_DBS_TOTAL	1
+#define PROGRESS_DATACHECKSUMS_DBS_DONE		2
+#define PROGRESS_DATACHECKSUMS_RELS_TOTAL	3
+#define PROGRESS_DATACHECKSUMS_RELS_DONE	4
+#define PROGRESS_DATACHECKSUMS_BLOCKS_TOTAL	5
+#define PROGRESS_DATACHECKSUMS_BLOCKS_DONE	6
 
 /* Phases of datachecksumsworker operation */
-#define PROGRESS_DATACHECKSUMS_PHASE_ENABLING 0
-#define PROGRESS_DATACHECKSUMS_PHASE_DISABLING 1
-#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_BACKENDS 2
-#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_TEMPREL 3
-#define PROGRESS_DATACHECKSUMS_DONE 4
+#define PROGRESS_DATACHECKSUMS_PHASE_ENABLING			0
+#define PROGRESS_DATACHECKSUMS_PHASE_DISABLING			1
+#define PROGRESS_DATACHECKSUMS_PHASE_WAITING			2
+#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_BACKENDS	3
+#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_TEMPREL	4
+#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_CHECKPOINT 5
 
 #endif
-- 
2.48.1

From 3015bbc4b6301d87ac7a186e19fa2eec0763b6c8 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Sun, 9 Mar 2025 13:34:10 +0100
Subject: [PATCH v20250309 5/5] update docs

---
 doc/src/sgml/monitoring.sgml | 84 ++++++++++++++++++++----------------
 doc/src/sgml/wal.sgml        |  8 ++--
 2 files changed, 51 insertions(+), 41 deletions(-)

diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 4a6ef5f1605..2a568ee2357 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6804,8 +6804,8 @@ FROM pg_stat_get_backend_idset() AS backendid;
   <para>
    When data checksums are being enabled on a running cluster, the
    <structname>pg_stat_progress_data_checksums</structname> view will contain
-   a row for each background worker which is currently calculating checksums
-   for the data pages.
+   a row for the launcher process, and one row for each worker process which
+   is currently calculating checksums for the data pages in one database.
   </para>
 
   <table id="pg-stat-progress-data-checksums-view" xreflabel="pg_stat_progress_data_checksums">
@@ -6837,37 +6837,33 @@ FROM pg_stat_get_backend_idset() AS backendid;
      </row>
 
      <row>
-      <entry role="catalog_table_entry">
-       <para role="column_definition">
-        <structfield>phase</structfield> <type>text</type>
-       </para>
-       <para>
-        Current processing phase, see <xref linkend="datachecksum-phases"/>
-        for description of the phases.
-       </para>
-      </entry>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>datid</structfield> <type>oid</type>
+      </para>
+      <para>
+       OID of this database, or 0 for the launcher process
+       relation
+      </para></entry>
      </row>
 
      <row>
-      <entry role="catalog_table_entry">
-       <para role="column_definition">
-        <structfield>databases_total</structfield> <type>integer</type>
-       </para>
-       <para>
-        The total number of databases which will be processed.
-       </para>
-      </entry>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>datname</structfield> <type>name</type>
+      </para>
+      <para>
+       Name of this database, or <literal>NULL</literal> for the
+       launcher process.
+      </para></entry>
      </row>
 
      <row>
       <entry role="catalog_table_entry">
        <para role="column_definition">
-        <structfield>relations_total</structfield> <type>integer</type>
+        <structfield>phase</structfield> <type>text</type>
        </para>
        <para>
-        The total number of relations which will be processed, or
-        <literal>NULL</literal> if the data checksums worker process hasn't
-        calculated the number of relations yet.
+        Current processing phase, see <xref linkend="datachecksum-phases"/>
+        for description of the phases.
        </para>
       </entry>
      </row>
@@ -6875,10 +6871,12 @@ FROM pg_stat_get_backend_idset() AS backendid;
      <row>
       <entry role="catalog_table_entry">
        <para role="column_definition">
-        <structfield>databases_processed</structfield> <type>integer</type>
+        <structfield>databases_total</structfield> <type>integer</type>
        </para>
        <para>
-        The number of databases which have been processed.
+        The total number of databases which will be processed. Only the
+        launcher worker has this value set, the other worker processes
+        have this <literal>NULL</literal>.
        </para>
       </entry>
      </row>
@@ -6886,10 +6884,12 @@ FROM pg_stat_get_backend_idset() AS backendid;
      <row>
       <entry role="catalog_table_entry">
        <para role="column_definition">
-        <structfield>relations_processed</structfield> <type>integer</type>
+        <structfield>databases_done</structfield> <type>integer</type>
        </para>
        <para>
-        The number of relations which have been processed.
+        The number of databases which have been processed. Only the
+        launcher worker has this value set, the other worker processes
+        have this <literal>NULL</literal>.
        </para>
       </entry>
      </row>
@@ -6897,10 +6897,13 @@ FROM pg_stat_get_backend_idset() AS backendid;
      <row>
       <entry role="catalog_table_entry">
        <para role="column_definition">
-        <structfield>databases_current</structfield> <type>oid</type>
+        <structfield>relations_total</structfield> <type>integer</type>
        </para>
        <para>
-        OID of the database currently being processed.
+        The total number of relations which will be processed, or
+        <literal>NULL</literal> if the data checksums worker process hasn't
+        calculated the number of relations yet. The launcher process has
+        this <literal>NULL</literal>.
        </para>
       </entry>
      </row>
@@ -6908,10 +6911,11 @@ FROM pg_stat_get_backend_idset() AS backendid;
      <row>
       <entry role="catalog_table_entry">
        <para role="column_definition">
-        <structfield>relation_current</structfield> <type>oid</type>
+        <structfield>relations_done</structfield> <type>integer</type>
        </para>
        <para>
-        OID of the relation currently being processed.
+        The number of relations which have been processed. The launcher
+        process has this <literal>NULL</literal>.
        </para>
       </entry>
      </row>
@@ -6919,10 +6923,13 @@ FROM pg_stat_get_backend_idset() AS backendid;
      <row>
       <entry role="catalog_table_entry">
        <para role="column_definition">
-        <structfield>relation_current_blocks</structfield> <type>integer</type>
+        <structfield>blocks_total</structfield> <type>integer</type>
        </para>
        <para>
-        The total number of blocks in the relation currently being processed.
+        The number of blocks in the current relation which will be processed,
+        or <literal>NULL</literal> if the data checksums worker process hasn't
+        calculated the number of blocks yet. The launcher process has
+        this <literal>NULL</literal>.
        </para>
       </entry>
      </row>
@@ -6930,11 +6937,11 @@ FROM pg_stat_get_backend_idset() AS backendid;
      <row>
       <entry role="catalog_table_entry">
        <para role="column_definition">
-        <structfield>relation_current_blocks_processed</structfield> <type>integer</type>
+        <structfield>blocks_done</structfield> <type>integer</type>
        </para>
        <para>
-        The number of blocks which have been processed in the relation currently
-        being processed.
+        The number of blocks in the current relation which have been processed.
+        The launcher process has this <literal>NULL</literal>.
        </para>
       </entry>
      </row>
@@ -6982,9 +6989,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
       </entry>
      </row>
      <row>
-      <entry><literal>done</literal></entry>
+      <entry><literal>waiting on checkpoint</literal></entry>
       <entry>
-       The command has finished processing and is exiting.
+       The command is currently waiting for a checkpoint to update the checksum
+       state at the end.
       </entry>
      </row>
     </tbody>
diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgml
index 2562b1a002c..0ada90ca0b1 100644
--- a/doc/src/sgml/wal.sgml
+++ b/doc/src/sgml/wal.sgml
@@ -310,15 +310,17 @@
     If the cluster is stopped while in <literal>inprogress-on</literal> mode, for
     any reason, then this process must be restarted manually. To do this,
     re-execute the function <function>pg_enable_data_checksums()</function>
-    once the cluster has been restarted. The background worker will attempt
-    to resume the work from where it was interrupted.
+    once the cluster has been restarted. The process will start over, there is
+    no support for resuming work from where it was interrupted.
    </para>
 
    <note>
     <para>
      Enabling checksums can cause significant I/O to the system, as most of the
      database pages will need to be rewritten, and will be written both to the
-     data files and the WAL.
+     data files and the WAL. The impact may be limited by throttling using the
+     <parameter>cost_delay</parameter> and <parameter>cost_limit</parameter>
+     parameters of the <function>pg_enable_data_checksums</function> function.
     </para>
    </note>
 
-- 
2.48.1

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 00:35:29 +0100

Seems cfbot was unhappy with the patches, so here's an improved version,
fixing some minor issues in expected output and a compiler warning.

There however seems to be some issue with 003_standby_restarts, which
causes failures on freebsd and macos. I don't know what that is about,
but the test runs much longer than on debian.

regards

-- 
Tomas Vondra
From c8e48010604a0042d22d1b0d3b31c335277b1ceb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 19:21:22 +0100
Subject: [PATCH v20250310 1/5] Online enabling and disabling of data checksums

 
   </sect2>

 

 
  </sect2>

 
+  <para>
 
   <para>
@@ -265,7 +266,7 @@
   </para>

    <para>
 
   </sect2>
+
 
   <sect1 id="wal-intro">

 const char *
 
 	return id;
 
+/*

 		if (doPageWrites &&
 			(!prevDoPageWrites ||

 
 /*
 
 /*
 
 /*
 
+	/*
 
+/*
 
 /*

 

 

 
 
 	/*

 				/*

 

 	/*

 
 #

 
 	/* call only once */
 
+	/*
 
 	/*

 
-	{
 
+	{

 

 /*
 
+	/*

 

 /*

 

 

 
 /*

 
 ifeq ($(with_icu),yes)
 SUBDIRS += icu
 
 if ssl.found()
   subdir('ssl')

 =back

From 54fda18ca79d1ab2626c8c5a5177f462a43f4976 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 20:19:29 +0100
Subject: [PATCH v20250310 2/5] simple post-rebase fixes

- Cast PID to long in elog call (same as we do in postmaster.c)
---
 src/backend/postmaster/datachecksumsworker.c | 7 ++-----
 src/backend/postmaster/postmaster.c          | 5 +++++
 src/backend/utils/activity/pgstat_backend.c  | 2 ++
 3 files changed, 9 insertions(+), 5 deletions(-)

index de7a077f9c2..b30d5481b06 100644
 
-		vacuum_delay_point();
 
 	pfree(relns);
@@ -587,7 +587,7 @@ ProcessDatabase(DataChecksumsWorkerDatabase *db)
 					db->dbname)));
 
 	snprintf(activity, sizeof(activity) - 1,
-			 "Waiting for worker in database %s (pid %d)", db->dbname, pid);
+			 "Waiting for worker in database %s (pid %ld)", db->dbname, (long) pid);
 	pgstat_report_activity(STATE_RUNNING, activity);
 
 	status = WaitForBackgroundWorkerShutdown(bgw_handle);
@@ -752,9 +752,6 @@ DataChecksumsWorkerLauncherMain(Datum arg)
 	 */
 	BackgroundWorkerInitializeConnectionByOid(InvalidOid, InvalidOid, 0);

 
 		case B_AUTOVAC_WORKER:
-- 
2.48.1

From 8d4315cb29088fe46907ab638656df295f3ab43c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 23:07:44 +0100
Subject: [PATCH v20250310 3/5] sync the data_checksums GUC with the local
 variable

 
+/*

 
 /*

 bool
 
 bool

+/*

 
-- 
2.48.1

From 1f043a7de9e36005c75b5386d17ea2d75dda9e43 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Sat, 8 Mar 2025 19:34:50 +0100
Subject: [PATCH v20250310 4/5] make progress reporting work

- Issue: Not sure what to do about relation forks, at this point it
  tracks only blocks for MAIN fork.
---
 src/backend/catalog/system_views.sql         | 36 ++++----
 src/backend/postmaster/datachecksumsworker.c | 91 +++++++++++++++++---
 src/include/commands/progress.h              | 27 +++---
 src/test/regress/expected/rules.out          | 29 ++++---
 4 files changed, 131 insertions(+), 52 deletions(-)

 
index b30d5481b06..4101fdff107 100644

 

 	while (true)
 	{
 
-			/*
 
+			/*
 
+	/*

 

 

+	/* worker done */

 
 #endif
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 36bed4b168d..2cfea837554 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2050,25 +2050,34 @@ pg_stat_progress_data_checksums| SELECT s.pid,
             WHEN 2 THEN 'waiting'::text
             WHEN 3 THEN 'waiting on backends'::text
             WHEN 4 THEN 'waiting on temporary tables'::text
-            WHEN 5 THEN 'done'::text
+            WHEN 5 THEN 'waiting on checkpoint'::text
+            WHEN 6 THEN 'done'::text
             ELSE NULL::text
         END AS phase,
         CASE s.param2
             WHEN '-1'::integer THEN NULL::bigint
             ELSE s.param2
         END AS databases_total,
-        CASE s.param3
+    s.param3 AS databases_done,
+        CASE s.param4
             WHEN '-1'::integer THEN NULL::bigint
-            ELSE s.param3
+            ELSE s.param4
         END AS relations_total,
-    s.param4 AS databases_processed,
-    s.param5 AS relations_processed,
-    s.param6 AS databases_current,
-    s.param7 AS relation_current,
-    s.param8 AS relation_current_blocks,
-    s.param9 AS relation_current_blocks_processed
+        CASE s.param5
+            WHEN '-1'::integer THEN NULL::bigint
+            ELSE s.param5
+        END AS relations_done,
+        CASE s.param6
+            WHEN '-1'::integer THEN NULL::bigint
+            ELSE s.param6
+        END AS blocks_total,
+        CASE s.param7
+            WHEN '-1'::integer THEN NULL::bigint
+            ELSE s.param7
+        END AS blocks_done
    FROM (pg_stat_get_progress_info('DATACHECKSUMS'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
-     LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+     LEFT JOIN pg_database d ON ((s.datid = d.oid)))
+  ORDER BY s.datid;
 pg_stat_progress_vacuum| SELECT s.pid,
     s.datid,
     d.datname,
-- 
2.48.1

From 318c3f960707f80a53437635140ab935e936262e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Sun, 9 Mar 2025 13:34:10 +0100
Subject: [PATCH v20250310 5/5] update docs

---

 
      <row>
 
      <row>
 
      <row>
 
    <note>
     <para>
 
-- 
2.48.1

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 01:18:23 +0100

> but the test runs much longer than on debian.

OK, turns out the failures were caused by the test creating a standby
from a backup, without a slot, so sometimes the primary removed the
necessary WAL. Fixed in the attached version.

There's still a failure on windows, though. I'd bet that's due to the
data_checksum/LocalDatachecksumVersion sync not working correctly on
builds with EXEC_BACKEND, or something like that, but it's too late so
I'll take a closer look tomorrow.

regards

-- 
Tomas Vondra
From c8e48010604a0042d22d1b0d3b31c335277b1ceb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 19:21:22 +0100
Subject: [PATCH v20250310b 1/5] Online enabling and disabling of data
 checksums

 
   </sect2>

 

 
  </sect2>

 
+  <para>
 
   <para>
@@ -265,7 +266,7 @@
   </para>

    <para>
 
   </sect2>
+
 
   <sect1 id="wal-intro">

 const char *
 
 	return id;
 
+/*

 		if (doPageWrites &&
 			(!prevDoPageWrites ||

 
 /*
 
 /*
 
 /*
 
+	/*
 
+/*
 
 /*

 

 

 
 
 	/*

 				/*

 

 	/*

 
 #

 
 	/* call only once */
 
+	/*
 
 	/*

 
-	{
 
+	{

 

 /*
 
+	/*

 

 /*

 

 

 
 /*

 
 ifeq ($(with_icu),yes)
 SUBDIRS += icu
 
 if ssl.found()
   subdir('ssl')

 =back

From 252b6c3ad06b555e1cc20ca33155ee408f3cff33 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 20:19:29 +0100
Subject: [PATCH v20250310b 2/5] simple post-rebase fixes

- Fix test 003_standby_restarts by adding a replication slot
---
 src/backend/postmaster/datachecksumsworker.c |  7 ++-----
 src/backend/postmaster/postmaster.c          |  5 +++++
 src/backend/utils/activity/pgstat_backend.c  |  2 ++
 src/test/checksum/t/003_standby_restarts.pl  | 10 +++++++++-
 4 files changed, 18 insertions(+), 6 deletions(-)

-		vacuum_delay_point();
 
 	pfree(relns);

 

 		case B_AUTOVAC_WORKER:
diff --git a/src/test/checksum/t/003_standby_restarts.pl b/src/test/checksum/t/003_standby_restarts.pl
index 6c0fe8f3bf8..6782664f4e6 100644
--- a/src/test/checksum/t/003_standby_restarts.pl
+++ b/src/test/checksum/t/003_standby_restarts.pl
@@ -20,15 +20,23 @@ my $enable_params = '0, 100, true';
 my $node_primary = PostgreSQL::Test::Cluster->new('primary');
 $node_primary->init(allows_streaming => 1, no_data_checksums => 1);
 $node_primary->start;
-my $backup_name = 'my_backup';
+
+my $slotname = 'physical_slot';
+$node_primary->safe_psql('postgres',
+	"SELECT pg_create_physical_replication_slot('$slotname')");
 
 # Take backup
+my $backup_name = 'my_backup';
 $node_primary->backup($backup_name);
 
 # Create streaming standby linking to primary
 my $node_standby_1 = PostgreSQL::Test::Cluster->new('standby_1');
 $node_standby_1->init_from_backup($node_primary, $backup_name,
 	has_streaming => 1);
+$node_standby_1->append_conf(
+	'postgresql.conf', qq[
+primary_slot_name = '$slotname'
+]);
 $node_standby_1->start;
 
 # Create some content on the primary to have un-checksummed data in the cluster
-- 
2.48.1

From 1b6849f9eb1fc52b7cf3c59e9be2559e13ce40c0 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 23:07:44 +0100
Subject: [PATCH v20250310b 3/5] sync the data_checksums GUC with the local
 variable

 
+/*

 
 /*

 bool
 
 bool

+/*

 
-- 
2.48.1

From 351f66d51c9f604a956080aa3cc5cbd91becc8f0 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Sat, 8 Mar 2025 19:34:50 +0100
Subject: [PATCH v20250310b 4/5] make progress reporting work

 

 

 
 	while (true)
 	{
 
-			/*
 
+			/*
 
+	/*

 

 

+	/* worker done */

 
 #endif

From b23637b35099a581aab737a3afd58fbeddc9016a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Sun, 9 Mar 2025 13:34:10 +0100
Subject: [PATCH v20250310b 5/5] update docs

---

 
      <row>
 
      <row>
 
      <row>
 
    <note>
     <para>
 
-- 
2.48.1

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 10:46:32 +0100

On 3/10/25 01:18, Tomas Vondra wrote:
> ...

Just like I suspected, there was a bug in EXEC_BACKEND, although a bit
different from what I guessed - the worker state in shmem was zeroed
every time, not just once. And a second issue was child_process_kinds
got out of sync with BackendType (mea culpa).

For me, this passes all CI tests, hopefully cfbot will be happy too.

regards

-- 
Tomas Vondra
From adf6848e0cff38ac861f921dc3b7ef6462f0d86a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 19:21:22 +0100
Subject: [PATCH v20250310c 1/5] Online enabling and disabling of data
 checksums

 
   </sect2>

 

 
  </sect2>

 
+  <para>
 
   <para>
@@ -265,7 +266,7 @@
   </para>

    <para>
 
   </sect2>
+
 
   <sect1 id="wal-intro">

 const char *
 
 	return id;
 
+/*

 		if (doPageWrites &&
 			(!prevDoPageWrites ||

 
 /*
 
 /*
 
 /*
 
+	/*
 
+/*
 
 /*

 

 

 
 
 	/*

 				/*

 

 	/*

index 3c594415bfd..b1b5cdcf36c 100644
 
 #

 
 	/* call only once */
 
+	/*
 
 	/*

 
-	{
 
+	{

 

 /*
 
+	/*

 

 /*

 

 

 
 /*

 
 ifeq ($(with_icu),yes)
 SUBDIRS += icu
 
 if ssl.found()
   subdir('ssl')

 =back

From f01c15887f1c612cd7769bf9f3f6dd22525c493d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 20:19:29 +0100
Subject: [PATCH v20250310c 2/5] simple post-rebase fixes

- Fix DataChecksumsWorkerShmemInit to only zero the memory once.

- Update child_process_kinds to keep it in sync with BackendType.
---
 src/backend/postmaster/datachecksumsworker.c | 28 ++++++++++----------
 src/backend/postmaster/launch_backend.c      |  3 +++
 src/backend/postmaster/postmaster.c          |  5 ++++
 src/backend/utils/activity/pgstat_backend.c  |  2 ++
 src/test/checksum/t/003_standby_restarts.pl  | 10 ++++++-
 5 files changed, 33 insertions(+), 15 deletions(-)

index de7a077f9c2..6df92684a3b 100644
 
-		vacuum_delay_point();
 
 	pfree(relns);

 
-	/* Initialize backend status information */
-	pgstat_bestart();
-
 	enabling_checksums = DataChecksumsWorkerShmem->launch_enable_checksums;
 	DataChecksumsWorkerShmem->launcher_running = true;
 	DataChecksumsWorkerShmem->enabling_checksums = enabling_checksums;
@@ -1095,16 +1092,19 @@ DataChecksumsWorkerShmemInit(void)
 						DataChecksumsWorkerShmemSize(),
 						&found);
 
-	MemSet(DataChecksumsWorkerShmem, 0, DataChecksumsWorkerShmemSize());
+	if (!found)
+	{
+		MemSet(DataChecksumsWorkerShmem, 0, DataChecksumsWorkerShmemSize());
 
-	/*
-	 * Even if this is a redundant assignment, we want to be explicit about
-	 * our intent for readability, since we want to be able to query this
-	 * state in case of restartability.
-	 */
-	DataChecksumsWorkerShmem->launch_enable_checksums = false;
-	DataChecksumsWorkerShmem->launcher_running = false;
-	DataChecksumsWorkerShmem->launch_fast = false;
+		/*
+		 * Even if this is a redundant assignment, we want to be explicit about
+		 * our intent for readability, since we want to be able to query this
+		 * state in case of restartability.
+		 */
+		DataChecksumsWorkerShmem->launch_enable_checksums = false;
+		DataChecksumsWorkerShmem->launcher_running = false;
+		DataChecksumsWorkerShmem->launch_fast = false;
+	}
 }
 
 /*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..92d8017fd56 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -202,6 +202,9 @@ static child_process_kind child_process_kinds[] = {
 	[B_WAL_SUMMARIZER] = {"wal_summarizer", WalSummarizerMain, true},
 	[B_WAL_WRITER] = {"wal_writer", WalWriterMain, true},
 
+	[B_DATACHECKSUMSWORKER_LAUNCHER] = {"datachecksum launcher", NULL, false},
+	[B_DATACHECKSUMSWORKER_WORKER] = {"datachecksum worker", NULL, false},
+
 	[B_LOGGER] = {"syslogger", SysLoggerMain, false},
 };

index 6efbb650aa8..bd458f8c1af 100644
@@ -295,6 +295,8 @@ pgstat_tracks_backend_bktype(BackendType bktype)
 
 		case B_AUTOVAC_WORKER:
 
 # Take backup

From 5fe7c540af4c3bf004cae9f6b6dfa69d2ab9aa84 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 23:07:44 +0100
Subject: [PATCH v20250310c 3/5] sync the data_checksums GUC with the local
 variable

 
+/*

 
 /*

 bool
 
 bool

+/*

 
-- 
2.48.1

From d03c53fca689f5303a5f4f8e8d142687ed2c44d9 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Sat, 8 Mar 2025 19:34:50 +0100
Subject: [PATCH v20250310c 4/5] make progress reporting work

 
 
index 6df92684a3b..bbbce61cfa6 100644

 

 	while (true)
 	{
 
-			/*
 
+			/*
 
+	/*
@@ -1258,6 +1296,7 @@ DataChecksumsWorkerMain(Datum arg)

@@ -1271,6 +1310,10 @@ DataChecksumsWorkerMain(Datum arg)
 	BackgroundWorkerInitializeConnectionByOid(dboid, InvalidOid,
 											  BGWORKER_BYPASS_ALLOWCONN);
 
@@ -1297,6 +1340,24 @@ DataChecksumsWorkerMain(Datum arg)
 
@@ -1304,6 +1365,9 @@ DataChecksumsWorkerMain(Datum arg)
 
@@ -1316,6 +1380,10 @@ DataChecksumsWorkerMain(Datum arg)
 		return;
 	}
 
@@ -1372,5 +1440,8 @@ DataChecksumsWorkerMain(Datum arg)

+	/* worker done */

 
 #endif

From 10c411612677b7fa1bc01d26fd923e92f4ec108b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Sun, 9 Mar 2025 13:34:10 +0100
Subject: [PATCH v20250310c 5/5] update docs

---

 
      <row>
 
      <row>
 
      <row>
 
    <note>
     <para>
 
-- 
2.48.1

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 12:17:58 +0100

> For me, this passes all CI tests, hopefully cfbot will be happy too.

A bit embarrassing, I did not notice updating child_process_kinds breaks
the stats regression test, so here's a version fixing that.

regards

-- 
Tomas Vondra
From adf6848e0cff38ac861f921dc3b7ef6462f0d86a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 19:21:22 +0100
Subject: [PATCH v20250310d 1/5] Online enabling and disabling of data
 checksums

 
   </sect2>

 

 
  </sect2>

 
+  <para>
 
   <para>
@@ -265,7 +266,7 @@
   </para>

    <para>
 
   </sect2>
+
 
   <sect1 id="wal-intro">

 const char *
 
 	return id;
 
+/*

 		if (doPageWrites &&
 			(!prevDoPageWrites ||

 
 /*
 
 /*
 
 /*
 
+	/*
 
+/*
 
 /*

 

 

 
 
 	/*

 				/*

 

 	/*

 
 #

 
 	/* call only once */
 
+	/*
 
 	/*

 
-	{
 
+	{

 

 /*
 
+	/*

 

 /*

 

 

 
 /*

 
 ifeq ($(with_icu),yes)
 SUBDIRS += icu
 
 if ssl.found()
   subdir('ssl')

 =back

From 8a72efd62662efd8672d84d2872fbf54173c9337 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 20:19:29 +0100
Subject: [PATCH v20250310d 2/5] simple post-rebase fixes

- Fix expected output for stats regression test.
---
 src/backend/postmaster/datachecksumsworker.c | 28 ++++++++++----------
 src/backend/postmaster/launch_backend.c      |  3 +++
 src/backend/postmaster/postmaster.c          |  5 ++++
 src/backend/utils/activity/pgstat_backend.c  |  2 ++
 src/test/checksum/t/003_standby_restarts.pl  | 10 ++++++-
 src/test/regress/expected/stats.out          | 18 ++++++++++++-
 6 files changed, 50 insertions(+), 16 deletions(-)

-		vacuum_delay_point();
 
 	pfree(relns);

 

-	/*
 
 /*

 
 
 		case B_AUTOVAC_WORKER:
 
 # Take backup

 # Create some content on the primary to have un-checksummed data in the cluster
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 30d763c4aee..da6645f0d7b 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -51,6 +51,22 @@ client backend|relation|vacuum
 client backend|temp relation|normal
 client backend|wal|init
 client backend|wal|normal
+datachecksumsworker launcher|relation|bulkread
+datachecksumsworker launcher|relation|bulkwrite
+datachecksumsworker launcher|relation|init
+datachecksumsworker launcher|relation|normal
+datachecksumsworker launcher|relation|vacuum
+datachecksumsworker launcher|temp relation|normal
+datachecksumsworker launcher|wal|init
+datachecksumsworker launcher|wal|normal
+datachecksumsworker worker|relation|bulkread
+datachecksumsworker worker|relation|bulkwrite
+datachecksumsworker worker|relation|init
+datachecksumsworker worker|relation|normal
+datachecksumsworker worker|relation|vacuum
+datachecksumsworker worker|temp relation|normal
+datachecksumsworker worker|wal|init
+datachecksumsworker worker|wal|normal
 slotsync worker|relation|bulkread
 slotsync worker|relation|bulkwrite
 slotsync worker|relation|init
@@ -87,7 +103,7 @@ walsummarizer|wal|init
 walsummarizer|wal|normal
 walwriter|wal|init
 walwriter|wal|normal
-(71 rows)
+(87 rows)
 \a
 -- ensure that both seqscan and indexscan plans are allowed
 SET enable_seqscan TO on;
-- 
2.48.1

From 3ad310ac23abc138e926dc72c9b4a3d4c8e45c28 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 23:07:44 +0100
Subject: [PATCH v20250310d 3/5] sync the data_checksums GUC with the local
 variable

 
+/*

 
 /*

 bool
 
 bool

+/*

 
-- 
2.48.1

From 6e5f04d2ae4e037f80fb0d3bedd73e5fe54d79b0 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Sat, 8 Mar 2025 19:34:50 +0100
Subject: [PATCH v20250310d 4/5] make progress reporting work

 

 

 
 	while (true)
 	{
 
-			/*
 
+			/*
 
+	/*

 

 

+	/* worker done */

 
 #endif

From 1f053c9b7d1e12030b10240a54008b6f1e898b93 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Sun, 9 Mar 2025 13:34:10 +0100
Subject: [PATCH v20250310d 5/5] update docs

---

 
      <row>
 
      <row>
 
      <row>
 
    <note>
     <para>
 
-- 
2.48.1

---

From: Daniel Gustafsson <[email-masked]>
Date: Mon, 10 Mar 2025 14:27:06 +0100

> On 10 Mar 2025, at 12:17, Tomas Vondra <[email-masked]> wrote:
> 

Thank you so much for picking up and fixing the blockers, it's highly appreciated!

>> For me, this passes all CI tests, hopefully cfbot will be happy too.

Confirmed, it compiles clean, builds docs and passes all tests for me as well.

A few comments from reading over your changes:

+   launcher worker has this value set, the other worker processes
+   have this <literal>NULL</literal>.
There seems to be a word or two missing (same in a few places), should this be
"have this set to NULL"?

+   The command is currently waiting for a checkpoint to update the checksum
+   state at the end.
s/at the end/before finishing/?

+ * XXX aren't PG_DATA_ and DATA_ constants the same? why do we need both?
They aren't mapping 1:1 as PG_DATA_ has the version numbers, and if checksums
aren't enabled there is no version and thus there is no PG_DATA_CHECKSUMS_OFF.
This could of course be remedied.  IIRC one reason for adding the enum was to
get compiler warnings on missing cases when switch()ing over the value, but I
don't think the current code has any switch.

+   /* XXX isn't it weird there's no wait between the phase updates? */
It is, I think we should skip PROGRESS_DATACHECKSUMS_PHASE_WAITING_BACKENDS in
favor of PROGRESS_DATACHECKSUMS_PHASE_ENABLING.

+   * When enabling checksums, we have to wait for a checkpoint for the
+   * checksums to e.
Seems to be missing the punchline, "for the checksum state to be moved from
in-progress to on" perhaps?

It also needs a pgindent and pgperltidy but there were only small trivial
changes there.

Thanks again for updating the patch!

--
Daniel Gustafsson

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 16:01:35 +0100

> There seems to be a word or two missing (same in a few places), should this be

done

> s/at the end/before finishing/?

done

> don't think the current code has any switch.

I haven't done anything about this. I'm not convinced it's an issue we
need to fix, and I haven't tried how much work would it be.

> favor of PROGRESS_DATACHECKSUMS_PHASE_ENABLING.

Removed the WAITING_BACKENDS phase.

> in-progress to on" perhaps?

done

> It also needs a pgindent and pgperltidy but there were only small trivial

done

Attached is an updated version.

-- 
Tomas Vondra
From 1cb65478673dc8ffa01312f7a06fc488164c8f20 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 19:21:22 +0100
Subject: [PATCH v20250310e 1/4] Online enabling and disabling of data
 checksums

 doc/src/sgml/monitoring.sgml                  |  215 ++-
 doc/src/sgml/wal.sgml                         |   59 +-
 src/backend/access/transam/xlog.c             |  489 +++++-
 src/backend/postmaster/datachecksumsworker.c  | 1447 +++++++++++++++++
 src/backend/postmaster/launch_backend.c       |    3 +
 src/backend/postmaster/postmaster.c           |    5 +
 src/backend/utils/activity/pgstat_backend.c   |    2 +
 src/backend/utils/misc/guc_tables.c           |   31 +-
 src/include/access/xlog.h                     |   17 +-
 src/include/commands/progress.h               |   17 +
 src/test/checksum/t/001_basic.pl              |   88 +
 src/test/checksum/t/003_standby_restarts.pl   |  139 ++
 src/test/regress/expected/rules.out           |   37 +
 src/test/regress/expected/stats.out           |   18 +-
 58 files changed, 3194 insertions(+), 50 deletions(-)

   </sect2>

 
index 16646f560e8..2a568ee2357 100644

@@ -6793,6 +6794,212 @@ FROM pg_stat_get_backend_idset() AS backendid;
 
  </sect2>

 
+  <para>
index f3b86b26be9..0ada90ca0b1 100644
 
   <para>
@@ -265,7 +266,7 @@
   </para>

    <para>
@@ -274,6 +275,56 @@
    </para>
 
   </sect2>
+
 
   <sect1 id="wal-intro">

 const char *
 
 	return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 799fc739e18..f137cdc6d42 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -647,6 +647,24 @@ static XLogRecPtr LocalMinRecoveryPoint;
 static TimeLineID LocalMinRecoveryPointTLI;
 static bool updateMinRecoveryPoint = true;
 
+/*
@@ -715,6 +733,8 @@ static void WALInsertLockAcquireExclusive(void);
 
@@ -828,9 +848,10 @@ XLogInsertRecord(XLogRecData *rdata,
@@ -843,7 +864,9 @@ XLogInsertRecord(XLogRecData *rdata,
 
 		if (doPageWrites &&
 			(!prevDoPageWrites ||
@@ -4579,9 +4602,7 @@ ReadControlFile(void)

 
 /*
@@ -4615,13 +4636,376 @@ GetMockAuthenticationNonce(void)
 }
 
 /*
 
 /*
@@ -6194,6 +6578,33 @@ StartupXLOG(void)
 	 */
 	CompleteCommitTsInitialization();
 
+	/*
@@ -8201,6 +8612,24 @@ XLogReportParameters(void)
 	}
 }
 
+/*
@@ -8629,6 +9058,46 @@ xlog_redo(XLogReaderState *record)
 
 /*

 

 
index a4d2cfdcaf5..4330d0ad656 100644

index 00000000000..bbbce61cfa6
@@ -0,0 +1,1447 @@

 

 	/*

 				/*

 

 	/*

 		case B_AUTOVAC_WORKER:

 #

 
 	/* call only once */
 
+	/*
 
 	/*
index ad25cbb39c5..66abf056159 100644
 
@@ -601,7 +609,6 @@ static int	shared_memory_size_mb;
 
 #ifdef USE_ASSERT_CHECKING
@@ -1952,17 +1959,6 @@ struct config_bool ConfigureNamesBool[] =
 		NULL, NULL, NULL
 	},
 
-	{
@@ -5299,6 +5295,17 @@ struct config_enum ConfigureNamesEnum[] =
 		NULL, NULL, NULL
 	},
 
+	{

 

 /*
 
+	/*
index d313099c027..aec3ea0bc63 100644

@@ -117,7 +118,7 @@ extern PGDLLIMPORT int wal_level;
 
 /* Do we need to WAL-log information required only for Hot Standby and logical replication? */
 #define XLogStandbyInfoActive() (wal_level >= WAL_LEVEL_REPLICA)
@@ -230,7 +231,19 @@ extern XLogRecPtr GetXLogWriteRecPtr(void);

 

 /*
 
index 7c736e7b03b..94b478a6cc9 100644
@@ -157,4 +157,21 @@

 

 

 /*

 
 ifeq ($(with_icu),yes)
 SUBDIRS += icu
index 00000000000..6782664f4e6
@@ -0,0 +1,139 @@
 
 if ssl.found()
   subdir('ssl')

 =back
index 62f69ac20b2..2cfea837554 100644
@@ -2041,6 +2041,43 @@ pg_stat_progress_create_index| SELECT s.pid,

From 98b6bf1adb3a5a009d130b93e1089c284a17d132 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 14:56:14 +0100
Subject: [PATCH v20250310e 2/4] review fixes

---
 doc/src/sgml/monitoring.sgml                 | 13 +++----------
 src/backend/catalog/system_views.sql         |  7 +++----
 src/backend/postmaster/datachecksumsworker.c | 11 +----------
 src/include/commands/progress.h              |  5 ++---
 src/include/miscadmin.h                      |  4 +++-
 src/test/regress/expected/rules.out          |  7 +++----
 6 files changed, 15 insertions(+), 32 deletions(-)

diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 2a568ee2357..4602134cbde 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6876,7 +6876,7 @@ FROM pg_stat_get_backend_idset() AS backendid;
        <para>
         The total number of databases which will be processed. Only the
         launcher worker has this value set, the other worker processes
-        have this <literal>NULL</literal>.
+        have this set to <literal>NULL</literal>.
        </para>
       </entry>
      </row>
@@ -6889,7 +6889,7 @@ FROM pg_stat_get_backend_idset() AS backendid;
        <para>
         The number of databases which have been processed. Only the
         launcher worker has this value set, the other worker processes
-        have this <literal>NULL</literal>.
+        have this set to <literal>NULL</literal>.
        </para>
       </entry>
      </row>
@@ -6974,13 +6974,6 @@ FROM pg_stat_get_backend_idset() AS backendid;
        The command is currently disabling data checksums on the cluster.
       </entry>
      </row>
-     <row>
-      <entry><literal>waiting on backends</literal></entry>
-      <entry>
-       The command is currently waiting for backends to acknowledge the data
-       checksum operation.
-      </entry>
-     </row>
      <row>
       <entry><literal>waiting on temporary tables</literal></entry>
       <entry>
@@ -6992,7 +6985,7 @@ FROM pg_stat_get_backend_idset() AS backendid;
       <entry><literal>waiting on checkpoint</literal></entry>
       <entry>
        The command is currently waiting for a checkpoint to update the checksum
-       state at the end.
+       state before finishing.
       </entry>
      </row>
     </tbody>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 4330d0ad656..6ffd31ce39c 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1340,10 +1340,9 @@ CREATE VIEW pg_stat_progress_data_checksums AS
         CASE S.param1 WHEN 0 THEN 'enabling'
                       WHEN 1 THEN 'disabling'
                       WHEN 2 THEN 'waiting'
-                      WHEN 3 THEN 'waiting on backends'
-                      WHEN 4 THEN 'waiting on temporary tables'
-                      WHEN 5 THEN 'waiting on checkpoint'
-                      WHEN 6 THEN 'done'
+                      WHEN 3 THEN 'waiting on temporary tables'
+                      WHEN 4 THEN 'waiting on checkpoint'
+                      WHEN 5 THEN 'done'
                       END AS phase,
         CASE S.param2 WHEN -1 THEN NULL ELSE S.param2 END AS databases_total,
         S.param3 AS databases_done,
diff --git a/src/backend/postmaster/datachecksumsworker.c b/src/backend/postmaster/datachecksumsworker.c
index bbbce61cfa6..d9833cd4c98 100644
--- a/src/backend/postmaster/datachecksumsworker.c
+++ b/src/backend/postmaster/datachecksumsworker.c
@@ -801,15 +801,6 @@ again:
 		}
 		RESUME_INTERRUPTS();
 
-		/*
-		 * Initialize progress and indicate that we are waiting on the other
-		 * backends to clear the procsignalbarrier.
-		 */
-		pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,
-									 PROGRESS_DATACHECKSUMS_PHASE_WAITING_BACKENDS);
-
-		/* XXX isn't it weird there's no wait between the phase updates? */
-
 		/*
 		 * Set the state to inprogress-on and wait on the procsignal barrier.
 		 */
@@ -1083,7 +1074,7 @@ ProcessAllDatabases(bool immediate_checkpoint)
 
 	/*
 	 * When enabling checksums, we have to wait for a checkpoint for the
-	 * checksums to e.
+	 * checksums to change from in-progress to on.
 	 */
 	pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,
 								 PROGRESS_DATACHECKSUMS_PHASE_WAITING_CHECKPOINT);
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 94b478a6cc9..b172a5f24ce 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -170,8 +170,7 @@
 #define PROGRESS_DATACHECKSUMS_PHASE_ENABLING			0
 #define PROGRESS_DATACHECKSUMS_PHASE_DISABLING			1
 #define PROGRESS_DATACHECKSUMS_PHASE_WAITING			2
-#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_BACKENDS	3
-#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_TEMPREL	4
-#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_CHECKPOINT 5
+#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_TEMPREL	3
+#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_CHECKPOINT 4
 
 #endif
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 9923c7f518d..69b1dc720f6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -392,7 +392,9 @@ extern PGDLLIMPORT BackendType MyBackendType;
 #define AmWalReceiverProcess()		(MyBackendType == B_WAL_RECEIVER)
 #define AmWalSummarizerProcess()	(MyBackendType == B_WAL_SUMMARIZER)
 #define AmWalWriterProcess()		(MyBackendType == B_WAL_WRITER)
-#define AmDataChecksumsWorkerProcess()	(MyBackendType == B_DATACHECKSUMSWORKER_LAUNCHER || MyBackendType == B_DATACHECKSUMSWORKER_WORKER)
+#define AmDataChecksumsWorkerProcess() \
+	(MyBackendType == B_DATACHECKSUMSWORKER_LAUNCHER || \
+	 MyBackendType == B_DATACHECKSUMSWORKER_WORKER)
 
 #define AmSpecialWorkerProcess() \
 	(AmAutoVacuumLauncherProcess() || \
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2cfea837554..0dd383a76dd 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2048,10 +2048,9 @@ pg_stat_progress_data_checksums| SELECT s.pid,
             WHEN 0 THEN 'enabling'::text
             WHEN 1 THEN 'disabling'::text
             WHEN 2 THEN 'waiting'::text
-            WHEN 3 THEN 'waiting on backends'::text
-            WHEN 4 THEN 'waiting on temporary tables'::text
-            WHEN 5 THEN 'waiting on checkpoint'::text
-            WHEN 6 THEN 'done'::text
+            WHEN 3 THEN 'waiting on temporary tables'::text
+            WHEN 4 THEN 'waiting on checkpoint'::text
+            WHEN 5 THEN 'done'::text
             ELSE NULL::text
         END AS phase,
         CASE s.param2
-- 
2.48.1

From 239be70853039067f1408c0e46f18a6888403275 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 14:57:05 +0100
Subject: [PATCH v20250310e 3/4] pgindent

---
 src/backend/postmaster/datachecksumsworker.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/src/backend/postmaster/datachecksumsworker.c b/src/backend/postmaster/datachecksumsworker.c
index d9833cd4c98..6a201dca8de 100644
--- a/src/backend/postmaster/datachecksumsworker.c
+++ b/src/backend/postmaster/datachecksumsworker.c
@@ -921,7 +921,7 @@ ProcessAllDatabases(bool immediate_checkpoint)
 			PROGRESS_DATACHECKSUMS_BLOCKS_DONE,
 		};
 
-		int64	vals[6];
+		int64		vals[6];
 
 		vals[0] = list_length(DatabaseList);
 		vals[1] = 0;
@@ -974,7 +974,8 @@ ProcessAllDatabases(bool immediate_checkpoint)
 			processed_databases++;
 
 			/*
-			 * Update the number of processed databases in the progress report.
+			 * Update the number of processed databases in the progress
+			 * report.
 			 */
 			pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_DBS_DONE,
 										 processed_databases);
@@ -1126,9 +1127,9 @@ DataChecksumsWorkerShmemInit(void)
 		MemSet(DataChecksumsWorkerShmem, 0, DataChecksumsWorkerShmemSize());
 
 		/*
-		 * Even if this is a redundant assignment, we want to be explicit about
-		 * our intent for readability, since we want to be able to query this
-		 * state in case of restartability.
+		 * Even if this is a redundant assignment, we want to be explicit
+		 * about our intent for readability, since we want to be able to query
+		 * this state in case of restartability.
 		 */
 		DataChecksumsWorkerShmem->launch_enable_checksums = false;
 		DataChecksumsWorkerShmem->launcher_running = false;
@@ -1339,7 +1340,7 @@ DataChecksumsWorkerMain(Datum arg)
 			PROGRESS_DATACHECKSUMS_RELS_DONE
 		};
 
-		int64	vals[2];
+		int64		vals[2];
 
 		vals[0] = list_length(RelationList);
 		vals[1] = 0;
-- 
2.48.1

From 219dc80d840db30365c78ea1f5bcd3ffd198a340 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 15:05:43 +0100
Subject: [PATCH v20250310e 4/4] perltidy

---
 src/test/perl/PostgreSQL/Test/Cluster.pm | 6 ++++--
 src/test/subscription/t/013_partition.pl | 3 +--
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 666bd2a2d4c..1c66360c16c 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -3761,7 +3761,8 @@ sub checksum_enable_offline
 	my ($self) = @_;
 
 	print "### Enabling checksums in \"$self->data_dir\"\n";
-	PostgreSQL::Test::Utils::system_or_bail('pg_checksums', '-D', $self->data_dir, '-e');
+	PostgreSQL::Test::Utils::system_or_bail('pg_checksums', '-D',
+		$self->data_dir, '-e');
 	return;
 }
 
@@ -3778,7 +3779,8 @@ sub checksum_disable_offline
 	my ($self) = @_;
 
 	print "### Disabling checksums in \"$self->data_dir\"\n";
-	PostgreSQL::Test::Utils::system_or_bail('pg_checksums', '-D', $self->data_dir, '-d');
+	PostgreSQL::Test::Utils::system_or_bail('pg_checksums', '-D',
+		$self->data_dir, '-d');
 	return;
 }
 
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 61b0cb4aa1a..4f78dd48815 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -51,8 +51,7 @@ $node_subscriber1->safe_psql('postgres',
 );
 # make a BRIN index to test aminsertcleanup logic in subscriber
 $node_subscriber1->safe_psql('postgres',
-	"CREATE INDEX tab1_c_brin_idx ON tab1 USING brin (c)"
-);
+	"CREATE INDEX tab1_c_brin_idx ON tab1 USING brin (c)");
 $node_subscriber1->safe_psql('postgres',
 	"CREATE TABLE tab1_1 (b text, c text DEFAULT 'sub1_tab1', a int NOT NULL)"
 );
-- 
2.48.1

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 16:16:02 +0100

One thing I forgot to mention is the progress reporting only updates
blocks for the FORK_MAIN. It wouldn't be difficult to report blocks for
each fork, but it'd be confusing - the relation counters would remain
the same, but the block counters would change for each fork.

I guess we could report the current_relation/fork, but it seems like an
overkill. The main fork is by far the largest one, so this seems OK.

regards

-- 
Tomas Vondra

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 18:35:56 +0100

Hi,

I continued stress testing this, as I was rather unsure why the assert
failures reported in [1] disappeared. And I managed to reproduce that
again, and I think I actually understand why it happens.

I modified the test script (attached) to setup replication, not just a
single instance. And then it does a bit of work, flips the checksums,
restarts the instances (randomly, fast/immediate), verifies the checkums
and so on. And I can hit this assert in AbsorbChecksumsOnBarrier()
pretty easily:

  Assert(LocalDataChecksumVersion ==
         PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);

The reason is pretty simple - this happens on the standby:

1) standby receives XLOG_CHECKSUMS and applies it from 2 to 1 (i.e. it
sets ControlFile->data_checksum_version from "inprogress-on" to "on"),
and signals all other processes to refresh LocalDataChecksumVersion

2) the control file gets written to disk for whatever reason (redo does
this in a number of places)

3) standby gets restarted with "immediate" mode (I'm not sure if this
can happen with "fast" mode, I only recall seeing "immediate")

4) the standby receives the XLOG_CHECKSUMS record *again*, updates the
ControlFile->data_checksum_version (to the same value, no noop), and
then signals the other processes again

5) the other processes already have LocalDataChecksumVersion=1 (on), but
the assert says it should be 2 (inprogress-on) => kaboom

I believe this can happen for changes in either direction, although the
window while disabling checksums is more narrow.

I'm not sure what to do about this. Maybe we could relax the assert in
some way? But that seems a bit ... possibly risky. It's not necessarily
true we'll see the immediately preceding checksum state, we might see a
couple updates back (if the control file was not updated in between).

Could this affect checksum verification during recovery? Imagine we get
to the "on" state, the controlfile gets flushed, and then the standby
restarts and starts receiving older records again. The control file says
we should be verifying checksums, but couldn't some of the writes have
been lost (and so the pages may not have a valid checksum)?

The one idea I have is to create an "immediate" restartpoint in
xlog_redo() right after XLOG_CHECKSUMS updates the control file. AFAICS
a "spread" restartpoint would not be enough, because then we could get
into the same situation with a control file of sync (ahead of WAL) after
a restart. It'd not be cheap, but it should be a rare operation ...

I was wondering if the primary has the same issue, but AFAICS it does
not. It flushes the control file in only a couple places, I couldn't
think of a way to get it out of sync.

regards

[1]

-- 
Tomas Vondra

---

From: Dagfinn Ilmari Mannsåker <[email-masked]>
Date: Tue, 11 Mar 2025 13:07:35 +0000

As the resident perl style pedant, I'd just like to complain about the
below:

> +		$self->data_dir, '-e');

This breaking between the command line options and its arguments is why
we're switching to using fat commas. We're also using long options for
improved self-documentation, so this should be written as:

	PostgreSQL::Test::Utils::system_or_bail('pg_checksums',
		'--pgdata' => $self->data_dir,
		'--enable');

And likewise below in the disable method.

- ilmari

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 12 Mar 2025 13:28:50 +0100

> And likewise below in the disable method.

I don't know what fat comma is, but that's simply what perltidy did. I
don't mind formatting it differently, if there's a better way.

thanks

-- 
Tomas Vondra

---

From: Dagfinn Ilmari Mannsåker <[email-masked]>
Date: Wed, 12 Mar 2025 13:08:50 +0000

> don't mind formatting it differently, if there's a better way.

Fat comma is the perlish name for the => arrow, which is semantically
equivalent to a comma (except it auto-quotes any immediately preceding
bareword), but looks fatter.  Perltidy knows to not wrap lines around
them, keeping the key and value (or option and argument in this case)
together.  See commit ce1b0f9da03 for a large (but not complete, I have
more patches pending) conversion to this new style.

- ilmari

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 12 Mar 2025 14:16:41 +0100

> think of a way to get it out of sync.

I continued investigating this and experimenting with alternative
approaches, and I think the way the patch relies on ControlFile is not
quite right. That is, it always sets data_checksum_version to the last
("current") value, but that's not what ControlFile is for ...

The ControlFile is meant to be a safe/consistent state, e.g. for crash
recovery. By setting data_checksum_version to the "last" value we've
seen, that's broken - if the control file gets persisted (haven't seen
this on primary, but pretty common on replica, per the report), the
recovery will start with a "future" data_checksum_version value. Which
is wrong - we'll read the XLOG_CHECKUMS record, triggering the assert. I
suspect it might also lead to confusion whether checksums should be
verified or not.

In my earlier message I suggested maybe this could be solved by forcing
a checkpoint every time we see the XLOG_CHECKUMS record (or rather a
restart point, as it'd be on the replica). Sure, that would have some
undesirable consequences (forcing an immediate checkpoint is not cheap,
and the redo would need to wait for that). But the assumption was it'd
be very rare (how often you enable checksums?), so this cost might be
acceptable.

But when I started experimenting with this, I realized it has a couple
other issues:

1) We can't do the checkpoint/restartpoint when handling XLOG_CHECKUMS,
because that'd mean we see this XLOG record again, which we don't want.
So the checkpoint would need to happen the *next* time we update the
control file.

2) But we can't trigger a checkpoint from UpdateControlFile, because of
locking (because CreateCheckPoint also calls UpdateControlFile). So this
would require much more invasive changes to all places updating the
control file.

3) It does not resolve the mismatch with using ControlFile to store
"current" data_checksums_version value.

4) ... probably more minor issues that I already forgot about.

In the end, I decided to try to rework this by storing the current value
elsewhere, and only updating the "persistent" value in the control file
when necessary.

XLogCtl seemed like a good place, so I used that - after all, it's a
value from XLOG. Maybe there's a better place? I'm open to suggestions,
but it does not really affect the overall approach.

So all the places now update XLogCtl->data_checksums_version instead of
the ControlFile, and also query this flag for *current* value.

The value is copied from XLogCtl to ControlFile when creating checkpoint
(or restartpoint), and the control file is persisted. This means (a) the
current value can't get written to the control file prematurely, and (b)
the value is consistent with the checkpoint (i.e. with the LSN where we
start crash recovery, if needed).

The attached 0005 patch implements this. It's a bit WIP and I'm sure it
can be improved, but I'm yet to see a single crash/failure with it. With
the original patch I've seen crashes after 5-10 loops (i.e. a couple
minutes), I'm now at loop 1000 and it's still OK.

I believe the approach is correct, but the number of possible states
(e.g. after a crash/restart) seems a bit complex. I wonder if there's a
better way to handle this, but I can't think of any. Ideas?

One issue I ran into is the postmaster does not seem to be processing
the barriers, and thus not getting info about the data_checksum_version
changes. That's fine until it needs to launch a child process (e.g. a
walreceiver), which will then see the LocalDataChecksumVersion as of the
start of the instance, not the "current" one. I fixed this by explicitly
refreshing the value in postmaster_child_launch(), but maybe I'm missing
something. (Also, EXEC_BACKEND may need to handle this too.)

regards

-- 
Tomas Vondra
From ae739e9c25f1fe979b501f42d164ab92be46dad5 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 19:21:22 +0100
Subject: [PATCH v20250312 1/6] Online enabling and disabling of data checksums

 
   </sect2>

 
index aaa6586d3a4..e51bf902dc2 100644

@@ -6828,6 +6829,212 @@ FROM pg_stat_get_backend_idset() AS backendid;
 
  </sect2>

 
+  <para>
 
   <para>
@@ -265,7 +266,7 @@
   </para>

    <para>
 
   </sect2>
+
 
   <sect1 id="wal-intro">

 const char *
 
 	return id;
 
+/*

 		if (doPageWrites &&
 			(!prevDoPageWrites ||

 
 /*
 
 /*
 
 /*
 
+	/*
 
+/*
 
 /*

 

 

 

 
 
 	/*

 				/*

 

 	/*
 
 	((PageHeader) page)->pd_checksum = pg_checksum_page(page, blkno);
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index a8cb54a7732..7c9ff5c19c5 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -376,6 +376,8 @@ pgstat_tracks_backend_bktype(BackendType bktype)
 		case B_BG_WRITER:
 		case B_CHECKPOINTER:
 		case B_STARTUP:
+		case B_DATACHECKSUMSWORKER_LAUNCHER:
+		case B_DATACHECKSUMSWORKER_WORKER:
 			return false;
 
 		case B_AUTOVAC_WORKER:

 #
index 662ce46cbc2..2cb7766c1e2 100644

 
 	/* call only once */
 
+	/*
 
 	/*

 
-	{
 
+	{

 

 /*
 
+	/*

 

 

 /*
index 42e427f8fe8..b9de4da8c53 100644
@@ -12253,6 +12253,23 @@

 

 

 
 /*

 
 ifeq ($(with_icu),yes)
 SUBDIRS += icu
 
 if ssl.found()
   subdir('ssl')

 =back
index f77caacc17d..85e15d4cc2b 100644
index dfe2690bdd3..e5be803e159 100644
@@ -4143,6 +4148,7 @@ xl_btree_split

From ebf63bf17a022d6c8068aefba38d66538642240b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 14:56:14 +0100
Subject: [PATCH v20250312 2/6] review fixes

---

index e51bf902dc2..8fec0277f97 100644
@@ -6911,7 +6911,7 @@ FROM pg_stat_get_backend_idset() AS backendid;
@@ -6924,7 +6924,7 @@ FROM pg_stat_get_backend_idset() AS backendid;
@@ -7009,13 +7009,6 @@ FROM pg_stat_get_backend_idset() AS backendid;
@@ -7027,7 +7020,7 @@ FROM pg_stat_get_backend_idset() AS backendid;
 
-		/*
 
 	/*
 
 #endif

From 3fb57e3846f6f015b0a4d9d9165197d5d67dd5bc Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 14:57:05 +0100
Subject: [PATCH v20250312 3/6] pgindent

---

-		int64	vals[6];
+		int64		vals[6];

 			/*
 
 		/*
 
-		int64	vals[2];
+		int64		vals[2];

From 1585dc3d1c8d2887a82ab41cc26433acac7039f5 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Mar 2025 15:05:43 +0100
Subject: [PATCH v20250312 4/6] perltidy

---

 

From b78ba0ddda20dd2331358c157589ac80c807705a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Tue, 11 Mar 2025 19:16:23 +0100
Subject: [PATCH v20250312 5/6] data_checksum_version reworks

---
 src/backend/access/transam/xlog.c       | 133 +++++++++++++++++-------
 src/backend/postmaster/launch_backend.c |  10 ++
 src/include/catalog/pg_control.h        |   5 +-
 src/include/miscadmin.h                 |   4 +
 4 files changed, 116 insertions(+), 36 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f137cdc6d42..61da6d583cd 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -550,6 +550,9 @@ typedef struct XLogCtlData
 	 */
 	XLogRecPtr	lastFpwDisableRecPtr;
 
+	/* last data_checksum_version we've seen */
+	uint32		data_checksum_version;
+
 	slock_t		info_lck;		/* locks shared variables shown above */
 } XLogCtlData;
 
@@ -4262,6 +4265,12 @@ InitControlFile(uint64 sysidentifier, uint32 data_checksum_version)
 	ControlFile->wal_log_hints = wal_log_hints;
 	ControlFile->track_commit_timestamp = track_commit_timestamp;
 	ControlFile->data_checksum_version = data_checksum_version;
+
+	/*
+	 * Set the data_checksum_version value into XLogCtl, which is where all
+	 * processes get the current value from. (Maybe it should go just there?)
+	 */
+	XLogCtl->data_checksum_version = data_checksum_version;
 }
 
 static void
@@ -4601,8 +4610,6 @@ ReadControlFile(void)
 		(SizeOfXLogLongPHD - SizeOfXLogShortPHD);
 
 	CalculateCheckpointSegments();
-
-	SetLocalDataChecksumVersion(ControlFile->data_checksum_version);
 }
 
 /*
@@ -4734,9 +4741,9 @@ SetDataChecksumsOnInProgress(void)
 
 	XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
 
-	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
-	ControlFile->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION;
-	LWLockRelease(ControlFileLock);
+	SpinLockAcquire(&XLogCtl->info_lck);
+	XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION;
+	SpinLockRelease(&XLogCtl->info_lck);
 
 	barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);
 
@@ -4780,28 +4787,28 @@ SetDataChecksumsOn(void)

-	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+	SpinLockAcquire(&XLogCtl->info_lck);
 
 	/*
 	 * The only allowed state transition to "on" is from "inprogress-on" since
 	 * that state ensures that all pages will have data checksums written.
 	 */
-	if (ControlFile->data_checksum_version != PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION)
+	if (XLogCtl->data_checksum_version != PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION)
 	{
-		LWLockRelease(ControlFileLock);
+		SpinLockRelease(&XLogCtl->info_lck);
 		elog(ERROR, "checksums not in \"inprogress-on\" mode");
 	}
 
-	LWLockRelease(ControlFileLock);
+	SpinLockRelease(&XLogCtl->info_lck);
 
 	MyProc->delayChkptFlags |= DELAY_CHKPT_START;
 	START_CRIT_SECTION();
 
 	XLogChecksums(PG_DATA_CHECKSUM_VERSION);
 
-	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
-	ControlFile->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
-	LWLockRelease(ControlFileLock);
+	SpinLockAcquire(&XLogCtl->info_lck);
+	XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
+	SpinLockRelease(&XLogCtl->info_lck);
 
 	barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);
 
@@ -4836,12 +4843,12 @@ SetDataChecksumsOff(void)
 
 	Assert(ControlFile);
 
-	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+	SpinLockAcquire(&XLogCtl->info_lck);
 
 	/* If data checksums are already disabled there is nothing to do */
-	if (ControlFile->data_checksum_version == 0)
+	if (XLogCtl->data_checksum_version == 0)
 	{
-		LWLockRelease(ControlFileLock);
+		SpinLockRelease(&XLogCtl->info_lck);
 		return;
 	}
 
@@ -4852,18 +4859,18 @@ SetDataChecksumsOff(void)
 	 * "inprogress-off" the next transition to "off" can be performed, after
 	 * which all data checksum processing is disabled.
 	 */
-	if (ControlFile->data_checksum_version == PG_DATA_CHECKSUM_VERSION)
+	if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_VERSION)
 	{
-		LWLockRelease(ControlFileLock);
+		SpinLockRelease(&XLogCtl->info_lck);
 
 		MyProc->delayChkptFlags |= DELAY_CHKPT_START;
 		START_CRIT_SECTION();
 
 		XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);
 
-		LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
-		ControlFile->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION;
-		LWLockRelease(ControlFileLock);
+		SpinLockAcquire(&XLogCtl->info_lck);
+		XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION;
+		SpinLockRelease(&XLogCtl->info_lck);
 
 		barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);
 
@@ -4890,7 +4897,7 @@ SetDataChecksumsOff(void)
 		 * or "inprogress-off" and we can transition directly to "off" from
 		 * there.
 		 */
-		LWLockRelease(ControlFileLock);
+		SpinLockRelease(&XLogCtl->info_lck);
 	}
 
 	/*
@@ -4901,9 +4908,9 @@ SetDataChecksumsOff(void)
 
 	XLogChecksums(0);
 
-	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
-	ControlFile->data_checksum_version = 0;
-	LWLockRelease(ControlFileLock);
+	SpinLockAcquire(&XLogCtl->info_lck);
+	XLogCtl->data_checksum_version = 0;
+	SpinLockRelease(&XLogCtl->info_lck);
 
 	barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);
 
@@ -4958,9 +4965,9 @@ AbsorbChecksumsOffBarrier(void)
 void
 InitLocalControldata(void)
 {
-	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
-	SetLocalDataChecksumVersion(ControlFile->data_checksum_version);
-	LWLockRelease(ControlFileLock);
+	SpinLockAcquire(&XLogCtl->info_lck);
+	SetLocalDataChecksumVersion(XLogCtl->data_checksum_version);
+	SpinLockRelease(&XLogCtl->info_lck);
 }
 
 /*
@@ -4994,6 +5001,40 @@ SetLocalDataChecksumVersion(uint32 data_checksum_version)
 	}
 }
 
+/*
+ * Initialize the various data checksum values - GUC, local, ....
+ */
+void
+InitLocalDataChecksumVersion(void)
+{
+	uint32	data_checksum_version;
+
+	SpinLockAcquire(&XLogCtl->info_lck);
+	data_checksum_version = XLogCtl->data_checksum_version;
+	SpinLockRelease(&XLogCtl->info_lck);
+
+	SetLocalDataChecksumVersion(data_checksum_version);
+}
+
+/*
+ * Get the local data_checksum_version (cached XLogCtl value).
+ */
+uint32
+GetLocalDataChecksumVersion(void)
+{
+	return LocalDataChecksumVersion;
+}
+
+/*
+ * Get the *current* data_checksum_version (might not be written to control
+ * file yet).
+ */
+uint32
+GetCurrentDataChecksumVersion(void)
+{
+	return XLogCtl->data_checksum_version;
+}
+
 /* guc hook */
 const char *
 show_data_checksums(void)
@@ -5447,6 +5488,11 @@ XLOGShmemInit(void)
 	XLogCtl->InstallXLogFileSegmentActive = false;
 	XLogCtl->WalWriterSleeping = false;
 
+	/* use the checksum info from control file */
+	XLogCtl->data_checksum_version = ControlFile->data_checksum_version;
+
+	SetLocalDataChecksumVersion(XLogCtl->data_checksum_version);
+
 	SpinLockInit(&XLogCtl->Insert.insertpos_lck);
 	SpinLockInit(&XLogCtl->info_lck);
 	pg_atomic_init_u64(&XLogCtl->logInsertResult, InvalidXLogRecPtr);
@@ -6585,7 +6631,7 @@ StartupXLOG(void)
 	 * background worker directly from here, it has to be launched from a
 	 * regular backend.
 	 */
-	if (ControlFile->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION)
+	if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION)
 		ereport(WARNING,
 				(errmsg("data checksums are being enabled, but no worker is running"),
 				 errhint("If checksums were being enabled during shutdown then processing must be manually restarted.")));
@@ -6596,13 +6642,13 @@ StartupXLOG(void)
 	 * checksums and we can move to off instead of prompting the user to
 	 * perform any action.
 	 */
-	if (ControlFile->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION)
+	if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION)
 	{
 		XLogChecksums(0);
 
-		LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
-		ControlFile->data_checksum_version = 0;
-		LWLockRelease(ControlFileLock);
+		SpinLockAcquire(&XLogCtl->info_lck);
+		XLogCtl->data_checksum_version = 0;
+		SpinLockRelease(&XLogCtl->info_lck);
 	}
 
 	/*
@@ -7460,6 +7506,12 @@ CreateCheckPoint(int flags)
 	checkPoint.fullPageWrites = Insert->fullPageWrites;
 	checkPoint.wal_level = wal_level;
 
+	/*
+	 * Get the current data_checksum_version value from xlogctl, valid at
+	 * the time of the checkpoint.
+	 */
+	checkPoint.data_checksum_version = XLogCtl->data_checksum_version;
+
 	if (shutdown)
 	{
 		XLogRecPtr	curInsert = XLogBytePosToRecPtr(Insert->CurrBytePos);
@@ -7715,6 +7767,9 @@ CreateCheckPoint(int flags)
 	ControlFile->minRecoveryPoint = InvalidXLogRecPtr;
 	ControlFile->minRecoveryPointTLI = 0;
 
+	/* make sure we start with the checksum version as of the checkpoint */
+	ControlFile->data_checksum_version = checkPoint.data_checksum_version;
+
 	/*
 	 * Persist unloggedLSN value. It's reset on crash recovery, so this goes
 	 * unused on non-shutdown checkpoints, but seems useful to store it always
@@ -7861,6 +7916,10 @@ CreateEndOfRecoveryRecord(void)
 	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
 	ControlFile->minRecoveryPoint = recptr;
 	ControlFile->minRecoveryPointTLI = xlrec.ThisTimeLineID;
+
+	/* start with the latest checksum version (as of the end of recovery) */
+	ControlFile->data_checksum_version = XLogCtl->data_checksum_version;
+
 	UpdateControlFile();
 	LWLockRelease(ControlFileLock);
 
@@ -8203,6 +8262,10 @@ CreateRestartPoint(int flags)
 			if (flags & CHECKPOINT_IS_SHUTDOWN)
 				ControlFile->state = DB_SHUTDOWNED_IN_RECOVERY;
 		}
+
+		/* we shall start with the latest checksum version */
+		ControlFile->data_checksum_version = lastCheckPoint.data_checksum_version;
+
 		UpdateControlFile();
 	}
 	LWLockRelease(ControlFileLock);
@@ -9065,9 +9128,9 @@ xlog_redo(XLogReaderState *record)
 
 		memcpy(&state, XLogRecGetData(record), sizeof(xl_checksum_state));
 
-		LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
-		ControlFile->data_checksum_version = state.new_checksumtype;
-		LWLockRelease(ControlFileLock);
+		SpinLockAcquire(&XLogCtl->info_lck);
+		XLogCtl->data_checksum_version = state.new_checksumtype;
+		SpinLockRelease(&XLogCtl->info_lck);
 
 		/*
 		 * Block on a procsignalbarrier to await all processes having seen the
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 92d8017fd56..fc6e2c6ed41 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -271,6 +271,16 @@ postmaster_child_launch(BackendType child_type, int child_slot,
 			memcpy(MyClientSocket, client_sock, sizeof(ClientSocket));
 		}
 
+		/*
+		 * update the LocalProcessControlFile to match XLogCtl->data_checksum_version
+		 *
+		 * XXX It seems the postmaster (which is what gets forked into the new
+		 * child process) does not absorb the checksum barriers, therefore it
+		 * does not update the value (except after a restart). Not sure if there
+		 * is some sort of race condition.
+		 */
+		InitLocalDataChecksumVersion();
+
 		/*
 		 * Run the appropriate Main function
 		 */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 9b5a50adf54..727042b73be 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -62,6 +62,9 @@ typedef struct CheckPoint
 	 * set to InvalidTransactionId.
 	 */
 	TransactionId oldestActiveXid;
+
+	/* data checksums at the time of the checkpoint  */
+	uint32		data_checksum_version;
 } CheckPoint;
 
 /* XLOG info values for XLOG rmgr */
@@ -220,7 +223,7 @@ typedef struct ControlFileData
 	bool		float8ByVal;	/* float8, int8, etc pass-by-value? */
 
 	/* Are data pages protected by checksums? Zero if no checksum version */
-	uint32		data_checksum_version;
+	uint32		data_checksum_version;		/* persistent */
 
 	/*
 	 * True if the default signedness of char is "signed" on a platform where
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 69b1dc720f6..befab157ecf 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -532,6 +532,10 @@ extern Size EstimateClientConnectionInfoSpace(void);
 extern void SerializeClientConnectionInfo(Size maxsize, char *start_address);
 extern void RestoreClientConnectionInfo(char *conninfo);
 
+extern uint32 GetLocalDataChecksumVersion(void);
+extern uint32 GetCurrentDataChecksumVersion(void);
+extern void InitLocalDataChecksumVersion(void);
+
 /* in executor/nodeHash.c */
 extern size_t get_hash_memory_limit(void);
 
-- 
2.48.1

From a7aa7c355418db44b23cdcb3e920bd95a9891d13 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Wed, 12 Mar 2025 12:40:46 +0100
Subject: [PATCH v20250312 6/6] debug

---
 src/backend/access/transam/xlog.c       | 43 +++++++++++++++++++++++++
 src/backend/postmaster/launch_backend.c |  3 ++
 src/backend/storage/ipc/procsignal.c    | 34 +++++++++++++++++++
 src/backend/utils/init/miscinit.c       |  3 ++
 4 files changed, 83 insertions(+)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 61da6d583cd..63d74e95075 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4271,6 +4271,8 @@ InitControlFile(uint64 sysidentifier, uint32 data_checksum_version)
 	 * processes get the current value from. (Maybe it should go just there?)
 	 */
 	XLogCtl->data_checksum_version = data_checksum_version;
+
+	elog(LOG, "InitControlFile %p data_checksum_version %u", XLogCtl, ControlFile->data_checksum_version);
 }
 
 static void
@@ -4928,6 +4930,7 @@ SetDataChecksumsOff(void)
 bool
 AbsorbChecksumsOnInProgressBarrier(void)
 {
+	elog(LOG, "AbsorbChecksumsOnInProgressBarrier");
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
 	return true;
 }
@@ -4935,6 +4938,7 @@ AbsorbChecksumsOnInProgressBarrier(void)
 bool
 AbsorbChecksumsOnBarrier(void)
 {
+	elog(LOG, "AbsorbChecksumsOnBarrier");
 	Assert(LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_VERSION);
 	return true;
@@ -4943,6 +4947,7 @@ AbsorbChecksumsOnBarrier(void)
 bool
 AbsorbChecksumsOffInProgressBarrier(void)
 {
+	elog(LOG, "AbsorbChecksumsOffInProgressBarrier");
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);
 	return true;
 }
@@ -4950,6 +4955,7 @@ AbsorbChecksumsOffInProgressBarrier(void)
 bool
 AbsorbChecksumsOffBarrier(void)
 {
+	elog(LOG, "AbsorbChecksumsOffBarrier");
 	SetLocalDataChecksumVersion(0);
 	return true;
 }
@@ -4979,6 +4985,7 @@ InitLocalControldata(void)
 void
 SetLocalDataChecksumVersion(uint32 data_checksum_version)
 {
+	elog(LOG, "SetLocalDataChecksumVersion %u", data_checksum_version);
 	LocalDataChecksumVersion = data_checksum_version;
 
 	switch (LocalDataChecksumVersion)
@@ -5488,6 +5495,9 @@ XLOGShmemInit(void)
 	XLogCtl->InstallXLogFileSegmentActive = false;
 	XLogCtl->WalWriterSleeping = false;
 
+	elog(LOG, "XLogCtl->data_checksum_version %u ControlFile->data_checksum_version %u",
+		 XLogCtl->data_checksum_version, ControlFile->data_checksum_version);
+
 	/* use the checksum info from control file */
 	XLogCtl->data_checksum_version = ControlFile->data_checksum_version;
 
@@ -7512,6 +7522,8 @@ CreateCheckPoint(int flags)
 	 */
 	checkPoint.data_checksum_version = XLogCtl->data_checksum_version;
 
+	elog(WARNING, "CREATECHECKPOINT XLogCtl->data_checksum_version %u", XLogCtl->data_checksum_version);
+
 	if (shutdown)
 	{
 		XLogRecPtr	curInsert = XLogBytePosToRecPtr(Insert->CurrBytePos);
@@ -7767,6 +7779,10 @@ CreateCheckPoint(int flags)
 	ControlFile->minRecoveryPoint = InvalidXLogRecPtr;
 	ControlFile->minRecoveryPointTLI = 0;
 
+	elog(LOG, "CreateCheckPoint data_checksum_version %u %u",
+		 ControlFile->data_checksum_version,
+		 checkPoint.data_checksum_version);
+
 	/* make sure we start with the checksum version as of the checkpoint */
 	ControlFile->data_checksum_version = checkPoint.data_checksum_version;
 
@@ -7917,6 +7933,10 @@ CreateEndOfRecoveryRecord(void)
 	ControlFile->minRecoveryPoint = recptr;
 	ControlFile->minRecoveryPointTLI = xlrec.ThisTimeLineID;
 
+	elog(LOG, "CreateEndOfRecoveryRecord data_checksum_version %u xlog %u",
+		 ControlFile->data_checksum_version,
+		 XLogCtl->data_checksum_version);
+
 	/* start with the latest checksum version (as of the end of recovery) */
 	ControlFile->data_checksum_version = XLogCtl->data_checksum_version;
 
@@ -8127,6 +8147,9 @@ CreateRestartPoint(int flags)
 	lastCheckPoint = XLogCtl->lastCheckPoint;
 	SpinLockRelease(&XLogCtl->info_lck);
 
+	elog(LOG, "CreateRestartPoint lastCheckPointRecPtr %X/%X lastCheckPointEndPtr %X/%X",
+		 LSN_FORMAT_ARGS(lastCheckPointRecPtr), LSN_FORMAT_ARGS(lastCheckPointEndPtr));
+
 	/*
 	 * Check that we're still in recovery mode. It's ok if we exit recovery
 	 * mode after this check, the restart point is valid anyway.
@@ -8263,6 +8286,10 @@ CreateRestartPoint(int flags)
 				ControlFile->state = DB_SHUTDOWNED_IN_RECOVERY;
 		}
 
+		elog(LOG, "CreateRestartPoint data_checksum_version %u %u",
+			 ControlFile->data_checksum_version,
+			 lastCheckPoint.data_checksum_version);
+
 		/* we shall start with the latest checksum version */
 		ControlFile->data_checksum_version = lastCheckPoint.data_checksum_version;
 
@@ -9125,13 +9152,25 @@ xlog_redo(XLogReaderState *record)
 	{
 		xl_checksum_state state;
 		uint64		barrier;
+		XLogRecPtr	checkpointLsn;
+		uint32		value,
+					value_last;
 
 		memcpy(&state, XLogRecGetData(record), sizeof(xl_checksum_state));
 
 		SpinLockAcquire(&XLogCtl->info_lck);
+		value_last = XLogCtl->data_checksum_version;
 		XLogCtl->data_checksum_version = state.new_checksumtype;
 		SpinLockRelease(&XLogCtl->info_lck);
 
+		LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+		checkpointLsn = ControlFile->checkPoint;
+		value = ControlFile->data_checksum_version;
+		LWLockRelease(ControlFileLock);
+
+		elog(LOG, "XLOG_CHECKSUMS xlog_redo %X/%X control checkpoint %X/%X control %u last %u record %u",
+			 LSN_FORMAT_ARGS(lsn), LSN_FORMAT_ARGS(checkpointLsn), value, value_last, state.new_checksumtype);
+
 		/*
 		 * Block on a procsignalbarrier to await all processes having seen the
 		 * change to checksum status. Once the barrier has been passed we can
@@ -9140,22 +9179,26 @@ xlog_redo(XLogReaderState *record)
 		switch (state.new_checksumtype)
 		{
 			case PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION:
+				elog(LOG, "XLOG_CHECKSUMS emit PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON");
 				barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);
 				WaitForProcSignalBarrier(barrier);
 				break;
 
 			case PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION:
+				elog(LOG, "XLOG_CHECKSUMS emit PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF");
 				barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);
 				WaitForProcSignalBarrier(barrier);
 				break;
 
 			case PG_DATA_CHECKSUM_VERSION:
+				elog(LOG, "XLOG_CHECKSUMS emit PROCSIGNAL_BARRIER_CHECKSUM_ON");
 				barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);
 				WaitForProcSignalBarrier(barrier);
 				break;
 
 			default:
 				Assert(state.new_checksumtype == 0);
+				elog(LOG, "XLOG_CHECKSUMS emit PROCSIGNAL_BARRIER_CHECKSUM_OFF");
 				barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);
 				WaitForProcSignalBarrier(barrier);
 				break;
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index fc6e2c6ed41..2034c204cee 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -235,6 +235,9 @@ postmaster_child_launch(BackendType child_type, int child_slot,
 
 	Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
 
+	elog(LOG, "postmaster_child_launch: LocalDataChecksumVersion %u xlog %u", GetLocalDataChecksumVersion(),
+	GetCurrentDataChecksumVersion());
+
 #ifdef EXEC_BACKEND
 	pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
 							startup_data, startup_data_len, client_sock);
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 2b13a8cd260..b88d1d07431 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -553,6 +553,40 @@ ProcessProcSignalBarrier(void)
 
 		PG_TRY();
 		{
+			/* print info about barriers */
+			{
+				uint32	tmp = flags;
+
+				elog(LOG, "ProcessProcSignalBarrier flags %u", tmp);
+
+				while (tmp != 0)
+				{
+					ProcSignalBarrierType type;
+
+					type = (ProcSignalBarrierType) pg_rightmost_one_pos32(tmp);
+					switch (type)
+					{
+						case PROCSIGNAL_BARRIER_SMGRRELEASE:
+							elog(LOG, "PROCSIGNAL_BARRIER_SMGRRELEASE");
+							break;
+						case PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON:
+							elog(LOG, "PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON");
+							break;
+						case PROCSIGNAL_BARRIER_CHECKSUM_ON:
+							elog(LOG, "PROCSIGNAL_BARRIER_CHECKSUM_ON");
+							break;
+						case PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF:
+							elog(LOG, "PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF");
+							break;
+						case PROCSIGNAL_BARRIER_CHECKSUM_OFF:
+							elog(LOG, "PROCSIGNAL_BARRIER_CHECKSUM_OFF");
+							break;
+					}
+
+					BARRIER_CLEAR_BIT(tmp, type);
+				}
+			}
+
 			/*
 			 * Process each type of barrier. The barrier-processing functions
 			 * should normally return true, but may return false if the
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index a071ba6f455..df52ce8ad7f 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -164,6 +164,9 @@ InitPostmasterChild(void)
 				(errcode_for_socket_access(),
 				 errmsg_internal("could not set postmaster death monitoring pipe to FD_CLOEXEC mode: %m")));
 #endif
+
+	elog(LOG, "InitPostmasterChild: LocalDataChecksumVersion %u xlog %u", GetLocalDataChecksumVersion(),
+	GetCurrentDataChecksumVersion());
 }
 
 /*
-- 
2.48.1

---

From: Daniel Gustafsson <[email-masked]>
Date: Thu, 13 Mar 2025 10:54:28 +0100

> On 12 Mar 2025, at 14:16, Tomas Vondra <[email-masked]> wrote:

> ("current") value, but that's not what ControlFile is for ...

Agreed, that's a thinko on my part.  Reading it makes it clear, but I had
failed to see that when hacking =/

> but it does not really affect the overall approach.

Seems like a good place for it.

> start crash recovery, if needed).

+1

> minutes), I'm now at loop 1000 and it's still OK.

Given how successful this test has been at stressing out errors that is indeed
comforting to hear.

> better way to handle this, but I can't think of any. Ideas?

Not sure if this moves the needle too far in terms of complexity wrt to the
previous version of the patch, there were already multiple copies.

> changes.

Makes sense, that seems like a pretty reasonable constraint for the barrier.

> That's fine until it needs to launch a child process (e.g. a

The pg_checksums test is failing for me on this version due to the GUC not
being initialized, don't we need something like the below as well?  (With a
comment explaining why ReadControlFile wasn't enough.)

@@ -5319,6 +5319,7 @@ LocalProcessControlFile(bool reset)
        Assert(reset || ControlFile == NULL);
        ControlFile = palloc(sizeof(ControlFileData));
        ReadControlFile();
+       SetLocalDataChecksumVersion(ControlFile->data_checksum_version);

A few comments on the patchset:

+ * Local state fror Controlfile data_checksum_version. After initialization
s/fror/for/.  Also, this is no longer true as it's a local copy of the XlogCtl
value and not the Controlfile value (which may or may not be equal).

- if (ControlFile->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION)
+ if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION)
 	ereport(WARNING,
 		(errmsg("data checksums are being enabled, but no worker is running"),
 		 errhint("If checksums were being enabled during shutdown then processing must be manually restarted.")));
Reading this made me realize what a terrible error message I had placed there,
the hint is good but the message says checksums are being enabled but they're
not being enabled.  Maybe "data checksums are marked as being in-progress, but
no worker is running"

+uint32
+GetLocalDataChecksumVersion(void)
+{
+ return LocalDataChecksumVersion;
+}
+
+/*
+ * Get the *current* data_checksum_version (might not be written to control
+ * file yet).
+ */
+uint32
+GetCurrentDataChecksumVersion(void)
+{
+ return XLogCtl->data_checksum_version;
+}
I wonder if CachedDataChecksumVersion would be more appropriate to distinguish
it from the Current value, and also to make appear less like actual copies of
controlfile values like LocalMinRecoveryPoint.  Another thought is if we should
have the GetLocalDataChecksumVersion() API?  GetCurrentDataChecksumVersion()
should be a better API no?

--
Daniel Gustafsson

---

From: Tomas Vondra <[email-masked]>
Date: Thu, 13 Mar 2025 12:03:29 +0100

> failed to see that when hacking =/

It wasn't obvious to me either, until I managed to trigger the failure
and investigated the root cause.

> Seems like a good place for it.

OK

>> start crash recovery, if needed).

OK. I still want to go over the places once more and double check it
sets the ControlFile value to the right data_checksum_version.

> Given how successful this test has been at stressing out errors that is indeed

It is. I plan to vary the stress test a bit more, and also run it on
another machine (rpi5, to get some non-x86 testing).

> previous version of the patch, there were already multiple copies.

It does add one more place (XLogCtl->data_checksum_version) to store the
current state, so it's not that much more complex, ofc. But I was not
really comparing this to the previous patch version, I meant the state
space in general - all possible combinations of all the flags (control
file, local + xlogct).

I wonder if it might be possible to have a more thorough validation of
the transitions. We already have that for the LocalDataChecksumVersion,
thanks to the asserts - and it was damn useful, otherwise we would not
have noticed this issue for a long time, I think.

I wonder if we can have similar checks for the other flags. I'm pretty
sure we can have the same checks for XLogCtl, right? I'm not quite sure
about ControlFile - can't that "skip" some of the changes, e.g. if we do
(enable->disable->enable) within a single checkpoint? Need to check.

This also reminds me I had a question about the barrier - can't it
happen a process gets to process multiple barriers at the same time? I
mean, let's say it gets stuck for a while, and the cluster happens to go
through disable+enable. Won't it then see both barriers? That'd be a
problem, because the core processes the barriers in the order determined
by the enum value, not in the order the barriers happened. Which means
it might break the expected state transitions again (and end with the
wrong local value). I haven't tried, though.

> Makes sense, that seems like a pretty reasonable constraint for the barrier.

Not sure I follow. What's a reasonable constraint?

> +       SetLocalDataChecksumVersion(ControlFile->data_checksum_version);

Yeah, I think this (or something like it) is missing.

> not being enabled.  Maybe "data checksums are marked as being in-progress, but

Makes sense, will reword.

> should be a better API no?

FWIW those functions are for debug logging only, I needed to print the
values in a couple places outside xlog.c. I don't intend to make that
part of the patch.

regards

-- 
Tomas Vondra

---

From: Daniel Gustafsson <[email-masked]>
Date: Thu, 13 Mar 2025 13:32:20 +0100

> On 13 Mar 2025, at 12:03, Tomas Vondra <[email-masked]> wrote:

> space in general - all possible combinations of all the flags (control

Fair point.

> sure we can have the same checks for XLogCtl, right?

I don't see why not, they should abide by the same rules.

> I'm not quite sure

For enable->disable->enable within a single checkpoint then ControlFile should
never see the disable state.

> wrong local value). I haven't tried, though.

Interesting, that seems like a general deficiency in the barriers, surely
processing them in-order would be more intuitive?  That would probably require
some form of Lamport clock though.

> Not sure I follow. What's a reasonable constraint?

That the postmaster deosn't process them.

> Yeah, I think this (or something like it) is missing.

Thanks for confirming.

> values in a couple places outside xlog.c. I don't intend to make that

Ah, gotcha, I never applied the debug patch from the patchset so I figured this
was a planned API.  The main question still stands though, if LocalDataCheckXX
can be confusing and CachedDataCheckXX would be better in order to
distinguish it from actual controlfile copies?

--
Daniel Gustafsson

---

From: Tomas Vondra <[email-masked]>
Date: Thu, 13 Mar 2025 17:26:47 +0100

> I don't see why not, they should abide by the same rules.

OK, I'll add these asserts.

> never see the disable state.

Hmm, that means we can't have the same checks for the ControlFile
fields, but I don't think that's a problem. We've verified the "path" to
that (on the XLogCtl field), so that seems fine.

> some form of Lamport clock though.

Yeah, that seems non-trivial. What if we instead ensured there can't be
two barriers set at the same time? Say, if we (somehow) ensured all
processes saw the previous barrier before allowing a new one, we would
not have this issue, right?

But I don't know what would be a good way to ensure this. Is there a way
to check if all processes saw the barrier? Any ideas?

> That the postmaster deosn't process them.

OK, that means we need a way to "refresh" the value for new child
processses, similar to what my patch does. But I suspect there might be
a race condition - if the child process starts while processing the
XLOG_CHECKUMS record, it might happen to get the new value and then also
the barrier (if it does the "refresh" in between the XLogCtl update and
the barrier). Doesn't this need some sort of interlock, preventing this?

The child startup would need to do this:

1) acquire lock
2) reset barriers
3) refresh the LocalDataChecksumValue (from XLogCtl)
4) release lock

while the walreceiver would do this

1) acquire lock
2) update XLogCtl value
3) emit barrier
4) release lock

Or is there a reason why this would be unnecessary?

> distinguish it from actual controlfile copies?

Yeah, I'll think about the naming.

regards

-- 
Tomas Vondra

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 14 Mar 2025 00:11:39 +0100

> to check if all processes saw the barrier? Any ideas?

Actually, scratch this. There already is a way to do this, by using
WaitForProcSignalBarrier. And the XLOG_CHECKSUMS processing already
calls this. So we should not see two barriers at the same time ...

> Or is there a reason why this would be unnecessary?

I still think this might be a problem. I wonder if we could maybe
leverage the barrier generation, to detect that we don't need to process
this barrier, because we already got the value directly ...

FWIW we'd have this problem even if postmaster was processing barriers,
because there'd always be a "gap" between the fork and ProcSignalInit()
registering the new process into the procsignal array.

regards

-- 
Tomas Vondra

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 14 Mar 2025 13:20:37 +0100

> registering the new process into the procsignal array.

I experimented with this a little bit, and unfortunately I ran into not
one, but two race conditions in this :-( I don't have reproducers, all
of this was done by manually adding sleep() calls / gdb breakpoints to
pause the processes for a while, but I'll try to explain what/why ...

1) race #1: SetDataChecksumsOn

The function (and all the other "SetDataChecksums" funcs) does this

    SpinLockAcquire(&XLogCtl->info_lck);
    XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
    SpinLockRelease(&XLogCtl->info_lck);

Now, imagine there's a sleep() before the EmitProcSignalBarrier. A new
process may start during that, and it'll read the current checksum value
from XLogCtl. And then the SetDataChecksumsOn() wakes up, and emits the
barrier. So far so good.

But the new backend is already registered in ProcSignal, so it'll get
the barrier too, and will try to set the local version to "on" again.
And kaboom - that hits the assert in AbsorbChecksumsOnBarrier():

The other "SetDataChecksums" have the same issue, except that in those
cases there are no asserts to trip. Only AbsorbChecksumsOnBarrier() has
such assert to check the state transition.

This is "ephemeral" in the sense that setting the value to "on" again
would be harmless, and indeed a non-assert build will run just fine.

2) race #2: InitPostgres

The InitPostgres does this:

    InitLocalControldata();

where InitLocalControldata gets the current checksum value from XLogCtl,
and ProcSignalInit registers the backend into the procsignal (which is
what barriers are based on).

Imagine there's a sleep() between these two calls, and the cluster does
not have checksums enabled. A backend will start, will read "off" from
XLogCtl, and then gets stuck on the sleep before it gets added to the
procsignal/barrier array.

Now, we enable checksums, and the instance goes through 'inprogress-on'
and 'on' states. This completes, and the backend wakes up and registers
itself into procsignal - but it won't get any barriers, of course.

So we end up with an instance with data_checksums="on", but this one
backend still believes data_checksums="on". This can cause a lot of
trouble, because it won't write blocks with checksums. I.e. this is
persistent data corruption.

I have been thinking about how to fix this. One way would be to
introduce some sort of locking, so that the two steps (update of the
XLogCtl version + barrier emit) and (local flag init + procsignal init)
would always happen atomically. So, something like this:

    SpinLockAcquire(&XLogCtl->info_lck);
    XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
    barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);
    SpinLockRelease(&XLogCtl->info_lck);

and

    InitLocalControldata();

But that seems pretty heavy-handed, it's definitely much more work while
holding a spinlock than I'm comfortable with, and I wouldn't be
surprised if there were deadlock cases etc. (FWIW I believe it needs to
use XLogCtl->info_lck, to make the value consistent with checkpoints.)

Anyway, I think a much simpler solution would be to reorder InitPostgres
like this:

    InitLocalControldata();

i.e. to first register into procsignal, and then read the new value.
AFAICS this guarantees we won't lose any checksum version updates. It
does mean we still can get a barrier for a value we've already seen, but
I think we should simply ignore this for the very first update.

Opinions? Other ideas how to fix this?

regards

-- 
Tomas Vondra

---

From: Daniel Gustafsson <[email-masked]>
Date: Fri, 14 Mar 2025 14:38:06 +0100

> On 14 Mar 2025, at 13:20, Tomas Vondra <[email-masked]> wrote:

> pause the processes for a while, but I'll try to explain what/why ...

Ugh. Thanks for this!

> would be harmless, and indeed a non-assert build will run just fine.

As mentioned off-list, being able to loosen the restriction for the first
barrier seen seem like a good way to keep this assertion.  Removing it is of
course the alternative solution, as it's not causing any issues, but given how
handy it's been to find actual issues it would be good to be able to keep it.

> use XLogCtl->info_lck, to make the value consistent with checkpoints.)

Yeah, that seems quite likely to introduce a new set if issues.

>    ProcSignalInit(MyCancelKeyValid, MyCancelKey);

Agreed.

> I think we should simply ignore this for the very first update.

Calling functions with sideeffects in setting state seems like a bad idea
before ProcSignalInit has run, that's thinko on my part in this patch.  Your
solution of reordering seems like the right way to handle this.

--
Daniel Gustafsson

---

From: Daniel Gustafsson <[email-masked]>
Date: Fri, 14 Mar 2025 15:06:52 +0100

> On 14 Mar 2025, at 14:38, Daniel Gustafsson <[email-masked]> wrote:

> solution of reordering seems like the right way to handle this.

0006 in the attached version is what I have used when testing the above, along
with an update to the copyright year which I had missed doing earlier.  It also
contains the fix in LocalProcessControlFile which I had in my local tree, I
think we need something like that at least.

--
Daniel Gustafsson

---

From: Tomas Vondra <[email-masked]>
Date: Sat, 15 Mar 2025 16:50:02 +0100

> think we need something like that at least.

Thanks, here's an updated patch version - I squashed all the earlier
parts, but I kept your changes and my adjustments separate, for clarity.
A couple comments:

1) I don't think the comment before InitialDataChecksumTransition was
entirely accurate, because it said we can see the duplicate state only
for "on" state. But AFAICS we can see duplicate values for any states,
except that we only have an assert for the "on" so we don't notice the
other cases. I wonder if we could strengthen this a bit, by adding some
asserts for the other states too.

2) I admit it's rather subjective, but I didn't like how you did the
assert in AbsorbChecksumsOnBarrier. But looking at it now in the diff,
maybe it was more readable ...

3) I renamed InitLocalControldata() to InitLocalDataChecksumVersion().
The name was entirely misleading, because it now initializes the flag in
XLogCtl, it has nothing to do with control file.

4) I realized AuxiliaryProcessMainCommon() may be a better place to
initialize the checksum flag for non-backend processes. In fact, doing
it in postmaster_child_launch() had the same race condition because it
happened before ProcSignalInit().

I'm sure there's cleanup possible in a bunch of places, but the really
bad thing is I realized the handling on a standby is not quite correct.
I don't know what exactly is happening, there's too many moving bits,
but here's what I see ...

Every now and then, after restarting the standby, it logs a bunch of
page verification failures. Stuff like this:

  WARNING:  page verification failed, calculated checksum 9856 but
            expected 0
  CONTEXT:  WAL redo at 0/3447BA8 for Heap2/VISIBLE:
            snapshotConflictHorizon: 0, flags: 0x03; blkref #0: rel
            1663/16384/16401, fork 2, blk 0 FPW; blkref #1: rel
            1663/16384/16401, blk 0
  WARNING:  page verification failed, LSN 0/CF54C10

This is after an immediate shutdown, but I've seen similar failures for
fast shutdowns too (the root causes may be different / may need a
different fix, not sure).

The instance restarts, and the "startup" process starts recovery

  LOG:  redo starts at 0/2000028

This matches LSN from the very first start of the standby - there were
no restart points since then, apparently. And since then the primary did
this with the checksums (per pg_waldump):

  lsn: 0/0ECCFC48, prev 0/0ECCFBA0, desc: CHECKSUMS inprogress-off
  lsn: 0/0ECD0168, prev 0/0ECD0128, desc: CHECKSUMS off

The instance already saw both records before the immediate shutdown (per
the logging in patch 0004), but after the restart the instance goes back
to having checksums enabled again

  data_checksum_version = 1

Which is correct, because it starts at 0/2000028, which is before either
of the XLOG_CHECKSUMS records. But then at 0/3447BA8 (which is *before*
either of the checksum changes) it tries to read a page from disk, and
hits a checksum error. That page is from the future (per the page LSN
logged by patch 0004), but it's still before both XLOG_CHECKSUMS
messages. So how come the page has pd_checksum 0?

I'd have understood if the page came "broken" from the primary, but I've
not seen a single page verification failure on that side (and it's
subject to the same fast/immediate restarts, etc).

I wonder if this might be related to how we enforce checkpoints only
when setting the checksums to "on" on the primary. Maybe that's safe on
primary but not on a standby?

FWIW I've seen similar issues for "fast" shutdowns too - at least the
symptoms are similar, but the mechanism might be a bit different. In
particular, I suspect there's some sort of thinko in updating the
data_checksum_version in the control file, but I can't put my finger on
it yet.

Another thing I noticed is this comment in CreateRestartPoint(), before
one of the early exits:

/*
 * If the last checkpoint record we've replayed is already our last
 * restartpoint, we can't perform a new restart point. We still update
 * minRecoveryPoint in that case, so that if this is a shutdown restart
 * point, we won't start up earlier than before. That's not strictly
 * necessary, but when hot standby is enabled, it would be rather weird
 * if the database opened up for read-only connections at a
 * point-in-time before the last shutdown. Such time travel is still
 * possible in case of immediate shutdown, though.
 * ...

I wonder if this "time travel backwards" might be an issue for this too,
because it might mean we end up picking the wrong data_checksum_version
from the control file. In any case, if this happens, we don't get to the
ControlFile->data_checksum_version update a bit further down. And
there's another condition that can skip that.

I'll continue investigating this next week, but at this point I'm quite
confused and would be grateful for any insights ...

regards

-- 
Tomas Vondra
From f72f5aa466fb308ee5816b72968677d85537cb57 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 7 Mar 2025 19:21:22 +0100
Subject: [PATCH v20250315 1/4] Online enabling and disabling of data checksums

 doc/src/sgml/monitoring.sgml                  |  208 ++-
 src/backend/access/transam/xlog.c             |  554 ++++++-
 src/backend/catalog/system_views.sql          |   20 +
 src/backend/postmaster/datachecksumsworker.c  | 1439 +++++++++++++++++
 src/backend/postmaster/launch_backend.c       |   13 +
 src/include/catalog/pg_control.h              |    6 +-
 src/include/commands/progress.h               |   16 +
 src/include/miscadmin.h                       |   10 +
 src/test/perl/PostgreSQL/Test/Cluster.pm      |   36 +
 src/test/regress/expected/rules.out           |   36 +
 src/test/subscription/t/013_partition.pl      |    3 +-
 59 files changed, 3263 insertions(+), 54 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 1c3810e1a04..0df5067c2c6 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -29882,6 +29882,77 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
 
   </sect2>

 
index aaa6586d3a4..8fec0277f97 100644

@@ -6828,6 +6829,205 @@ FROM pg_stat_get_backend_idset() AS backendid;
 
  </sect2>

 
+  <para>
 
   <para>
@@ -265,7 +266,7 @@
   </para>

    <para>
 
   </sect2>
+
 
   <sect1 id="wal-intro">

 const char *
 
 	return id;
index 799fc739e18..61da6d583cd 100644

@@ -647,6 +650,24 @@ static XLogRecPtr LocalMinRecoveryPoint;
 static TimeLineID LocalMinRecoveryPointTLI;
 static bool updateMinRecoveryPoint = true;
 
+/*
@@ -715,6 +736,8 @@ static void WALInsertLockAcquireExclusive(void);
 
@@ -828,9 +851,10 @@ XLogInsertRecord(XLogRecData *rdata,
@@ -843,7 +867,9 @@ XLogInsertRecord(XLogRecData *rdata,
 
 		if (doPageWrites &&
 			(!prevDoPageWrites ||
@@ -4239,6 +4265,12 @@ InitControlFile(uint64 sysidentifier, uint32 data_checksum_version)
 
 static void
@@ -4578,10 +4610,6 @@ ReadControlFile(void)
 		(SizeOfXLogLongPHD - SizeOfXLogShortPHD);

 /*
@@ -4615,13 +4643,410 @@ GetMockAuthenticationNonce(void)
 }
 
 /*
 
 /*
@@ -5063,6 +5488,11 @@ XLOGShmemInit(void)
 	XLogCtl->InstallXLogFileSegmentActive = false;
 	XLogCtl->WalWriterSleeping = false;
 
@@ -6194,6 +6624,33 @@ StartupXLOG(void)
 
+	/*
@@ -7049,6 +7506,12 @@ CreateCheckPoint(int flags)
 
+	/*
@@ -7304,6 +7767,9 @@ CreateCheckPoint(int flags)
 
@@ -7450,6 +7916,10 @@ CreateEndOfRecoveryRecord(void)
 
@@ -7792,6 +8262,10 @@ CreateRestartPoint(int flags)
 			if (flags & CHECKPOINT_IS_SHUTDOWN)
 				ControlFile->state = DB_SHUTDOWNED_IN_RECOVERY;
 		}
+
+		/* we shall start with the latest checksum version */
+		ControlFile->data_checksum_version = lastCheckPoint.data_checksum_version;
+
 		UpdateControlFile();
 	}
 	LWLockRelease(ControlFileLock);
@@ -8201,6 +8675,24 @@ XLogReportParameters(void)
 	}
 }
 
+/*
@@ -8629,6 +9121,46 @@ xlog_redo(XLogReaderState *record)
 
 /*

 

 
index a4d2cfdcaf5..6ffd31ce39c 100644
@@ -1334,6 +1334,26 @@ CREATE VIEW pg_stat_progress_copy AS

index 00000000000..6a201dca8de
@@ -0,0 +1,1439 @@
index 77fb877dbad..65bbf770d28 100644

@@ -282,6 +285,16 @@ postmaster_child_launch(BackendType child_type, int child_slot,
 			memcpy(MyClientSocket, client_sock, sizeof(ClientSocket));
 		}
 
+		/*
index d13846298bd..81eb84e7efb 100644
@@ -2946,6 +2946,11 @@ PostmasterStateMachine(void)
 									B_INVALID,
 									B_STANDALONE_BACKEND);
 
index 78f9a0a11c4..fe1922868b9 100644

 	/*

 				/*

 

 	/*

 		case B_AUTOVAC_WORKER:

 #

 
 	/* call only once */
 	Assert(!OidIsValid(AuthenticatedUserId));
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 4b2faf1ba9d..bae18b449aa 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -746,6 +746,11 @@ InitPostgres(const char *in_dbname, Oid dboid,
 	 */
 	SharedInvalBackendInit(false);
 
+	/*
 
 	/*
@@ -876,7 +881,7 @@ InitPostgres(const char *in_dbname, Oid dboid,
index 9c0b10ad4dc..5c00e6a5bf6 100644

 
-	{
@@ -5311,6 +5307,17 @@ struct config_enum ConfigureNamesEnum[] =
 		NULL, NULL, NULL
 	},
 
+	{

 

 /*
 
+	/*

 

 
index 63e834a6ce4..727042b73be 100644
 
@@ -80,6 +83,7 @@ typedef struct CheckPoint

 /*
@@ -219,7 +223,7 @@ typedef struct ControlFileData
 	bool		float8ByVal;	/* float8, int8, etc pass-by-value? */

 	/*
index 890822eaf79..6987bfb3ab9 100644
@@ -12256,6 +12256,23 @@
 
index 7c736e7b03b..b172a5f24ce 100644
@@ -157,4 +157,20 @@
 
index 6f16794eb63..133e7fde290 100644
 
@@ -389,6 +392,9 @@ extern PGDLLIMPORT BackendType MyBackendType;
 
@@ -534,6 +540,10 @@ extern Size EstimateClientConnectionInfoSpace(void);

 

index 0750ec3c474..17fb0decaff 100644

 /*

 
 ifeq ($(with_icu),yes)
 SUBDIRS += icu
 
 if ssl.found()
   subdir('ssl')
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index b105cba05a6..1c66360c16c 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -3748,6 +3748,42 @@ sub advance_wal
 	}
 }

 =back
index 62f69ac20b2..0dd383a76dd 100644
@@ -2041,6 +2041,42 @@ pg_stat_progress_create_index| SELECT s.pid,
index 93339ef3c58..3ed3f7d4216 100644
@@ -594,6 +595,10 @@ DataPageDeleteStack
@@ -4146,6 +4151,7 @@ xl_btree_split

From d56cb09458a5fbee539601c084fc7e72e5e9cfb5 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email-masked]>
Date: Fri, 14 Mar 2025 15:00:51 +0100
Subject: [PATCH v20250315 2/4] Reviewfixups

---
 src/backend/access/transam/xlog.c            | 26 +++++++++++++++++++-
 src/backend/postmaster/datachecksumsworker.c |  2 +-
 src/backend/utils/init/postinit.c            |  4 +--
 src/include/postmaster/datachecksumsworker.h |  2 +-
 src/test/checksum/Makefile                   |  2 +-
 5 files changed, 30 insertions(+), 6 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 61da6d583cd..e4c72f985e4 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -660,6 +660,16 @@ static bool updateMinRecoveryPoint = true;
  */
 static uint32 LocalDataChecksumVersion = 0;
 
+/*
+ * Flag to remember if the procsignalbarrier being absorbed for enabling
+ * checksums is the first one or not. The first procsignalbarrier can in rare
+ * circumstances cause a transition from 'on' to 'on' when a new process is
+ * spawned between the update of XLogCtl->data_checksum_version and the
+ * barrier being emitted.  This can only happen on the very first barrier so
+ * mark that with this flag.
+ */
+static bool InitialDataChecksumTransition = true;
+
 /*
  * Variable backing the GUC, keep it in sync with LocalDataChecksumVersion.
  * See SetLocalDataChecksumVersion().
@@ -4935,7 +4945,20 @@ AbsorbChecksumsOnInProgressBarrier(void)
 bool
 AbsorbChecksumsOnBarrier(void)
 {
-	Assert(LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
+	/*
+	 * If the process was spawned between updating XLogCtl and emitting the
+	 * barrier it will have seen the updated value, so for the first barrier
+	 * we accept both "on" and "inprogress-on".
+	 */
+	if (InitialDataChecksumTransition)
+	{
+		Assert((LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION) ||
+			   (LocalDataChecksumVersion == PG_DATA_CHECKSUM_VERSION));
+		InitialDataChecksumTransition = false;
+	}
+	else
+		Assert(LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
+
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_VERSION);
 	return true;
 }
@@ -5319,6 +5342,7 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc(sizeof(ControlFileData));
 	ReadControlFile();
+	SetLocalDataChecksumVersion(ControlFile->data_checksum_version);
 }
 
 /*
diff --git a/src/backend/postmaster/datachecksumsworker.c b/src/backend/postmaster/datachecksumsworker.c
index 6a201dca8de..81be2808895 100644
--- a/src/backend/postmaster/datachecksumsworker.c
+++ b/src/backend/postmaster/datachecksumsworker.c
@@ -150,7 +150,7 @@
  *     online operation).
  *
  *
- * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
  * Portions Copyright (c) 1994, Regents of the University of California
  *
  *
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index bae18b449aa..692570eb0f1 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -746,13 +746,13 @@ InitPostgres(const char *in_dbname, Oid dboid,
 	 */
 	SharedInvalBackendInit(false);
 
+	ProcSignalInit(MyCancelKeyValid, MyCancelKey);
+
 	/*
 	 * Set up backend local cache of Controldata values.
 	 */
 	InitLocalControldata();
 
-	ProcSignalInit(MyCancelKeyValid, MyCancelKey);
-
 	/*
 	 * Also set up timeout handlers needed for backend operation.  We need
 	 * these in every case except bootstrap.
diff --git a/src/include/postmaster/datachecksumsworker.h b/src/include/postmaster/datachecksumsworker.h
index 59c9000d646..0649232723d 100644
--- a/src/include/postmaster/datachecksumsworker.h
+++ b/src/include/postmaster/datachecksumsworker.h
@@ -4,7 +4,7 @@
  *	  header file for checksum helper background worker
  *
  *
- * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
  * Portions Copyright (c) 1994, Regents of the University of California
  *
  * src/include/postmaster/datachecksumsworker.h
diff --git a/src/test/checksum/Makefile b/src/test/checksum/Makefile
index fd03bf73df4..f287001301e 100644
--- a/src/test/checksum/Makefile
+++ b/src/test/checksum/Makefile
@@ -2,7 +2,7 @@
 #
 # Makefile for src/test/checksum
 #
-# Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+# Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
 # Portions Copyright (c) 1994, Regents of the University of California
 #
 # src/test/checksum/Makefile
-- 
2.48.1

From 95bfea2b12418671b5f9513dee107a0b18c8a78e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Fri, 14 Mar 2025 22:04:27 +0100
Subject: [PATCH v20250315 3/4] reworks

---
 src/backend/access/transam/xlog.c       | 45 +++++++++----------------
 src/backend/postmaster/auxprocess.c     | 19 +++++++++++
 src/backend/postmaster/launch_backend.c | 10 ------
 src/backend/utils/init/postinit.c       | 17 ++++++++--
 src/include/access/xlog.h               |  2 +-
 src/include/miscadmin.h                 |  1 -
 6 files changed, 51 insertions(+), 43 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index e4c72f985e4..064cc5555dc 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -661,12 +661,15 @@ static bool updateMinRecoveryPoint = true;
 static uint32 LocalDataChecksumVersion = 0;
 
 /*
- * Flag to remember if the procsignalbarrier being absorbed for enabling
- * checksums is the first one or not. The first procsignalbarrier can in rare
- * circumstances cause a transition from 'on' to 'on' when a new process is
+ * Flag to remember if the procsignalbarrier being absorbed for checksums
+ * is the first one. The first procsignalbarrier can in rare cases be for
+ * the state we've initialized, i.e. a duplicate. This may happen for any
+ * data_checksum_version value, but for PG_DATA_CHECKSUM_ON_VERSION this
+ * would trigger an assert failure (this is the only transition with an
+ * assert) when processing the barrier. This may happen if the process is
  * spawned between the update of XLogCtl->data_checksum_version and the
- * barrier being emitted.  This can only happen on the very first barrier so
- * mark that with this flag.
+ * barrier being emitted. This can only happen on the very first barrier
+ * so mark that with this flag.
  */
 static bool InitialDataChecksumTransition = true;
 
@@ -4938,6 +4941,7 @@ SetDataChecksumsOff(void)
 bool
 AbsorbChecksumsOnInProgressBarrier(void)
 {
+	/* XXX can't we check we're in OFF or INPROGRESSS_ON? */
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
 	return true;
 }
@@ -4950,22 +4954,19 @@ AbsorbChecksumsOnBarrier(void)
 	 * barrier it will have seen the updated value, so for the first barrier
 	 * we accept both "on" and "inprogress-on".
 	 */
-	if (InitialDataChecksumTransition)
-	{
-		Assert((LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION) ||
-			   (LocalDataChecksumVersion == PG_DATA_CHECKSUM_VERSION));
-		InitialDataChecksumTransition = false;
-	}
-	else
-		Assert(LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
+	Assert((LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION) ||
+		   (InitialDataChecksumTransition &&
+			(LocalDataChecksumVersion == PG_DATA_CHECKSUM_VERSION)));
 
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_VERSION);
+	InitialDataChecksumTransition = false;
 	return true;
 }
 
 bool
 AbsorbChecksumsOffInProgressBarrier(void)
 {
+	/* XXX can't we check we're in ON or INPROGRESSS_OFF? */
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);
 	return true;
 }
@@ -4973,6 +4974,7 @@ AbsorbChecksumsOffInProgressBarrier(void)
 bool
 AbsorbChecksumsOffBarrier(void)
 {
+	/* XXX can't we check we're in INPROGRESSS_OFF? */
 	SetLocalDataChecksumVersion(0);
 	return true;
 }
@@ -4986,7 +4988,7 @@ AbsorbChecksumsOffBarrier(void)
  * purpose enough to handle future cases.
  */
 void
-InitLocalControldata(void)
+InitLocalDataChecksumVersion(void)
 {
 	SpinLockAcquire(&XLogCtl->info_lck);
 	SetLocalDataChecksumVersion(XLogCtl->data_checksum_version);
@@ -5024,21 +5026,6 @@ SetLocalDataChecksumVersion(uint32 data_checksum_version)
 	}
 }
 
-/*
- * Initialize the various data checksum values - GUC, local, ....
- */
-void
-InitLocalDataChecksumVersion(void)
-{
-	uint32	data_checksum_version;
-
-	SpinLockAcquire(&XLogCtl->info_lck);
-	data_checksum_version = XLogCtl->data_checksum_version;
-	SpinLockRelease(&XLogCtl->info_lck);
-
-	SetLocalDataChecksumVersion(data_checksum_version);
-}
-
 /*
  * Get the local data_checksum_version (cached XLogCtl value).
  */
diff --git a/src/backend/postmaster/auxprocess.c b/src/backend/postmaster/auxprocess.c
index 4f6795f7265..50d5308816c 100644
--- a/src/backend/postmaster/auxprocess.c
+++ b/src/backend/postmaster/auxprocess.c
@@ -15,6 +15,7 @@
 #include <unistd.h>
 #include <signal.h>
 
+#include "access/xlog.h"
 #include "miscadmin.h"
 #include "pgstat.h"
 #include "postmaster/auxprocess.h"
@@ -68,6 +69,24 @@ AuxiliaryProcessMainCommon(void)
 
 	ProcSignalInit(false, 0);
 
+	/*
+	 * Initialize a local cache of the data_checksum_version, to be updated
+	 * by the procsignal-based barriers.
+	 *
+	 * This intentionally happens after initializing the procsignal, otherwise
+	 * we might miss a state change. This means we can get a barrier for the
+	 * state we've just initialized - but it can happen only once.
+	 *
+	 * The postmaster (which is what gets forked into the new child process)
+	 * does not handle barriers, therefore it may not have the current value
+	 * of LocalDataChecksumVersion value (it'll have the value read from the
+	 * control file, which may be arbitrarily old).
+	 *
+	 * NB: Even if the postmaster handled barriers, the value might still be
+	 * stale, as it might have changed after this process forked.
+	 */
+	InitLocalDataChecksumVersion();
+
 	/*
 	 * Auxiliary processes don't run transactions, but they may need a
 	 * resource owner anyway to manage buffer pins acquired outside
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 65bbf770d28..b06b5fb45dd 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -285,16 +285,6 @@ postmaster_child_launch(BackendType child_type, int child_slot,
 			memcpy(MyClientSocket, client_sock, sizeof(ClientSocket));
 		}
 
-		/*
-		 * update the LocalProcessControlFile to match XLogCtl->data_checksum_version
-		 *
-		 * XXX It seems the postmaster (which is what gets forked into the new
-		 * child process) does not absorb the checksum barriers, therefore it
-		 * does not update the value (except after a restart). Not sure if there
-		 * is some sort of race condition.
-		 */
-		InitLocalDataChecksumVersion();
-
 		/*
 		 * Run the appropriate Main function
 		 */
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 692570eb0f1..d1394fc05f9 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -749,9 +749,22 @@ InitPostgres(const char *in_dbname, Oid dboid,
 	ProcSignalInit(MyCancelKeyValid, MyCancelKey);
 
 	/*
-	 * Set up backend local cache of Controldata values.
+	 * Initialize a local cache of the data_checksum_version, to be updated
+	 * by the procsignal-based barriers.
+	 *
+	 * This intentionally happens after initializing the procsignal, otherwise
+	 * we might miss a state change. This means we can get a barrier for the
+	 * state we've just initialized - but it can happen only once.
+	 *
+	 * The postmaster (which is what gets forked into the new child process)
+	 * does not handle barriers, therefore it may not have the current value
+	 * of LocalDataChecksumVersion value (it'll have the value read from the
+	 * control file, which may be arbitrarily old).
+	 *
+	 * NB: Even if the postmaster handled barriers, the value might still be
+	 * stale, as it might have changed after this process forked.
 	 */
-	InitLocalControldata();
+	InitLocalDataChecksumVersion();
 
 	/*
 	 * Also set up timeout handlers needed for backend operation.  We need
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index aec3ea0bc63..615b2cf4ec8 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -243,7 +243,7 @@ extern bool AbsorbChecksumsOffInProgressBarrier(void);
 extern bool AbsorbChecksumsOnBarrier(void);
 extern bool AbsorbChecksumsOffBarrier(void);
 extern const char *show_data_checksums(void);
-extern void InitLocalControldata(void);
+extern void InitLocalDataChecksumVersion(void);
 extern bool GetDefaultCharSignedness(void);
 extern XLogRecPtr GetFakeLSNForUnloggedRel(void);
 extern Size XLOGShmemSize(void);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 133e7fde290..193ce1b4514 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -542,7 +542,6 @@ extern void RestoreClientConnectionInfo(char *conninfo);
 
 extern uint32 GetLocalDataChecksumVersion(void);
 extern uint32 GetCurrentDataChecksumVersion(void);
-extern void InitLocalDataChecksumVersion(void);

From d9a9caf0425543ef21700239dcd8d62ccaadfd5b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Sat, 15 Mar 2025 10:42:16 +0100
Subject: [PATCH v20250315 4/4] debug stuff

---
 src/backend/access/transam/xlog.c       | 86 ++++++++++++++++++++++++-
 src/backend/postmaster/launch_backend.c |  3 +
 src/backend/storage/ipc/procsignal.c    | 34 ++++++++++
 src/backend/storage/page/bufpage.c      | 35 ++++++++++
 src/backend/utils/init/miscinit.c       |  3 +
 src/bin/pg_controldata/pg_controldata.c |  2 +
 6 files changed, 160 insertions(+), 3 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 064cc5555dc..a01afc32af8 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4284,6 +4284,8 @@ InitControlFile(uint64 sysidentifier, uint32 data_checksum_version)
 	 * processes get the current value from. (Maybe it should go just there?)
 	 */
 	XLogCtl->data_checksum_version = data_checksum_version;
+
+	elog(LOG, "InitControlFile %p data_checksum_version %u", XLogCtl, ControlFile->data_checksum_version);
 }
 
 static void
@@ -4623,6 +4625,8 @@ ReadControlFile(void)
 		(SizeOfXLogLongPHD - SizeOfXLogShortPHD);
 
 	CalculateCheckpointSegments();
+
+	elog(LOG, "ReadControlFile ControlFile->data_checksum_version = %u", ControlFile->data_checksum_version);
 }
 
 /*
@@ -4758,6 +4762,8 @@ SetDataChecksumsOnInProgress(void)
 	XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION;
 	SpinLockRelease(&XLogCtl->info_lck);
 
+	elog(LOG, "SetDataChecksumsOnInProgress XLogCtl->data_checksum_version = %u", XLogCtl->data_checksum_version);
+
 	barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);
 
 	END_CRIT_SECTION();
@@ -4823,6 +4829,8 @@ SetDataChecksumsOn(void)
 	XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
 	SpinLockRelease(&XLogCtl->info_lck);
 
+	elog(LOG, "SetDataChecksumsOn XLogCtl->data_checksum_version = %u", XLogCtl->data_checksum_version);
+
 	barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);
 
 	END_CRIT_SECTION();
@@ -4861,6 +4869,7 @@ SetDataChecksumsOff(void)
 	/* If data checksums are already disabled there is nothing to do */
 	if (XLogCtl->data_checksum_version == 0)
 	{
+		elog(LOG, "SetDataChecksumsOff XLogCtl->data_checksum_version = %u (SKIP)", XLogCtl->data_checksum_version);
 		SpinLockRelease(&XLogCtl->info_lck);
 		return;
 	}
@@ -4885,6 +4894,8 @@ SetDataChecksumsOff(void)
 		XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION;
 		SpinLockRelease(&XLogCtl->info_lck);
 
+		elog(LOG, "SetDataChecksumsOff XLogCtl->data_checksum_version = %u", XLogCtl->data_checksum_version);
+
 		barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);
 
 		END_CRIT_SECTION();
@@ -4905,6 +4916,8 @@ SetDataChecksumsOff(void)
 	}
 	else
 	{
+		elog(LOG, "SetDataChecksumsOff XLogCtl->data_checksum_version = %u (SKIP)", XLogCtl->data_checksum_version);
+
 		/*
 		 * Ending up here implies that the checksums state is "inprogress-on"
 		 * or "inprogress-off" and we can transition directly to "off" from
@@ -4942,6 +4955,7 @@ bool
 AbsorbChecksumsOnInProgressBarrier(void)
 {
 	/* XXX can't we check we're in OFF or INPROGRESSS_ON? */
+	elog(LOG, "AbsorbChecksumsOnInProgressBarrier");
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
 	return true;
 }
@@ -4949,6 +4963,8 @@ AbsorbChecksumsOnInProgressBarrier(void)
 bool
 AbsorbChecksumsOnBarrier(void)
 {
+	elog(LOG, "AbsorbChecksumsOnBarrier");
+
 	/*
 	 * If the process was spawned between updating XLogCtl and emitting the
 	 * barrier it will have seen the updated value, so for the first barrier
@@ -4967,6 +4983,7 @@ bool
 AbsorbChecksumsOffInProgressBarrier(void)
 {
 	/* XXX can't we check we're in ON or INPROGRESSS_OFF? */
+	elog(LOG, "AbsorbChecksumsOffInProgressBarrier");
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);
 	return true;
 }
@@ -4975,6 +4992,7 @@ bool
 AbsorbChecksumsOffBarrier(void)
 {
 	/* XXX can't we check we're in INPROGRESSS_OFF? */
+	elog(LOG, "AbsorbChecksumsOffBarrier");
 	SetLocalDataChecksumVersion(0);
 	return true;
 }
@@ -4990,6 +5008,7 @@ AbsorbChecksumsOffBarrier(void)
 void
 InitLocalDataChecksumVersion(void)
 {
+	elog(LOG, "InitLocalDataChecksumVersion XLogCtl->data_checksum_version = %u", XLogCtl->data_checksum_version);
 	SpinLockAcquire(&XLogCtl->info_lck);
 	SetLocalDataChecksumVersion(XLogCtl->data_checksum_version);
 	SpinLockRelease(&XLogCtl->info_lck);
@@ -5004,6 +5023,7 @@ InitLocalDataChecksumVersion(void)
 void
 SetLocalDataChecksumVersion(uint32 data_checksum_version)
 {
+	elog(LOG, "SetLocalDataChecksumVersion %u", data_checksum_version);
 	LocalDataChecksumVersion = data_checksum_version;
 
 	switch (LocalDataChecksumVersion)
@@ -5499,9 +5519,15 @@ XLOGShmemInit(void)
 	XLogCtl->InstallXLogFileSegmentActive = false;
 	XLogCtl->WalWriterSleeping = false;

+	elog(LOG, "XLogCtl->data_checksum_version %u ControlFile->data_checksum_version %u (UPDATED)",
+		 XLogCtl->data_checksum_version, ControlFile->data_checksum_version);
+
 	SetLocalDataChecksumVersion(XLogCtl->data_checksum_version);
 
 	SpinLockInit(&XLogCtl->Insert.insertpos_lck);
@@ -6635,6 +6661,8 @@ StartupXLOG(void)
 	 */
 	CompleteCommitTsInitialization();
 
+	elog(LOG, "StartupXLOG XLogCtl->data_checksum_version = %u", XLogCtl->data_checksum_version);
+
 	/*
 	 * If we reach this point with checksums being enabled ("inprogress-on"
 	 * state), we notify the user that they need to manually restart the
@@ -6652,6 +6680,9 @@ StartupXLOG(void)
 	 * we know that we have a state where all backends have stopped validating
 	 * checksums and we can move to off instead of prompting the user to
 	 * perform any action.
+	 *
+	 * XXX Is it safe to access the data_checksum_version without holding the
+	 * spinlock?
 	 */
 	if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION)
 	{
@@ -6660,6 +6691,8 @@ StartupXLOG(void)
 		SpinLockAcquire(&XLogCtl->info_lck);
 		XLogCtl->data_checksum_version = 0;
 		SpinLockRelease(&XLogCtl->info_lck);
+
+		elog(LOG, "StartupXLOG XLogCtl->data_checksum_version = %u (UPDATED)", XLogCtl->data_checksum_version);
 	}
 
 	/*
@@ -7523,6 +7556,8 @@ CreateCheckPoint(int flags)
 	 */
 	checkPoint.data_checksum_version = XLogCtl->data_checksum_version;
 
@@ -7778,6 +7813,10 @@ CreateCheckPoint(int flags)

@@ -7928,6 +7967,10 @@ CreateEndOfRecoveryRecord(void)
 	ControlFile->minRecoveryPoint = recptr;
 	ControlFile->minRecoveryPointTLI = xlrec.ThisTimeLineID;

@@ -8101,6 +8144,14 @@ RecoveryRestartPoint(const CheckPoint *checkPoint, XLogReaderState *record)
 	XLogCtl->lastCheckPointEndPtr = record->EndRecPtr;
 	XLogCtl->lastCheckPoint = *checkPoint;
 	SpinLockRelease(&XLogCtl->info_lck);
+
+	elog(LOG, "RecoveryRestartPoint lastCheckPointRecPtr %X/%X XLogCtl->lastCheckPointEndPtr %X/%X redo %X/%X checksums xlogctl %u record %d file %u",
+		 LSN_FORMAT_ARGS(XLogCtl->lastCheckPointRecPtr),
+		 LSN_FORMAT_ARGS(XLogCtl->lastCheckPointEndPtr),
+		 LSN_FORMAT_ARGS(XLogCtl->lastCheckPoint.redo),
+		 XLogCtl->lastCheckPoint.data_checksum_version,
+		 checkPoint->data_checksum_version,
+		 ControlFile->data_checksum_version);
 }
 
 /*
@@ -8138,6 +8189,9 @@ CreateRestartPoint(int flags)
 	lastCheckPoint = XLogCtl->lastCheckPoint;
 	SpinLockRelease(&XLogCtl->info_lck);
 
+	elog(LOG, "CreateRestartPoint lastCheckPointRecPtr %X/%X lastCheckPointEndPtr %X/%X",
+		 LSN_FORMAT_ARGS(lastCheckPointRecPtr), LSN_FORMAT_ARGS(lastCheckPointEndPtr));
+
 	/*
 	 * Check that we're still in recovery mode. It's ok if we exit recovery
 	 * mode after this check, the restart point is valid anyway.
@@ -8166,9 +8220,11 @@ CreateRestartPoint(int flags)
 	if (XLogRecPtrIsInvalid(lastCheckPointRecPtr) ||
 		lastCheckPoint.redo <= ControlFile->checkPointCopy.redo)
 	{
-		ereport(DEBUG2,
-				(errmsg_internal("skipping restartpoint, already performed at %X/%X",
-								 LSN_FORMAT_ARGS(lastCheckPoint.redo))));
+		ereport(LOG,
+				(errmsg_internal("CreateRestartPoint: skipping restartpoint, already performed at %X/%X <= %X/%X lastCheckPointRecPtr %X/%X",
+								 LSN_FORMAT_ARGS(lastCheckPoint.redo),
+								 LSN_FORMAT_ARGS(ControlFile->checkPointCopy.redo),
+								 LSN_FORMAT_ARGS(lastCheckPointRecPtr))));
 
 		UpdateMinRecoveryPoint(InvalidXLogRecPtr, true);
 		if (flags & CHECKPOINT_IS_SHUTDOWN)
@@ -8236,6 +8292,8 @@ CreateRestartPoint(int flags)
 	 * end-of-recovery checkpoint.
 	 */
 	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+	elog(LOG, "CreateRestartPoint ControlFile->checkPointCopy.redo %X/%X lastCheckPoint.redo %X/%X",
+		 LSN_FORMAT_ARGS(ControlFile->checkPointCopy.redo), LSN_FORMAT_ARGS(lastCheckPoint.redo));
 	if (ControlFile->checkPointCopy.redo < lastCheckPoint.redo)
 	{
 		/*
@@ -8274,11 +8332,17 @@ CreateRestartPoint(int flags)
 				ControlFile->state = DB_SHUTDOWNED_IN_RECOVERY;
 		}

 		UpdateControlFile();
 	}
+	else
+		elog(LOG, "CreateRestartPoint: skipped ControlFile update");
 	LWLockRelease(ControlFileLock);
 
 	/*
@@ -9136,13 +9200,25 @@ xlog_redo(XLogReaderState *record)
 	{
 		xl_checksum_state state;
 		uint64		barrier;
+		XLogRecPtr	checkpointLsn;
+		uint32		value,
+					value_last;

 
@@ -9151,22 +9227,26 @@ xlog_redo(XLogReaderState *record)

 
 			default:
 				Assert(state.new_checksumtype == 0);
+				elog(LOG, "XLOG_CHECKSUMS emit PROCSIGNAL_BARRIER_CHECKSUM_OFF");
 				barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);
 				WaitForProcSignalBarrier(barrier);
 				break;
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index b06b5fb45dd..2119c5d59b4 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -239,6 +239,9 @@ postmaster_child_launch(BackendType child_type, int child_slot,
 	if (IsExternalConnectionBackend(child_type))
 		((BackendStartupData *) startup_data)->fork_started = GetCurrentTimestamp();

 		PG_TRY();
 		{
index 8bd5fed8c85..a8af9a7068b 100644
@@ -22,6 +22,7 @@
 #include "utils/memdebug.h"
 #include "utils/memutils.h"
 
+#include <execinfo.h>
 
 /* GUC variable */
 bool		ignore_checksum_failure = false;
@@ -136,10 +137,44 @@ PageIsVerifiedExtended(PageData *page, BlockNumber blkno, int flags)
 	if (checksum_failure)
 	{
 		if ((flags & PIV_LOG_WARNING) != 0)
+		{
+			XLogRecPtr	lsn = PageGetLSN(page);
+
 			ereport(WARNING,
 					(errcode(ERRCODE_DATA_CORRUPTED),
 					 errmsg("page verification failed, calculated checksum %u but expected %u",
 							checksum, p->pd_checksum)));
+			ereport(WARNING,
+					(errcode(ERRCODE_DATA_CORRUPTED),
+					 errmsg("page verification failed, LSN %X/%X", LSN_FORMAT_ARGS(lsn))));
+			ereport(WARNING,
+					(errcode(ERRCODE_DATA_CORRUPTED),
+					 errmsg("page verification failed, header flags %u lower %u upper %u special %u pagesize_version %u prune_xid %u",
+							p->pd_flags,
+							p->pd_lower,
+							p->pd_upper,
+							p->pd_special,
+							p->pd_pagesize_version,
+							p->pd_prune_xid)));
+
+			{
+#define BT_BUF_SIZE 32
+				int nptrs;
+				void *buffer[BT_BUF_SIZE];
+				char **strings;
+
+				nptrs = backtrace(buffer, BT_BUF_SIZE);
+				strings = backtrace_symbols(buffer, nptrs);
+
+				if (strings != NULL)
+				{
+					for (int i = 0; i < nptrs; i++)
+					{
+						elog(WARNING, "backtrace %d : %s", i, strings[i]);
+					}
+				}
+			}
+		}
 
 		if ((flags & PIV_REPORT_STAT) != 0)
 			pgstat_report_checksum_failure();
 
 /*
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index bea779eef94..3d176fc0a9f 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -280,6 +280,8 @@ main(int argc, char *argv[])
 		   ControlFile->checkPointCopy.oldestCommitTsXid);
 	printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
 		   ControlFile->checkPointCopy.newestCommitTsXid);
+	printf(_("Latest checkpoint's data_checksum_version:%u\n"),
+		   ControlFile->checkPointCopy.data_checksum_version);
 	printf(_("Time of latest checkpoint:            %s\n"),
 		   ckpttime_str);
 	printf(_("Fake LSN counter for unlogged rels:   %X/%X\n"),
-- 
2.48.1

---

From: Andres Freund <[email-masked]>
Date: Sat, 15 Mar 2025 12:26:12 -0400

Jo.

On 2025-03-15 16:50:02 +0100, Tomas Vondra wrote:
> Thanks, here's an updated patch version

FWIW, this fails in CI;

https://cirrus-ci.com/build/4678473324691456
On all OSs:
[16:08:36.331] #   Failed test 'options --locale-provider=icu --locale=und --lc-*=C: no stderr'
[16:08:36.331] #   at /tmp/cirrus-ci-build/src/bin/initdb/t/001_initdb.pl line 132.
[16:08:36.331] #          got: '2025-03-15 16:08:26.216 UTC [63153] LOG:  XLogCtl->data_checksum_version 0 ControlFile->data_checksum_version 0
[16:08:36.331] # 2025-03-15 16:08:26.216 UTC [63153] LOG:  XLogCtl->data_checksum_version 0 ControlFile->data_checksum_version 0 (UPDATED)

Windows & Compiler warnings:
[16:05:08.723] ../src/backend/storage/page/bufpage.c(25): fatal error C1083: Cannot open include file: 'execinfo.h': No such file or directory

[16:18:52.385] bufpage.c:25:10: fatal error: execinfo.h: No such file or directory
[16:18:52.385]    25 | #include <execinfo.h>
[16:18:52.385]       |          ^~~~~~~~~~~~

Greetings,

Andres Freund

---

From: Tomas Vondra <[email-masked]>
Date: Sat, 15 Mar 2025 18:32:57 +0100

> [16:18:52.385]    25 | #include <execinfo.h>

Yeah, that's just the "debug stuff" - I don't expect any of that to be
included in the commit, I only posted it for convenience. It adds a lot
of debug logging, which I hope might help others to understand what the
problem with checksums on standby is.

regards

-- 
Tomas Vondra

---

From: Alexander Korotkov <[email-masked]>
Date: Fri, 4 Apr 2025 10:55:32 +0300

Hi!

On Sat, Mar 15, 2025 at 7:33 PM Tomas Vondra <[email-masked]> wrote:

> > [16:08:36.331] #          got: '2025-03-15 16:08:26.216 UTC [63153]
> LOG:  XLogCtl->data_checksum_version 0 ControlFile->data_checksum_version 0
> > [16:08:36.331] # 2025-03-15 16:08:26.216 UTC [63153] LOG:
> XLogCtl->data_checksum_version 0 ControlFile->data_checksum_version 0
> (UPDATED)
> >
> > [16:05:08.723] ../src/backend/storage/page/bufpage.c(25): fatal error
> C1083: Cannot open include file: 'execinfo.h': No such file or directory
> >

I took a look at this patch.  I have following notes.
1) I think reporting of these errors could be better, more detailed.
Especially the second one could be similar to some of other errors on
checksums processing.
            ereport(ERROR,
                    (errmsg("failed to start background worker to process
data checksums")));
            ereport(ERROR,
                    (errmsg("unable to enable data checksums in cluster")));

2) ProcessAllDatabases() contains loop, which repeats scanning the new
databases for checkums.  It continues while there are new database on each
iteration.  Could we just limit the number of iterations to 2?  Given at
each step we're calling WaitForAllTransactionsToFinish(), everything that
gets created after first WaitForAllTransactionsToFinish() call should have
checksums enabled in the beginning.

------
Regards,
Alexander Korotkov
Supabase

---

From: Bernd Helmle <[email-masked]>
Date: Fri, 11 Jul 2025 17:53:58 +0200

Am Samstag, dem 15.03.2025 um 16:50 +0100 schrieb Tomas Vondra:

Hi,

Since i wanted to dig a little deeper in this patch i took the
opportunity and rebased it to current master, hopefully not having
broken something seriously.

Thanks,
	Bernd
From 374aae5c819a75b4743849b91c5a7e4fdb224f17 Mon Sep 17 00:00:00 2001
From: Bernd Helmle <Bernd Helmle [email-masked]>
Date: Thu, 10 Jul 2025 17:25:57 +0200
Subject: [PATCH 1/4] Online enabling and disabling of data checksums

 src/include/storage/proc.h                    |    6 +-
 58 files changed, 3262 insertions(+), 53 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index c28aa71f570..b60514c6869 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -30053,6 +30053,77 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
 
   </sect2>

index b88cac598e9..a4e16d03aae 100644
@@ -184,6 +184,8 @@
@@ -573,6 +575,27 @@
    </glossdef>
   </glossentry>
 
index 4265a22d4de..5464e236056 100644
@@ -3516,8 +3516,9 @@ description | Waiting for a newly initialized WAL file to reach durable storage
 
@@ -3527,8 +3528,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
       </para>
       <para>
 
@@ -6854,6 +6855,205 @@ FROM pg_stat_get_backend_idset() AS backendid;
 
  </sect2>

 
+  <para>
 
   <para>
@@ -265,7 +266,7 @@
   </para>

    <para>
 
   </sect2>
+
 
   <sect1 id="wal-intro">
index cd6c2a2f650..c50d654db30 100644

 const char *
 
 	return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 304b60933c9..4350f6abe3a 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -561,6 +561,9 @@ typedef struct XLogCtlData
 	 */
 	XLogRecPtr	lastFpwDisableRecPtr;

@@ -658,6 +661,24 @@ static XLogRecPtr LocalMinRecoveryPoint;
 static TimeLineID LocalMinRecoveryPointTLI;
 static bool updateMinRecoveryPoint = true;
 
+/*
@@ -726,6 +747,8 @@ static void WALInsertLockAcquireExclusive(void);
 
@@ -839,9 +862,10 @@ XLogInsertRecord(XLogRecData *rdata,
@@ -854,7 +878,9 @@ XLogInsertRecord(XLogRecData *rdata,
 
 		if (doPageWrites &&
 			(!prevDoPageWrites ||
@@ -4353,6 +4379,12 @@ InitControlFile(uint64 sysidentifier, uint32 data_checksum_version)
 
 static void
@@ -4692,10 +4724,6 @@ ReadControlFile(void)
 		(SizeOfXLogLongPHD - SizeOfXLogShortPHD);

 /*
@@ -4729,13 +4757,410 @@ GetMockAuthenticationNonce(void)
 }
 
 /*
 
 /*
@@ -5179,6 +5604,11 @@ XLOGShmemInit(void)
 	XLogCtl->InstallXLogFileSegmentActive = false;
 	XLogCtl->WalWriterSleeping = false;
 
@@ -6317,6 +6747,33 @@ StartupXLOG(void)
 
+	/*
@@ -7172,6 +7629,12 @@ CreateCheckPoint(int flags)
 
+	/*
@@ -7427,6 +7890,9 @@ CreateCheckPoint(int flags)
 
@@ -7577,6 +8043,10 @@ CreateEndOfRecoveryRecord(void)
 
@@ -7918,6 +8388,10 @@ CreateRestartPoint(int flags)
 			if (flags & CHECKPOINT_IS_SHUTDOWN)
 				ControlFile->state = DB_SHUTDOWNED_IN_RECOVERY;
 		}
+
+		/* we shall start with the latest checksum version */
+		ControlFile->data_checksum_version = lastCheckPoint.data_checksum_version;
+
 		UpdateControlFile();
 	}
 	LWLockRelease(ControlFileLock);
@@ -8329,6 +8803,24 @@ XLogReportParameters(void)
 	}
 }
 
+/*
@@ -8757,6 +9249,46 @@ xlog_redo(XLogReaderState *record)
 
 /*
index f0f88838dc2..13c181a3f8c 100644
 
@@ -2007,6 +2008,9 @@ verify_page_checksum(Page page, XLogRecPtr start_lsn, BlockNumber blkno,
 	if (PageIsNew(page) || PageGetLSN(page) >= start_lsn)
 		return true;

 

index b2d5332effc..9df7c24d535 100644
@@ -1351,6 +1351,26 @@ CREATE VIEW pg_stat_progress_copy AS

index bf6b55ee830..4a86d2588ff 100644
@@ -204,6 +204,9 @@ static child_process_kind child_process_kinds[] = {

@@ -284,6 +287,16 @@ postmaster_child_launch(BackendType child_type, int child_slot,
 			memcpy(MyClientSocket, client_sock, sizeof(ClientSocket));
 		}
 
+		/*
index 490f7ce3664..bfe17988e44 100644
@@ -2973,6 +2973,11 @@ PostmasterStateMachine(void)
 									B_INVALID,
 									B_STANDALONE_BACKEND);
 
index cc03f0706e9..f9f06821a8f 100644
index 2fa045e6b0f..44213d140ae 100644
@@ -150,6 +152,7 @@ CalculateShmemSize(int *num_semaphores)
 	size = add_size(size, AioShmemSize());
 
@@ -332,6 +335,7 @@ CreateOrAttachShmemStructs(void)
 
 	/*
index a9bb540b55a..c48abee2b60 100644
 
@@ -576,6 +577,18 @@ ProcessProcSignalBarrier(void)
 
 				/*

index dbb49ed9197..19cf6512e52 100644
@@ -107,7 +107,7 @@ PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_fail
 
@@ -1511,7 +1511,7 @@ PageSetChecksumCopy(Page page, BlockNumber blkno)
 	static char *pageCopy = NULL;

 	/*
@@ -1541,7 +1541,7 @@ void
 
 	((PageHeader) page)->pd_checksum = pg_checksum_page(page, blkno);
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 51256277e8d..dfef4a39bb0 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -388,6 +388,8 @@ pgstat_tracks_backend_bktype(BackendType bktype)
 		case B_CHECKPOINTER:
 		case B_IO_WORKER:
 		case B_STARTUP:
+		case B_DATACHECKSUMSWORKER_LAUNCHER:
+		case B_DATACHECKSUMSWORKER_WORKER:
 			return false;
 
 		case B_AUTOVAC_WORKER:
index d8d26379a57..faa32dd9f83 100644
 
index 4da68312b5f..8cbf879a1c7 100644
@@ -116,6 +116,8 @@ CHECKPOINT_DELAY_COMPLETE	"Waiting for a backend that blocks a checkpoint from c
@@ -352,6 +354,7 @@ DSMRegistry	"Waiting to read or update the dynamic shared memory registry."
 AioWorkerSubmissionQueue	"Waiting to access AIO worker submission queue."
 
 #
index 1c12ddbae49..9b867d25909 100644

-	if (!DataChecksumsEnabled())
-		PG_RETURN_NULL();
-
 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
 		result = 0;
 	else
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 43b4dbccc3d..e39498d1250 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -296,6 +296,12 @@ GetBackendTypeDesc(BackendType backendType)
 		case B_IO_WORKER:
 			backendDesc = gettext_noop("io worker");
 			break;
+		case B_DATACHECKSUMSWORKER_LAUNCHER:
+			backendDesc = "datachecksumsworker launcher";
+			break;
+		case B_DATACHECKSUMSWORKER_WORKER:
+			backendDesc = "datachecksumsworker worker";
+			break;
 		case B_LOGGER:
 			backendDesc = gettext_noop("logger");
 			break;
@@ -895,7 +901,8 @@ InitializeSessionUserIdStandalone(void)
 	 * workers, in slot sync worker and in background workers.
 	 */
 	Assert(!IsUnderPostmaster || AmAutoVacuumWorkerProcess() ||
-		   AmLogicalSlotSyncWorkerProcess() || AmBackgroundWorkerProcess());
+		   AmLogicalSlotSyncWorkerProcess() || AmBackgroundWorkerProcess() ||
+		   AmDataChecksumsWorkerProcess());
 
 	/* call only once */
 	Assert(!OidIsValid(AuthenticatedUserId));
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index c86ceefda94..c5202a8a8db 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -749,6 +749,11 @@ InitPostgres(const char *in_dbname, Oid dboid,
 	 */
 	SharedInvalBackendInit(false);
 
+	/*
+	 * Set up backend local cache of Controldata values.
+	 */
+	InitLocalControldata();
+
 	ProcSignalInit(MyCancelKey, MyCancelKeyLength);
 
 	/*
@@ -879,7 +884,7 @@ InitPostgres(const char *in_dbname, Oid dboid,
 					 errhint("You should immediately run CREATE USER \"%s\" SUPERUSER;.",
 							 username != NULL ? username : "postgres")));
 	}
-	else if (AmBackgroundWorkerProcess())
+	else if (AmBackgroundWorkerProcess() || AmDataChecksumsWorkerProcess())
 	{
 		if (username == NULL && !OidIsValid(useroid))
 		{
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index d14b1678e7f..8539dab0e48 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -491,6 +491,14 @@ static const struct config_enum_entry file_copy_method_options[] = {
 	{NULL, 0, false}
 };
 
@@ -616,7 +624,6 @@ static int	shared_memory_size_mb;
 
 #ifdef USE_ASSERT_CHECKING
@@ -1968,17 +1975,6 @@ struct config_bool ConfigureNamesBool[] =
 		NULL, NULL, NULL
 	},
 
-	{
-		{"data_checksums", PGC_INTERNAL, PRESET_OPTIONS,
-			gettext_noop("Shows whether data checksums are turned on for this cluster."),
-			NULL,
-			GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_RUNTIME_COMPUTED
-		},
-		&data_checksums,
-		false,
-		NULL, NULL, NULL
-	},
-
 	{
 		{"syslog_sequence_numbers", PGC_SIGHUP, LOGGING_WHERE,
 			gettext_noop("Add sequence number to syslog messages to avoid duplicate suppression."),
@@ -5418,6 +5414,17 @@ struct config_enum ConfigureNamesEnum[] =
 		NULL, assign_io_method, NULL
 	},
 
+	{
index f20be82862a..463384d323b 100644

index 90cef0864de..29684e82440 100644

 /*
@@ -736,6 +737,14 @@ check_control_data(ControlData *oldctrl,
 	 * check_for_isn_and_int8_passing_mismatch().
 	 */
 
+	/*

 

 

 /*

 	/*
index 1fc19146f46..945d50453be 100644
@@ -12352,6 +12352,23 @@

index 1bef98471c3..c86294b4d19 100644
@@ -366,6 +366,9 @@ typedef enum BackendType
 	B_WAL_SUMMARIZER,
 	B_WAL_WRITER,
 
@@ -391,6 +394,9 @@ extern PGDLLIMPORT BackendType MyBackendType;
 #define AmIoWorkerProcess()			(MyBackendType == B_IO_WORKER)
 
@@ -536,6 +542,10 @@ extern Size EstimateClientConnectionInfoSpace(void);

index aeb67c498c5..20631844bac 100644

 
+typedef enum ChecksumType
+{
+	DATA_CHECKSUMS_OFF = 0,
+	DATA_CHECKSUMS_ON,
+	DATA_CHECKSUMS_INPROGRESS_ON,
+	DATA_CHECKSUMS_INPROGRESS_OFF
+} ChecksumType;
+
 /*
  * Compute the checksum for a Postgres page.  The page must be aligned on a
  * 4-byte boundary.
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index a9681738146..ab3751636ac 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -84,3 +84,4 @@ PG_LWLOCK(50, DSMRegistry)
 PG_LWLOCK(51, InjectionPoint)
 PG_LWLOCK(52, SerialControl)
 PG_LWLOCK(53, AioWorkerSubmissionQueue)
+PG_LWLOCK(54, DataChecksumsWorker)
\ No newline at end of file
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 9f9b3fcfbf1..e2c1e178bc8 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -455,11 +455,11 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
  * Background writer, checkpointer, WAL writer, WAL summarizer, and archiver
  * run during normal operation.  Startup process and WAL receiver also consume
  * 2 slots, but WAL writer is launched only after startup has exited, so we
- * only need 6 slots.
+ * only need 6 slots to cover these. The DataChecksums worker and launcher
+ * can consume 2 slots when data checksums are enabled or disabled.
  */
 #define MAX_IO_WORKERS          32
-#define NUM_AUXILIARY_PROCS		(6 + MAX_IO_WORKERS)
-
+#define NUM_AUXILIARY_PROCS		(8 + MAX_IO_WORKERS)
 
index afeeb1ca019..c54c61e2cd8 100644
 
 /*

 
 ifeq ($(with_icu),yes)
 SUBDIRS += icu
 
 if ssl.found()
   subdir('ssl')
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 301766d2ed9..567af4b9624 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -3845,6 +3845,42 @@ sub advance_wal
 	}
 }

 =back
index dce8c672b40..832e6bb6866 100644
@@ -2066,6 +2066,42 @@ pg_stat_progress_create_index| SELECT s.pid,
index 776f1ad0e53..3634a4cde0b 100644
 io worker|relation|bulkread
 io worker|relation|bulkwrite
 io worker|relation|init
@@ -95,7 +111,7 @@ walsummarizer|wal|init
-(79 rows)
+(95 rows)
index 83192038571..0b2e0deabc1 100644
@@ -419,6 +419,7 @@ CheckPointStmt
@@ -611,6 +612,10 @@ DataPageDeleteStack
@@ -4239,6 +4244,7 @@ xl_btree_split
2.50.0

From 7676617ec45f8aa08c6f8ef7c5fc889ade4a9994 Mon Sep 17 00:00:00 2001
From: Bernd Helmle <Bernd Helmle [email-masked]>
Date: Thu, 10 Jul 2025 18:23:51 +0200
Subject: [PATCH 2/4]  Reviewfixups

---

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 4350f6abe3a..53890c3e726 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -671,6 +671,16 @@ static bool updateMinRecoveryPoint = true;
  */
 static uint32 LocalDataChecksumVersion = 0;
 
+/*
@@ -5049,7 +5059,20 @@ AbsorbChecksumsOnInProgressBarrier(void)
@@ -5435,6 +5458,7 @@ LocalProcessControlFile(bool reset)
 
 /*
index c5202a8a8db..785b8d4b04f 100644
@@ -749,13 +749,13 @@ InitPostgres(const char *in_dbname, Oid dboid,
 
+	ProcSignalInit(MyCancelKey, MyCancelKeyLength);
+
 	/*
 	 * Set up backend local cache of Controldata values.
 	 */
 	InitLocalControldata();
 
-	ProcSignalInit(MyCancelKey, MyCancelKeyLength);
-
 	/*
2.50.0

From f14cf88958858c637f18c11c4221dfed0a04ccba Mon Sep 17 00:00:00 2001
From: Bernd Helmle <Bernd Helmle [email-masked]>
Date: Thu, 10 Jul 2025 18:35:51 +0200
Subject: [PATCH 3/4] Rework handling of procsignalbarrier and local cache for
 data_checksion_version.

---

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 53890c3e726..097db70fcfc 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -672,12 +672,15 @@ static bool updateMinRecoveryPoint = true;
 static uint32 LocalDataChecksumVersion = 0;
 
 /*
 
@@ -5052,6 +5055,7 @@ SetDataChecksumsOff(void)
 bool
@@ -5064,22 +5068,19 @@ AbsorbChecksumsOnBarrier(void)

 bool
 AbsorbChecksumsOffInProgressBarrier(void)
 {
+	/* XXX can't we check we're in ON or INPROGRESSS_OFF? */
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);
 	return true;
 }
@@ -5087,6 +5088,7 @@ AbsorbChecksumsOffInProgressBarrier(void)
 bool
 AbsorbChecksumsOffBarrier(void)
 {
+	/* XXX can't we check we're in INPROGRESSS_OFF? */
 	SetLocalDataChecksumVersion(0);
 	return true;
 }
@@ -5100,7 +5102,7 @@ AbsorbChecksumsOffBarrier(void)
  * purpose enough to handle future cases.
  */
 void
-InitLocalControldata(void)
+InitLocalDataChecksumVersion(void)
 {
 	SpinLockAcquire(&XLogCtl->info_lck);
 	SetLocalDataChecksumVersion(XLogCtl->data_checksum_version);
@@ -5138,21 +5140,6 @@ SetLocalDataChecksumVersion(uint32 data_checksum_version)
 	}
 }
 
-/*
index a6d3630398f..4b56ef0eb81 100644

 	ProcSignalInit(NULL, 0);
 
+	/*
index 4a86d2588ff..955df32be5d 100644
@@ -287,16 +287,6 @@ postmaster_child_launch(BackendType child_type, int child_slot,
 
-		/*
-		 * update the LocalProcessControlFile to match XLogCtl->data_checksum_version
-		 *
-		 * XXX It seems the postmaster (which is what gets forked into the new
-		 * child process) does not absorb the checksum barriers, therefore it
-		 * does not update the value (except after a restart). Not sure if there
-		 * is some sort of race condition.
-		 */
-		InitLocalDataChecksumVersion();
-
 		/*
 		 * Run the appropriate Main function
 		 */
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 785b8d4b04f..7418deb10f5 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -752,9 +752,22 @@ InitPostgres(const char *in_dbname, Oid dboid,
 	ProcSignalInit(MyCancelKey, MyCancelKeyLength);
 
 	/*
 
 	/*
index c86294b4d19..5ce7a1cbc65 100644
@@ -544,7 +544,6 @@ extern void RestoreClientConnectionInfo(char *conninfo);

2.50.0

From 9292aac9d8c3a78ceab7aaa5a1c2d80432fbcbaa Mon Sep 17 00:00:00 2001
From: Bernd Helmle <Bernd Helmle [email-masked]>
Date: Fri, 11 Jul 2025 16:10:32 +0200
Subject: [PATCH 4/4] debug stuff

---

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 097db70fcfc..a6b6c0eaddd 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4398,6 +4398,8 @@ InitControlFile(uint64 sysidentifier, uint32 data_checksum_version)
 	 * processes get the current value from. (Maybe it should go just there?)
 	 */
 	XLogCtl->data_checksum_version = data_checksum_version;
+
+	elog(LOG, "InitControlFile %p data_checksum_version %u", XLogCtl, ControlFile->data_checksum_version);
 }
 
 static void
@@ -4737,6 +4739,8 @@ ReadControlFile(void)
 		(SizeOfXLogLongPHD - SizeOfXLogShortPHD);

 /*
@@ -4872,6 +4876,8 @@ SetDataChecksumsOnInProgress(void)
 	XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION;
 	SpinLockRelease(&XLogCtl->info_lck);

 	END_CRIT_SECTION();
@@ -4937,6 +4943,8 @@ SetDataChecksumsOn(void)
 	XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
 	SpinLockRelease(&XLogCtl->info_lck);

 	END_CRIT_SECTION();
@@ -4975,6 +4983,7 @@ SetDataChecksumsOff(void)
 	/* If data checksums are already disabled there is nothing to do */
 	if (XLogCtl->data_checksum_version == 0)
 	{
+		elog(LOG, "SetDataChecksumsOff XLogCtl->data_checksum_version = %u (SKIP)", XLogCtl->data_checksum_version);
 		SpinLockRelease(&XLogCtl->info_lck);
 		return;
 	}
@@ -4999,6 +5008,8 @@ SetDataChecksumsOff(void)
 		XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION;
 		SpinLockRelease(&XLogCtl->info_lck);

 		END_CRIT_SECTION();
@@ -5019,6 +5030,8 @@ SetDataChecksumsOff(void)
 	}
 	else
 	{
+		elog(LOG, "SetDataChecksumsOff XLogCtl->data_checksum_version = %u (SKIP)", XLogCtl->data_checksum_version);
+
 		/*
 		 * Ending up here implies that the checksums state is "inprogress-on"
 		 * or "inprogress-off" and we can transition directly to "off" from
@@ -5056,6 +5069,7 @@ bool
 AbsorbChecksumsOnInProgressBarrier(void)
 {
 	/* XXX can't we check we're in OFF or INPROGRESSS_ON? */
+	elog(LOG, "AbsorbChecksumsOnInProgressBarrier");
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);
 	return true;
 }
@@ -5063,6 +5077,8 @@ AbsorbChecksumsOnInProgressBarrier(void)
 bool
 AbsorbChecksumsOnBarrier(void)
 {
+	elog(LOG, "AbsorbChecksumsOnBarrier");
+
 	/*
 	 * If the process was spawned between updating XLogCtl and emitting the
 	 * barrier it will have seen the updated value, so for the first barrier
@@ -5081,6 +5097,7 @@ bool
 AbsorbChecksumsOffInProgressBarrier(void)
 {
 	/* XXX can't we check we're in ON or INPROGRESSS_OFF? */
+	elog(LOG, "AbsorbChecksumsOffInProgressBarrier");
 	SetLocalDataChecksumVersion(PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);
 	return true;
 }
@@ -5089,6 +5106,7 @@ bool
 AbsorbChecksumsOffBarrier(void)
 {
 	/* XXX can't we check we're in INPROGRESSS_OFF? */
+	elog(LOG, "AbsorbChecksumsOffBarrier");
 	SetLocalDataChecksumVersion(0);
 	return true;
 }
@@ -5104,6 +5122,7 @@ AbsorbChecksumsOffBarrier(void)
 void
 InitLocalDataChecksumVersion(void)
 {
+	elog(LOG, "InitLocalDataChecksumVersion XLogCtl->data_checksum_version = %u", XLogCtl->data_checksum_version);
 	SpinLockAcquire(&XLogCtl->info_lck);
 	SetLocalDataChecksumVersion(XLogCtl->data_checksum_version);
 	SpinLockRelease(&XLogCtl->info_lck);
@@ -5118,6 +5137,7 @@ InitLocalDataChecksumVersion(void)
 void
 SetLocalDataChecksumVersion(uint32 data_checksum_version)
 {
+	elog(LOG, "SetLocalDataChecksumVersion %u", data_checksum_version);
 	LocalDataChecksumVersion = data_checksum_version;
 
 	switch (LocalDataChecksumVersion)
@@ -5615,9 +5635,15 @@ XLOGShmemInit(void)
 	XLogCtl->InstallXLogFileSegmentActive = false;
 	XLogCtl->WalWriterSleeping = false;

 
 	SpinLockInit(&XLogCtl->Insert.insertpos_lck);
@@ -6758,6 +6784,8 @@ StartupXLOG(void)
 	 */
 	CompleteCommitTsInitialization();
 
@@ -6775,6 +6803,9 @@ StartupXLOG(void)
@@ -6783,6 +6814,8 @@ StartupXLOG(void)
 
 	/*
@@ -7646,6 +7679,8 @@ CreateCheckPoint(int flags)
 	 */
 	checkPoint.data_checksum_version = XLogCtl->data_checksum_version;
 
@@ -7901,6 +7936,10 @@ CreateCheckPoint(int flags)

@@ -8055,6 +8094,10 @@ CreateEndOfRecoveryRecord(void)
 	ControlFile->minRecoveryPoint = recptr;
 	ControlFile->minRecoveryPointTLI = xlrec.ThisTimeLineID;

@@ -8227,6 +8270,14 @@ RecoveryRestartPoint(const CheckPoint *checkPoint, XLogReaderState *record)
 
 /*
@@ -8264,6 +8315,9 @@ CreateRestartPoint(int flags)
 	lastCheckPoint = XLogCtl->lastCheckPoint;
 	SpinLockRelease(&XLogCtl->info_lck);
 
+	elog(LOG, "CreateRestartPoint lastCheckPointRecPtr %X/%X lastCheckPointEndPtr %X/%X",
+		 LSN_FORMAT_ARGS(lastCheckPointRecPtr), LSN_FORMAT_ARGS(lastCheckPointEndPtr));
+
 	/*
 	 * Check that we're still in recovery mode. It's ok if we exit recovery
 	 * mode after this check, the restart point is valid anyway.
@@ -8292,9 +8346,11 @@ CreateRestartPoint(int flags)
 	if (XLogRecPtrIsInvalid(lastCheckPointRecPtr) ||
 		lastCheckPoint.redo <= ControlFile->checkPointCopy.redo)
 	{
-		ereport(DEBUG2,
-				errmsg_internal("skipping restartpoint, already performed at %X/%08X",
-								LSN_FORMAT_ARGS(lastCheckPoint.redo)));
+		ereport(LOG,
+				(errmsg_internal("CreateRestartPoint: skipping restartpoint, already performed at %X/%X <= %X/%X lastCheckPointRecPtr %X/%X",
+								 LSN_FORMAT_ARGS(lastCheckPoint.redo),
+								 LSN_FORMAT_ARGS(ControlFile->checkPointCopy.redo),
+								 LSN_FORMAT_ARGS(lastCheckPointRecPtr))));
 
@@ -8362,6 +8418,8 @@ CreateRestartPoint(int flags)
@@ -8400,11 +8458,17 @@ CreateRestartPoint(int flags)

 		UpdateControlFile();
 	}
+	else
 
 	/*
@@ -9264,13 +9328,25 @@ xlog_redo(XLogReaderState *record)
 	{
 		xl_checksum_state state;
 		uint64		barrier;
+		XLogRecPtr	checkpointLsn;
+		uint32		value,
+					value_last;

 
@@ -9279,22 +9355,26 @@ xlog_redo(XLogReaderState *record)

 
 			default:
index 955df32be5d..d2ae41172a6 100644
@@ -241,6 +241,9 @@ postmaster_child_launch(BackendType child_type, int child_slot,
 
+	elog(LOG, "postmaster_child_launch: LocalDataChecksumVersion %u xlog %u", GetLocalDataChecksumVersion(),
+	GetCurrentDataChecksumVersion());
+
 #ifdef EXEC_BACKEND
 	pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
 							startup_data, startup_data_len, client_sock);
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c48abee2b60..d5e1bb12ba5 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -554,6 +554,40 @@ ProcessProcSignalBarrier(void)
 
 		PG_TRY();
 		{
index 19cf6512e52..7c8f4d73606 100644
 
+#include <execinfo.h>
 
 /* GUC variable */
@@ -149,10 +150,44 @@ PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_fail
 		if ((flags & (PIV_LOG_WARNING | PIV_LOG_LOG)) != 0)
 			ereport(flags & PIV_LOG_WARNING ? WARNING : LOG,
 
 		if (header_sane && (flags & PIV_IGNORE_CHECKSUM_FAILURE))
 			return true;
index e39498d1250..25f35c2690c 100644
 
 /*
index 10de058ce91..acf5c7b026e 100644
 	printf(_("Fake LSN counter for unlogged rels:   %X/%08X\n"),
-- 
2.50.0

---

From: Daniel Gustafsson <[email-masked]>
Date: Sat, 12 Jul 2025 21:39:30 +0200

> On 11 Jul 2025, at 17:53, Bernd Helmle <[email-masked]> wrote:

> broken something seriously.

Thanks, much appreciated!

--
Daniel Gustafsson

---

From: Daniel Gustafsson <[email-masked]>
Date: Sat, 16 Aug 2025 21:34:03 +0200

Attached is a rebase on top of the func.sgml changes which caused this to no
longer apply.

This version is also substantially updated with a new injection point based
test suite, fixed a few bugs (found by said test suite), added checkpoint to
disabling checksums, code cleanup, more granular wait events, comment rewrites
and additions and more smaller cleanups.

--
Daniel Gustafsson

---

From: Bruce Momjian <[email-masked]>
Date: Tue, 19 Aug 2025 12:21:41 -0400

> and additions and more smaller cleanups.

I am very glad you went simple and didn't attempt restarting this
process from the place it stopped:

    If the cluster is stopped while in <literal>inprogress-on</literal>
    mode, for any reason, then this process must be
    restarted manually. To do this, re-execute the function
    <function>pg_enable_data_checksums()</function> once the cluster has
    been restarted. The process will start over, there is no support for
    resuming work from where it was interrupted.

-- 
  Bruce Momjian  <[email-masked]>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 20 Aug 2025 16:37:33 +0200

> and additions and more smaller cleanups.

Thanks for the updated patch.

The injection points seem like a huge improvement, allowing testing of
different code paths in a more deterministic way.

I started running the stress test, using pretty much exactly the version
posted in March [1]. And so far I noticed only one issue, when the
standby reports mismatched checksums on a fsm:

LOG:  page verification failed, calculated checksum 24786 but expected 24760
CONTEXT:  WAL redo at 0/0344A290 for Heap2/MULTI_INSERT+INIT: ntuples:
185, flags: 0x28; blkref #0: rel 1663/16384/16403, blk 0
LOG:  invalid page in block 2 of relation base/16384/16403_fsm; zeroing
out page
CONTEXT:  WAL redo at 0/0344A290 for Heap2/MULTI_INSERT+INIT: ntuples:
185, flags: 0x28; blkref #0: rel 1663/16384/16403, blk 0
WARNING:  invalid page in block 2 of relation base/16384/16403_fsm;
zeroing out page
CONTEXT:  WAL redo at 0/0344A290 for Heap2/MULTI_INSERT+INIT: ntuples:
185, flags: 0x28; blkref #0: rel 1663/16384/16403, blk 0
LOG:  page verification failed, calculated checksum 37048 but expected 0
CONTEXT:  WAL redo at 0/0344D7E0 for Heap2/MULTI_INSERT+INIT: ntuples:
61, flags: 0x28; blkref #0: rel 1663/16384/16400, blk 0
LOG:  invalid page in block 2 of relation base/16384/16400_fsm; zeroing
out page

This happens quite regularly, it's not hard to hit. But I've only seen
it to happen on a FSM, and only right after immediate shutdown. I don't
think that's quite expected.

I believe the built-in TAP tests (with injection points) can't catch
this, because there's no concurrent activity while flipping checksums
on/off. It'd be good to do something like that, by running pgbench in
the background, or something like that.

I also don't see any restarts of the primary/standby. That might be good
to do too.

I plan to randomize the stress test a bit more, once this FSM issue gets
fixed. Maybe that'll find some additional issues.

[1]
https://www.postgresql.org/message-id/[email-masked]

-- 
Tomas Vondra

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 20 Aug 2025 19:02:57 +0200

Hi,

I think there's a minor issue in how pg_checksums validates state before
checking the data.

The current patch simply does:

  if (ControlFile->data_checksum_version == 0 &&
      mode == PG_MODE_CHECK)
      pg_fatal("data checksums are not enabled in cluster");

and that worked when the version was either 0 or 1. But now it can be
also 2 or 3, for inprogress-on / inprogress-off, and if the cluster gets
shut down at the right moment, that can end in the control file.

It doesn't make sense to verify checksums in such cluster, pg_checksums
should handle that as "off", i.e. error out.

regards

-- 
Tomas Vondra

---

From: Daniel Gustafsson <[email-masked]>
Date: Mon, 25 Aug 2025 20:32:51 +0200

> On 20 Aug 2025, at 16:37, Tomas Vondra <[email-masked]> wrote:

> the background, or something like that.

In searching for this bug I opted for implementing a version of the stress
tests as a TAP test, see 006_concurrent_pgbench.pl in the attached patch
version.  It's gated behind PG_TEST_EXTRA since it's clearly not something
which can be enabled by default (if this goes in this need to be re-done to
provide two levels IMO, but during testing this is more convenient).  I'm
curious to see which improvements you can think to make it stress the code to
the breaking point.

> shut down at the right moment, that can end in the control file.

Good point, I've changed the test to check for checksums being enabled rather
than checking if they are disabled.

--
Daniel Gustafsson

---

From: Tomas Vondra <[email-masked]>
Date: Tue, 26 Aug 2025 01:06:24 +0200

> curious to see which improvements you can think to make it stress the code to

I think this TAP looks very nice, but there's a couple issues with it.
See the attached patch fixing those.

1) I think test_checksums should be in src/test/modules/Makefile?

2) The test_checksums/Makefile didn't seem to work for me, I was getting

Makefile:23: *** recipe commences before first target.  Stop.

Because there was a missing "\" so I had to fix that. And then it was
complaining about Makefile.global or something, so I fixed that by
cargo-culting what other Makefiles in test modules do. Now it seems to
work for me. I guess you're on meson?

3) I'm no perl expert, but AFAICS the test wasn't really running the
pgbench, for a couple of reasons. It was passing "-q" to pgbench, but
that's only for initialization. The clusters had max_connections=10, but
the pgbench was using "-c 10", so I was getting "too many connections".
It was not setting "$pgbench_running = 1" so the other loops were
getting "too many connections" too. Another thing is I'm not sure it's
OK to pass '' to IPC::Run::start, I think it'll take it as an argument,
confusing pgbench.

With these changes it runs for me, and I even saw some

   LOG: page verification failed

in tmp_check/log/006_concurrent_pgbench_standby_1.log. But it takes a
while - a couple minutes, maybe? I think I saw it at

    t/006_concurrent_pgbench.pl .. 427/?

or something like that. I think the bash version did a couple things
differently, which might make the failures more frequent (but it's just
a wild guess).

In particular, I think the script restarts the two nodes independently,
while the TAP always stops both primary and standby, in this order. I
think it'd be useful to restart one or both.

The other thing is the bash script added some random delays/sleep, which
increases the test duration, but it also means generating somewhat
random amounts of data, etc. It also randomized some other stuff (scale,
client count, ...). But that can wait.

regards

-- 
Tomas Vondra
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 903a8ac151a..c8f2747b261 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
 		  test_aio \
 		  test_binaryheap \
 		  test_bloomfilter \
+		  test_checksums \
 		  test_copy_callbacks \
 		  test_custom_rmgrs \
 		  test_ddl_deparse \
diff --git a/src/test/modules/test_checksums/Makefile b/src/test/modules/test_checksums/Makefile
index b9136bb513f..a5b6259a728 100644
--- a/src/test/modules/test_checksums/Makefile
+++ b/src/test/modules/test_checksums/Makefile
@@ -9,28 +9,32 @@
 #
 #-------------------------------------------------------------------------
 
-subdir = src/test/checksum
-top_builddir = ../../..
-include $(top_builddir)/src/Makefile.global
-
 EXTRA_INSTALL = src/test/modules/injection_points
 
 export enable_injection_points
 
 MODULE_big = test_checksums
 OBJS = \
-	$(WIN32RES)
+	$(WIN32RES) \
 	test_checksums.o
 PGFILEDESC = "test_checksums - test code for data checksums"
 
 EXTENSION = test_checksums
 DATA = test_checksums--1.0.sql
 
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_checksums
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
 check:
 	$(prove_check)
 
 installcheck:
 	$(prove_installcheck)
-
-clean distclean maintainer-clean:
-	rm -rf tmp_check
diff --git a/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl b/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl
index 630abee9c63..364225933ca 100644
--- a/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl
+++ b/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl
@@ -36,21 +36,31 @@ if (!$ENV{PG_TEST_EXTRA} || $ENV{PG_TEST_EXTRA} !~ /\bchecksum_extended\b/)
 	plan skip_all => 'Extended tests not enabled';
 }
 
-if ($ENV{enable_injection_points} ne 'yes')
+# Start a pgbench run in the background against the server specified via the
+# port passed as parameter
+sub background_ro_pgbench
 {
-	plan skip_all => 'Injection points not supported by this build';
+	my ($port, $readonly, $stdin, $stdout, $stderr) = @_;
+
+	my $pgbench_primary = IPC::Run::start(
+		[
+			'pgbench', '-p', $port, '-S',
+			'-T', '600', '-c', '10', 'postgres'
+		],
+		'<' => \$stdin,
+		'>' => \$stdout,
+		'2>' => \$stderr,
+		IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
 }
 
-# Start a pgbench run in the background against the server specified via the
-# port passed as parameter
-sub background_pgbench
+sub background_rw_pgbench
 {
 	my ($port, $readonly, $stdin, $stdout, $stderr) = @_;
 
 	my $pgbench_primary = IPC::Run::start(
 		[
-			'pgbench', '-p', $port, ($readonly == 1 ? '-S' : ''),
-			'-T', '600', '-c', '10', '-q', 'postgres'
+			'pgbench', '-p', $port,
+			'-T', '600', '-c', '10', 'postgres'
 		],
 		'<' => \$stdin,
 		'>' => \$stdout,
@@ -141,6 +151,7 @@ for (my $i = 1; $i <= 100; $i++)
 # they are caught up and in sync.
 $node_primary = PostgreSQL::Test::Cluster->new('main');
 $node_primary->init(allows_streaming => 1, no_data_checksums => 1);
+$node_primary->append_conf('postgresql.conf', "max_connections = 30");
 $node_primary->start;
 $node_primary->safe_psql('postgres', 'CREATE EXTENSION test_checksums;');
 # Create some content to have un-checksummed data in the cluster
@@ -177,15 +188,17 @@ for (my $i = 0; $i < $TEST_ITERATIONS; $i++)
 		# Start a pgbench in the background against the primary
 		my ($pgb_primary_stdin, $pgb_primary_stdout, $pgb_primary_stderr) =
 		  ('', '', '');
-		background_pgbench($node_primary->port, 0, $pgb_primary_stdin,
+		background_rw_pgbench($node_primary->port, 0, $pgb_primary_stdin,
 			$pgb_primary_stdout, $pgb_primary_stderr);
 
 		# Start a select-only pgbench in the background on the standby
 		my ($pgb_standby_1_stdin, $pgb_standby_1_stdout,
 			$pgb_standby_1_stderr)
 		  = ('', '', '');
-		background_pgbench($node_standby_1->port, 1, $pgb_standby_1_stdin,
+		background_ro_pgbench($node_standby_1->port, 1, $pgb_standby_1_stdin,
 			$pgb_standby_1_stdout, $pgb_standby_1_stderr);
+
+		$pgbench_running = 1;
 	}
 
 	$node_primary->safe_psql('postgres', "UPDATE t SET a = a + 1;");

---

From: Daniel Gustafsson <[email-masked]>
Date: Wed, 27 Aug 2025 10:30:18 +0200

> On 26 Aug 2025, at 01:06, Tomas Vondra <[email-masked]> wrote:

> See the attached patch fixing those.

Thanks, I have incorporated (most of) your patch in the attached.  I did keep
the PG_TEST_EXTRA check for injection points though which I assume were removed
out of mistake.

>    t/006_concurrent_pgbench.pl .. 427/?

That's very interesting, I have been running it to timeout several times in a
row without hitting any verification failures.  Will keep running.

> think it'd be useful to restart one or both.

Done in the attached, it will now randomly stop one or both or none.  If the
node is stopped I've added an offline pg_checksum step to validate the
datafiles as a why-not test.

> client count, ...). But that can wait.

Added as well in a few places, maybe more can be sprinkled in.

--
Daniel Gustafsson

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 27 Aug 2025 11:39:35 +0200

> the PG_TEST_EXTRA check for injection points though which I assume were removed

Yes, that was a mistake.

> row without hitting any verification failures.  Will keep running.

Just to be clear - I don't see any pg_checksums failures either. I only
see failures in the standby log, and I don't think the script checks
that (it probably should).

> Added as well in a few places, maybe more can be sprinkled in.

Thanks. I'll take a look.

regards

-- 
Tomas Vondra

---

From: Daniel Gustafsson <[email-masked]>
Date: Wed, 27 Aug 2025 13:00:04 +0200

> On 27 Aug 2025, at 11:39, Tomas Vondra <[email-masked]> wrote:

> that (it probably should).

Right, that's what I'm been checking too.  I have been considering adding
another background process for monitoring all the log entries but I just
thought of a much simpler solution.  When the clusters are turned off we can
take the opportunity to slurp the log written since last restart and inspect
it. The attached adds this.

It would probably be good to at some point clean this up a little by placing
all of variables for a single node in an associative hash which can be passed
around, and place repeated code in subroutines etc..

--
Daniel Gustafsson

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 27 Aug 2025 14:39:38 +0200

> it. The attached adds this.

There's still a couple issues, unfortunately. First, this may not do
what you intended:

  sub cointoss
  {
    return int(rand(2) == 1);
  }

The rand() call returns values in [0, 2.0), it's very unlikely to
produce 1, and so there are no restarts. I fixed this to

  sub cointoss
  {
    return int(rand() < 0.5);
  }

and then it starts working, restarting with ~50% probability.

Then it hits another problem when calling pg_checksums. That only works
after a "clean" shutdown, not after immediate one.This can't just check
is_alive, it needs to remember how exactly was the server stopped and
only verify checksums for 'fast' mode. I never saw any failures in
pg_checksums, so I just commented that out.

Then it starts working as intended, I think. I only did a couple runs so
far, but I haven't found any checksum failures ... Then I realized I did
the earlier runs with slightly stale master HEAD (38c5fbd97ee6a to be
precise), while this time I did git pull.

And this happened on Friday:

commit c13070a27b63d9ce4850d88a63bf889a6fde26f0
Author: Alexander Korotkov <[email-masked]>
Date:   Fri Aug 22 18:44:39 2025 +0300

    Revert "Get rid of WALBufMappingLock"

    This reverts commit bc22dc0e0ddc2dcb6043a732415019cc6b6bf683.
    It appears that conditional variables are not suitable for use
    inside critical sections.  If WaitLatch()/WaitEventSetWaitBlock()
    face postmaster death, they exit, releasing all locks instead of
    PANIC.  In certain situations, this leads to data corruption.

    ...

I think it's very likely the checksums were broken by this. After all,
that linked thread has subject "VM corruption on standby" and I've only
ever seen checksum failures on standby on the _vm fork.

There's also an issue when calling teardown_node at the end. That won't
work if the instance got stopped in the last round, and the test will
fail like this:

t/006_concurrent_pgbench.pl .. 379/? # Tests were run but no plan was
declared and done_testing() was not seen.
# Looks like your test exited with 29 just after 379.
t/006_concurrent_pgbench.pl .. Dubious, test returned 29 (wstat 7424,
0x1d00)
All 379 subtests passed

> around, and place repeated code in subroutines etc..

Yeah. For me Perl is hard to read in any case, but if we can cleanup
this a little bit before additional cases, that'd be nice.

regards

-- 
Tomas Vondra

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 27 Aug 2025 14:42:05 +0200

> ever seen checksum failures on standby on the _vm fork.

Forgot to mention - I did try with c13070a27b reverted, and with that I
can reproduce the checksum failures again (using the fixed TAP test).

It's not a definitive proof, but it's a hint c13070a27b63 was causing
the checksum failures.

regards

-- 
Tomas Vondra

---

From: Tomas Vondra <[email-masked]>
Date: Thu, 28 Aug 2025 18:11:18 +0200

Hi,

I spent a bit more time fixing the TAP test. The attached patch makes it
"work" for me (or I think it should, in principle). I'm not saying it's
the best way to do stuff.

With the patch applied, I tried running it, and I got a failure when
running pg_checksums. There's a log snippet describing the issue, but
AFAICS it's happening like this:

1) checksums are disabled
2) flip_data_checksums gets called
3) both clusters go through 'inprogress-on' and 'on' states
4) primary gets shutdown in 'immediate' mode
5) standby gets shutdown in 'fast' mode
6) we try to validate checksums on the standby, but control file still
says checksums=inprogress-on

This seems like a bug to me - AFAICS the expectation is that after fast
shutdown, we don't forget the checksum state. Or is that expected? In
that case the TAP test probably needs to check the control file, instead
of relying on the perl variable $data_checksum_state. Or maybe it should
check that the control file has the correct / expected state?

FWIW I don't think the primary shutdown matters. I've seen multiple of
these failures, and it happens even without primary shutdown. But the
standby "fast" shutdown is always there.

But this also shows a limitation of the TAP test - it never triggers the
shutdowns while flipping the checksums (in flip_data_checksums). I think
that's something worth testing.

regards

-- 
Tomas Vondra
diff --git a/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl b/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl
index b33ca6e0c26..5cee6d4a6b5 100644
--- a/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl
+++ b/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl
@@ -55,7 +55,7 @@ if ($ENV{enable_injection_points} ne 'yes')
 # whether to turn things off during testing.
 sub cointoss
 {
-	return int(rand(2) == 1);
+	return int(rand() < 0.5);
 }
 
 # Helper for injecting random sleeps here and there in the testrun. The sleep
@@ -74,7 +74,7 @@ sub background_ro_pgbench
 	my ($port, $stdin, $stdout, $stderr) = @_;
 
 	my $pgbench_primary = IPC::Run::start(
-		[ 'pgbench', '-p', $port, '-S', '-T', '600', '-c', '10', 'postgres' ],
+		[ 'pgbench', '-n', '-p', $port, '-S', '-T', '600', '-c', '10', 'postgres' ],
 		'<' => \$stdin,
 		'>' => \$stdout,
 		'2>' => \$stderr,
@@ -224,6 +224,9 @@ background_rw_pgbench(
 	$node_primary->port, $pgb_primary_stdin,
 	$pgb_primary_stdout, $pgb_primary_stderr);
 
+my $primary_shutdown_clean = 0;
+my $standby_shutdown_clean = 0;
+
 # Main test suite. This loop will start a pgbench run on the cluster and while
 # that's running flip the state of data checksums concurrently. It will then
 # randomly restart thec cluster (in fast or immediate) mode and then check for
@@ -246,9 +249,11 @@ for (my $i = 0; $i < $TEST_ITERATIONS; $i++)
 		$node_primary_loglocation = -s $node_primary->logfile;
 
 		# If data checksums are enabled, take the opportunity to verify them
-		# while the cluster is offline
+		# while the cluster is offline (but only if stopped in a clean way,
+		# not after immediate shutdown)
 		$node_primary->checksum_verify_offline()
-		  unless $data_checksum_state eq 'off';
+		  unless $data_checksum_state eq 'off' or !$primary_shutdown_clean;
+
 		random_sleep();
 		$node_primary->start;
 		# Start a pgbench in the background against the primary
@@ -270,9 +275,11 @@ for (my $i = 0; $i < $TEST_ITERATIONS; $i++)
 		$node_standby_1_loglocation = -s $node_standby_1->logfile;
 
 		# If data checksums are enabled, take the opportunity to verify them
-		# while the cluster is offline
+		# while the cluster is offline (but only if stopped in a clean way,
+		# not after immediate shutdown)
 		$node_standby_1->checksum_verify_offline()
-		  unless $data_checksum_state eq 'off';
+		  unless $data_checksum_state eq 'off' or !$standby_shutdown_clean;
+
 		random_sleep();
 		$node_standby_1->start;
 		# Start a select-only pgbench in the background on the standby
@@ -287,13 +294,41 @@ for (my $i = 0; $i < $TEST_ITERATIONS; $i++)
 	my $result = $node_primary->safe_psql('postgres',
 		"SELECT count(*) FROM t WHERE a > 1");
 	is($result, '100000', 'ensure data pages can be read back on primary');
+
 	random_sleep();
+
 	$node_primary->wait_for_catchup($node_standby_1, 'write');
 
-	# Potentially powercycle the cluster
-	$node_primary->stop($stop_modes[ int(rand(100)) ]) if cointoss();
 	random_sleep();
-	$node_standby_1->stop($stop_modes[ int(rand(100)) ]) if cointoss();
+
+	# Potentially powercycle the cluster (the nodes independently)
+	# XXX should maybe try stopping nodes in the opposite order too?
+	if (cointoss())
+	{
+		my $mode = $stop_modes[ int(rand(100)) ];
+		$node_primary->stop($mode);
+		$primary_shutdown_clean = ($mode eq 'fast');
+	}
+
+	random_sleep();
+
+	if (cointoss())
+	{
+		my $mode = $stop_modes[ int(rand(100)) ];
+		$node_standby_1->stop($mode);
+		$standby_shutdown_clean = ($mode eq 'fast');
+	}
+}
+
+# make sure the nodes are running
+if (!$node_primary->is_alive)
+{
+	$node_primary->start;
+}
+
+if (!$node_standby_1->is_alive)
+{
+        $node_standby_1->start;
 }
 
 # Testrun is over, ensure that data reads back as expected and perform a final

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 29 Aug 2025 16:26:41 +0200

> It's not a definitive proof, but it's a hint c13070a27b63 was causing

Unfortunately, it seems I spoke too soon :-( I decided to test this on
multiple machines overnight, and it still fails on the slower ones.

Attached is a patch addressing a couple more issues, to makes the TAP
test work well enough. (Attached as .txt, to not confuse cfbot).

- The pgbench started by IPC::Run::start() needs to be finished, to
release resources. Otherwise it leaks file descriptors (and there's a
bunch of "defunct" pgbench processes), which may be a problem with
increased number of iterations.

- AFAICS the pgbench can't use stdin/stdout/stderr, otherwise the pipes
get broken when the command fails (after restart). I just used /dev/null
instead, the stdout/stderr was not used anyway (or was it?).

- commented out the pg_checksums call, because of the issues mentioned
earlier (I was trying to make it work by remembering the state, but it
seems to not make it into control file before shutdown occasionally)

I increased the number of iterations to 1000+ and ran it on three machines:

- ryzen (new machine from ~2024)
- xeon (old slow machine from ~2016)
- rpi5 (very slow machine)

I haven't seen a single failure on ryzen, after ~3000 iterations. But
both xeon and rpi5 show a number of failures. Xeon has about 35 reports
of 'Failed test', rpi5 and about 10.

My guess is it's something about timing. It works on the "fast" ryzen,
but fails on xeon which is ~3-4x slower. And rpi5, which is even slower.

The other reason why it seems unrelated to the reverted commit is that
it's not just about visibility maps (which was got corrupted). I see
checksum failures on VM and FSM. I think I forgot about the FSM cases,
and by the fact that I saw no failures on the ryzen post revert. But
clearly, other machines still have issues.

Another interesting fact is that the checksum failures happen both on
the primary and the standby, it's not just a standby issue. But again,
this sees to be machine-dependent. On the rpi5 I've only seen standby
issues. The xeon sees failures both on primary/standby (roughly 1:1).

There are more weird things. If I grep for page failures, I see this (a
more detailed log attached):

-----------
# 2025-08-28 22:33:28.195 CEST startup[177466] LOG:  page verification
failed, calculated checksum 25350 but expected 44559
# 2025-08-28 22:33:28.197 CEST startup[177466] LOG:  page verification
failed, calculated checksum 25350 but expected 44559
# 2025-08-28 22:33:28.199 CEST startup[177466] LOG:  page verification
failed, calculated checksum 59909 but expected 53920
# 2025-08-28 22:33:28.201 CEST startup[177466] LOG:  page verification
failed, calculated checksum 59909 but expected 53920
# 2025-08-28 22:33:28.206 CEST startup[177466] LOG:  page verification
failed, calculated checksum 59909 but expected 53920
# 2025-08-28 22:33:28.207 CEST startup[177466] LOG:  page verification
failed, calculated checksum 25350 but expected 44559
-----------

This is right after a single restart, while doing the recovery. The
weird thing is, this is all for just two FSM pages!

-----------
LOG:  invalid page in block 2 of relation "base/5/16410_fsm"; zeroing
out page
LOG:  invalid page in block 2 of relation "base/5/16408_fsm"; zeroing
out page
-----------

And the calculated/expected checksums repeat! It's just different WAL
records hitting the same page, and complaining about the same issue,
after claiming the page was zeroed out. Isn't that weird? How come the
page doesn't "get" the correct checksum after the first redo?

I've seen these failures after changing checksums in both directions,
both after enabling and disabling. But I've only ever saw this after
immediate shutdown, never after fast shutdown. (It's interesting the
pg_checksums failed only after fast shutdowns ...).

Could it be that the redo happens to start from an older position, but
using the new checksum version?

regards

-- 
Tomas Vondra
From 57bb79b1bc8faac646131336abcc1596711c5f32 Mon Sep 17 00:00:00 2001
From: tomas <tomas>
Date: Thu, 28 Aug 2025 22:10:25 +0200
Subject: [PATCH] TAP fixes

---
 .../t/006_concurrent_pgbench.pl               | 88 ++++++++++++++-----
 1 file changed, 68 insertions(+), 20 deletions(-)

diff --git a/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl b/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl
index b33ca6e0c26..374eac7e6a3 100644
--- a/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl
+++ b/src/test/modules/test_checksums/t/006_concurrent_pgbench.pl
@@ -23,11 +23,14 @@ my $node_primary_loglocation = 0;
 my $node_standby_1;
 my $node_standby_1_loglocation = 0;
 
+my $pgbench_primary = undef;
+my $pgbench_standby = undef;
+
 # The number of full test iterations which will be performed. The exact number
 # of tests performed and the wall time taken is non-deterministic as the test
 # performs a lot of randomized actions, but 50 iterations will be a long test
 # run regardless.
-my $TEST_ITERATIONS = 50;
+my $TEST_ITERATIONS = 1000;
 
 # Variables which record the current state of the cluster
 my $data_checksum_state = 'off';
@@ -55,7 +58,7 @@ if ($ENV{enable_injection_points} ne 'yes')
 # whether to turn things off during testing.
 sub cointoss
 {
-	return int(rand(2) == 1);
+	return int(rand() < 0.5);
 }
 
 # Helper for injecting random sleeps here and there in the testrun. The sleep
@@ -73,11 +76,16 @@ sub background_ro_pgbench
 {
 	my ($port, $stdin, $stdout, $stderr) = @_;
 
-	my $pgbench_primary = IPC::Run::start(
-		[ 'pgbench', '-p', $port, '-S', '-T', '600', '-c', '10', 'postgres' ],
-		'<' => \$stdin,
-		'>' => \$stdout,
-		'2>' => \$stderr,
+	if ($pgbench_standby)
+	{
+		$pgbench_standby->finish;
+	}
+
+	$pgbench_standby = IPC::Run::start(
+		[ 'pgbench', '-n', '-p', $port, '-S', '-T', '600', '-c', '10', 'postgres' ],
+		'<' => '/dev/null',
+		'>' => '/dev/null',
+		'2>' => '/dev/null',
 		IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
 }
 
@@ -87,11 +95,16 @@ sub background_rw_pgbench
 {
 	my ($port, $stdin, $stdout, $stderr) = @_;
 
-	my $pgbench_primary = IPC::Run::start(
+	if ($pgbench_primary)
+	{
+		$pgbench_primary->finish;
+	}
+
+	$pgbench_primary = IPC::Run::start(
 		[ 'pgbench', '-p', $port, '-T', '600', '-c', '10', 'postgres' ],
-		'<' => \$stdin,
-		'>' => \$stdout,
-		'2>' => \$stderr,
+		'<' => '/dev/null',
+		'>' => '/dev/null',
+		'2>' => '/dev/null',
 		IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
 }
 
@@ -224,6 +237,9 @@ background_rw_pgbench(
 	$node_primary->port, $pgb_primary_stdin,
 	$pgb_primary_stdout, $pgb_primary_stderr);
 
@@ -246,9 +262,11 @@ for (my $i = 0; $i < $TEST_ITERATIONS; $i++)
 
 		# If data checksums are enabled, take the opportunity to verify them
-		# while the cluster is offline
-		$node_primary->checksum_verify_offline()
-		  unless $data_checksum_state eq 'off';
+		# while the cluster is offline (but only if stopped in a clean way,
+		# not after immediate shutdown)
+		#$node_primary->checksum_verify_offline()
+		#  unless $data_checksum_state eq 'off' or !$primary_shutdown_clean;
+
 		random_sleep();
 		$node_primary->start;
 		# Start a pgbench in the background against the primary
@@ -270,9 +288,11 @@ for (my $i = 0; $i < $TEST_ITERATIONS; $i++)
 		$node_standby_1_loglocation = -s $node_standby_1->logfile;
 
 		# If data checksums are enabled, take the opportunity to verify them
-		# while the cluster is offline
-		$node_standby_1->checksum_verify_offline()
-		  unless $data_checksum_state eq 'off';
+		# while the cluster is offline (but only if stopped in a clean way,
+		# not after immediate shutdown)
+		#$node_standby_1->checksum_verify_offline()
+		#  unless $data_checksum_state eq 'off' or !$standby_shutdown_clean;
+
 		random_sleep();
 		$node_standby_1->start;
 		# Start a select-only pgbench in the background on the standby
@@ -287,13 +307,41 @@ for (my $i = 0; $i < $TEST_ITERATIONS; $i++)
 	my $result = $node_primary->safe_psql('postgres',
 		"SELECT count(*) FROM t WHERE a > 1");
 	is($result, '100000', 'ensure data pages can be read back on primary');
+
 	random_sleep();
+
 	$node_primary->wait_for_catchup($node_standby_1, 'write');

2.39.5

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 29 Aug 2025 16:38:22 +0200

> pg_checksums failed only after fast shutdowns ...).

Of course, right after I send a message, it fails after a fast shutdown,
contradicting my observation ...

> using the new checksum version?

... but it also provided more data supporting this hypothesis. I added
logging of pg_current_wal_lsn() before / after changing checksums on the
primary, and I see this:

1) LSN before: 14/2B0F26A8
2) enable checksums
3) LSN after: 14/EE335D60
4) standby waits for 14/F4E786E8 (higher, likely thanks to pgbench)
5) standby restarts with -m fast
6) redo starts at 14/230043B0, which is *before* enabling checksums

I guess this is the root cause. A bit more detailed log attached.

regards

-- 
Tomas Vondra

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 1 Sep 2025 14:11:06 +0200

> I guess this is the root cause. A bit more detailed log attached.

I kept stress testing this over the weekend, and I think I found two
issues causing the checksum failures, both for a single node and on a
standby:

1) no checkpoint in the "disable path"

In the "enable" path, a checkpoint it enforced before flipping the state
from "inprogress-on" to "on". It's hidden in the ProcessAllDatabases,
but it's there. But the "off" path does not do that, probably on the
assumption that we'll always see the writes in the WAL order, so that
we'll see the XLOG_CHECKSUMS setting checksums=off before seeing any
writes without checksums.

And in the happy path this works fine - the standby is happy, etc. But
what about after a crash / immediate shutdown? Consider a sequence like
this:

a) we have checksums=on
b) write to page P, updating the checksum
c) start disabling checksums
d) progress to inprogress-off
e) progress to off
f) write to page P, without checksum update
g) the write to P gets evicted (small shared buffers, ...)
h) crash / immediate shutdown

Recovery starts from a LSN before (a), so we believe checksums=on. We
try to redo the write to P, which starts by reading the page from disk,
to check the page LSN. We still think checksums=on, and to read the LSN
we need to verify the checksum. But the page was modified without the
checksum, and evicted. Kabooom!

This is not that hard to trigger by hand. Add a long at the end of
SetDataChecksumsOff, start a pgbench on a scale larger than shared
buffers and call pg_disable_data_checksums(). Once it gets stuck on the
sleep, give it more time to dirty and evict some pages, then kill -9. On
recovery you should get the same checksum failures.

FWIW I've only ever seen failures on fsm/vm forks, which matches what I
see in the TAP tests. But isn't it a bit strange?

I think the "disable" path needs a checkpoint between inprogress-off and
off states, same as the "enable" path.

2) no restart point on the standby

The standby has a similar issue, I think. Even if the primary creates
all the necessary checkpoints, the standby may not need to create the
restart point for them. If you look into xlog_redo, it only "remembers"
the checkpoint position, it does not trigger a restart point. Than only
happens in XLogPageRead, based on distance from the previous one.

So a very similar failure to the primary is possible, even with the
extra checkpoint fixing (1). The primary flips checksums in either
direction, generating checkpoints, but the standby does not create the
restart points. But it applies WAL, and some of the pages without
checksums get evicted.

And then the standby fails, and goes to some redo position far back, and
runs into the same checksum failure when trying to check page LSN.

I think the standby needs some logic to force restart point creation
when the checksum flag changed.

I have an experimental WIP branch at:

https://github.com/tvondra/postgres/tree/online-checksums-tap-tweaks

It fixes the TAP issues reported earlier (and a couple more), and it
does a bunch of additional tweaks:

a) A lot of debug messages that helped me to figure this out. This is
probably way too much, especially the controlfile updates can be very
noisy on a standby.

b) Adds a simpler TAP, testing just a single node (should be easier to
understand than with failures on standby).

c) Adds an explicit checkpoints, to fix (1). It probably adds too many
checkpoints, though? AFAICS a checkpoint after the "inprogress" phase
should be enough, a checkpoint after the "on/off" can go away.

d) Forces creating a restart point on the first checkpoint after
XLOG_CHECKSUMS record. It's done in a bit silly way, using a static
flag. Maybe there's a more elegant approach, say by comparing the
checksum value in ControlFile to the received checkpoint?

e) Randomizes a couple more GUC values. This needs more thought, it was
done blindly before better understanding how the failures happen (it
requires buffers evicted, not hitting max_wal_size, ...). There are more
params worth randomizing (e.g. the "fast" flag).

Anyway, with (c) and (d) applied, the checksum failures go away. It may
not be 100% right (e.g. we could do away with fewer checkpoints), but it
seems to be the right direction.

I don't have time to cleanup the branch more, I've already spent too
much time looking at LSNs advancing in weird ways :-( Hopefully it's
good enough to show what needs to be fixed, etc. If there's a new
version, I'm happy to rerun the tests on my machines, ofc.

However, there still are more bugs. Attached is a log from a crash after
hitting the assert into AbsorbChecksumsOffBarrier:

 Assert((LocalDataChecksumVersion != PG_DATA_CHECKSUM_VERSION) &&
 (LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION ||
  LocalDataChecksumVersion == PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION));

This happened while flipping checksums to 'off, but the backend already
thinks checksum are 'off':

    LocalDataChecksumVersion==0

I think this implies some bug in setting up LocalDataChecksumVersion
after connection, because this is for a query checking the checksum
state, executed by the TAP test (in a new connection, right?).

I haven't looked into this more, but how come the "off" direction does
not need to check InitialDataChecksumTransition?

I think the TAP test turned out to be very useful, so far. While
investigating on this, I thought about a couple more tweaks to make it
detect additional issues (on top of the randomization).

- Right now the shutdowns/restarts happen only in very limited places.
The checksum flips from on/off or off/on, and then a restart happens.
AFAICS it never happens in the "inprogress" phases, right?

- The pgbench clients connect once, so there are almost no new
connections while flipping checksums. Maybe some of the pgbenches should
run with "-C", to open new connections. It was pretty lucky the TAP
query hit the assert, this would make it more likely.

regards

-- 
Tomas Vondra

---

From: Daniel Gustafsson <[email-masked]>
Date: Mon, 6 Oct 2025 14:53:01 +0200

> On 1 Sep 2025, at 14:11, Tomas Vondra <[email-masked]> wrote:

> issues causing the checksum failures, both for a single node and on a

Thanks a lot for all your testing. I was able to reproduce ..

> https://github.com/tvondra/postgres/tree/online-checksums-tap-tweaks

.. and with your changes I also see the reproducer go away.  I concur that
you likely found the issue and the right fix for it.  I have absorbed your
patches into my branch, the debug logging is left as 0002 but the other ones
are incorporated into 0001.

> probably way too much, especially the controlfile updates can be very

I've toned down the logging a bit, but kept most of it in 0002.

> understand than with failures on standby).

This turned out to be more useful than I initially thought, so I've kept this
in the attached version.  There could be value in separating the single and
dual node tests into different PG_TEST_EXTRA values given how intensive the
latter is.

> seems to be the right direction.

I think so too, and while I have removed one of them due to being issued just
before (or after) another checkpoint I do believe this is the right fix for the
issue.  There might well be more issues, but I wanted to get a new version out
on the thread to get more visibility on the the new tests.

> not need to check InitialDataChecksumTransition?

This boiled down into the barrier absorbing functions evolving out of sync with
one another over multiple versions of the patch.  To address this absorbing the
barrier has been converted into a single function which is driven by an array
of ChecksumBarrierCondition structs, one for each target state.  This struct
defines what the current state of the cluster must be for the barrier to be
successfully absorbed.  This removed a lot of duplicate code and also unifies
the previously quite varied levels of assertions at the barrier.

> AFAICS it never happens in the "inprogress" phases, right?

That would be a good idea, as well as use a crashing injection test on top
controlled shutdowns.

> query hit the assert, this would make it more likely.

I've added this to both tests using pgbench, randomized with a cointoss call.

The attached has the above as well as a few other changes:

* A new injection test which calls abort() right before checkpointing has been
added in 005_injection.

* Most ereport calls have gotten proper errcodes in order to aid analysis,
particurly fleet-wide analysis should this be deployed in a larger setting.

* More code-level documentation of test code and several tweaked (and added)
code comments to aid readability.

--
Daniel Gustafsson

---

From: Daniel Gustafsson <[email-masked]>
Date: Wed, 5 Nov 2025 09:56:13 +0100

Rebase due to recent conflicts with only a trivial whitespace fix, otherwise
the same as the previous version.

--
Daniel Gustafsson

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 5 Nov 2025 15:05:48 +0100

Hi Daniel,

I took a look at the patch again, focusing mostly on the docs and
comments. I think the code is OK, I haven't noticed anything serious.

testing
-------

I'm running the TAP tests - so far it looks fine, I've done 2000
iterations of the "single" test, now running ~2000 iterations of the
"standby" test. No issues/failures so far.

The question is whether we can/should make it even more "random", by
doing restarts in more places etc. I might give it a try, if I happen to
have some free time. But no promises, I'm not sure how feasible it is.
Making it "truly random" means it's hard to deduce what should be the
actual state of checksums, etc.

review
------

Attached is a patch adding FIXME/XXX comments to a bunch of places,
which I think makes it clearer which place I'm talking about. I'll
briefly go over the items, and maybe explain them a bit more.

1) func-admin.sgml

- This is missing documentation for the "fast" parameters for both
functions (enable/disable).

- The paragraph stars with "Initiates data checksums for ..." but that
doesn't sound right to me. I think you can initiate enabling/disabling,
not "data checksums".

- I think the docs should at least briefly describe the impact on the
cluster, and also on a standby, due to having to write everything into
WAL, waiting for checkpoints etc. And maybe discuss how to mitigate that
in some way. More about the standby stuff later.

2) glossary.sgml

- This describes "checksum worker" as process that enables or disables
checksums in a specific database, but we don't need any per-database
processes when disabling, no?

3) monitoring.sgml

- I'm not sure what "regardles" implies here. Does that mean we simply
don't hide/reset the counters?

- I added a brief explanation about using the "launcher" row for overall
progress, and per-database workers for "current progress".

- Do we want to refer to "datachecksumsworker"? Isn't it a bit too
low-level detail?

- The table of phases is missing "waiting" and "done" phases. IMHO if
the progress view can return it, it should be in the docs.

4) wal.sgml

- I added a sentence explaining that both enabling and disabling happens
in phases, with checkpoints in between. I think that's helpful for users
and DBAs.

- The section only described "enabling checksums", but it probably
should explain the process for disabling too. Added a para.

- Again, I think we should explain the checkpoints, restartpoints,
impact on standby ... somewhere. Not sure if this is the right place.

5) xlog.c

- Some minor corrections (typos, ...).

- Isn't the claim that PG_DATA_CHECKSUM_ON_VERSION is the only place
where we check InitialDataChecksumTransition stale? AFAIK we now check
this in AbsorbDataChecksumsBarrier for all transitions, no?

6) datachecksumsworker.c

- I understand what the comments at the beginning of the file say, but I
suspect it's partially due to already knowing the code. There's a couple
places that might be more explicit, I think. For example:

- One of the items in the synchronization/correctness section states
that "Backends SHALL NOT violate local data_checksums state" but what
does "violating local data_checksums state" mean? What even is "local
state in this context"? Should we explain/define that, or would that be
unnecessarily detailed?

- The section only talks about "enabling checksums" but also discusses
all four possible states. Maybe it should talk about disabling too, as
that requires the same synchronization/correctness.

- Maybe it'd be good make it more explicit at which point the process
waits on a barrier, which backend initiates that (and which backends are
required to absorb the signal). It kinda is there, but only indirectly.

- Another idea I had is that maybe it'd help to have some visualization
of the process (with data_checksum states, barriers, ...) e.g. in the
form of an ASCII image.

open questions
--------------

For me the main remaining question is impact people should expect in
production systems, and maybe ways to mitigate that.

In single-node systems this is entirely fine, I think. There will be
checkpoints, but those will be "spread" and it's just the checksum
worker waiting for that to complete.

It'll write everything into WAL, but that's fairly well understood /
should be expected. We should probably mention that in the sgml docs, so
that people are not surprised their WAL archive gets huge.

I'm much more concerned about streaming replicas, because there it
forces a restart point - and it *blocks redo* until it completes. Which
means there'll be replication lag, and for synchronous standbys this
would even block progress on the primary.

We should very clearly document this. But I'm also wondering if we might
mitigate this in some way / reduce the impact.

I see some similarities to shutdown checkpoints, which can take a lot of
time if there happen to be a lot of dirty data, increasing disruption
during planned restarts (when no one can connect). A common mitigation
is to run CHECKPOINT shortly before the restart, to flush most of the
dirty data while still allowing new connections.

Maybe we could do something like that for checksum changes? I don't know
how exactly we could do that, but let's say we can predict when roughly
to expect the next state change. And we'd ensure the standby starts
flushing stuff before that, so that creating the restartpoint is cheap.
Or maybe we'd (gradually?) lower max_wal_size on the standby, to reduce
the amount of WAL as we're getting closer to the end?

regards

-- 
Tomas Vondra
From 40d26a0406a213207101a837999e8b57df25fc1d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email-masked]>
Date: Wed, 5 Nov 2025 13:26:41 +0100
Subject: [PATCH v20251105 2/3] review

---
 doc/src/sgml/func/func-admin.sgml            | 20 ++++++++++++-----
 doc/src/sgml/glossary.sgml                   |  2 +-
 doc/src/sgml/monitoring.sgml                 | 23 ++++++++++++++++++--
 doc/src/sgml/wal.sgml                        | 18 +++++++++++++++
 src/backend/access/transam/xlog.c            | 17 +++++++++------
 src/backend/postmaster/datachecksumsworker.c | 14 ++++++++++++
 src/backend/storage/ipc/ipci.c               |  1 -
 src/include/postmaster/datachecksumsworker.h |  2 +-
 src/tools/pgindent/typedefs.list             |  2 ++
 9 files changed, 82 insertions(+), 17 deletions(-)

diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index f3a8782ede0..4d6f6a7e486 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -3008,11 +3008,11 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
         <indexterm>
          <primary>pg_enable_data_checksums</primary>
         </indexterm>
-        <function>pg_enable_data_checksums</function> ( <optional><parameter>cost_delay</parameter> <type>int</type>, <parameter>cost_limit</parameter> <type>int</type></optional> )
+        <function>pg_enable_data_checksums</function> ( <optional><parameter>cost_delay</parameter> <type>int</type>, <parameter>cost_limit</parameter> <type>int</type></optional>, <parameter>fast</parameter> <type>bool</type></optional> )
         <returnvalue>void</returnvalue>
        </para>
        <para>
-        Initiates data checksums for the cluster. This will switch the data
+        Initiates the process of enabling data checksums for the cluster. This will switch the data
         checksums mode to <literal>inprogress-on</literal> as well as start a
         background worker that will process all pages in the database and
         enable checksums on them. When all data pages have had checksums
@@ -3023,7 +3023,9 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
         If <parameter>cost_delay</parameter> and <parameter>cost_limit</parameter> are
         specified, the speed of the process is throttled using the same principles as
         <link linkend="runtime-config-resource-vacuum-cost">Cost-based Vacuum Delay</link>.
-       </para></entry>
+       </para>
+       <para>FIXME missing documentation of the "fast" parameter</para>
+       </entry>
       </row>
 
       <row>
@@ -3031,7 +3033,7 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
         <indexterm>
          <primary>pg_disable_data_checksums</primary>
         </indexterm>
-        <function>pg_disable_data_checksums</function> ()
+        <function>pg_disable_data_checksums</function> ( <parameter>fast</parameter> <type>bool</type></optional> )
         <returnvalue>void</returnvalue>
        </para>
        <para>
@@ -3042,12 +3044,20 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
         changed to <literal>off</literal>.  At this point the data pages will
         still have checksums recorded but they are not updated when pages are
         modified.
-       </para></entry>
+       </para>
+       <para>FIXME missing documentation of the "fast" parameter</para>
+       </entry>
       </row>
      </tbody>
     </tgroup>
    </table>
 
+   <para>
+    FIXME I think this should briefly explain how this interacts with checkpoints (through
+    the fast parameters). It should probably also discuss how this affects streaming standby
+    due to forcing a restart point, etc. And maybe comment on possible mitigations?
+   </para>
+
   </sect2>
 
   </sect1>
diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml
index 9bac0c96348..3ba0e8c6c5c 100644
--- a/doc/src/sgml/glossary.sgml
+++ b/doc/src/sgml/glossary.sgml
@@ -580,7 +580,7 @@
    <glossdef>
     <para>
      An <glossterm linkend="glossary-auxiliary-proc">auxiliary process</glossterm>
-     which enables or disables data checksums in a specific database.
+     which enables data checksums in a specific database.
     </para>
    </glossdef>
   </glossentry>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b56e220f3d8..7efa1af746a 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3554,7 +3554,10 @@ description | Waiting for a newly initialized WAL file to reach durable storage
        database (or on a shared object).
        Detected failures are reported regardless of the
        <xref linkend="guc-data-checksums"/> setting.
-      </para></entry>
+      </para>
+      <para>XXX I'm not sure what "regardless" means in this context. I guess
+      it means we don't reset the counters when disabling checksums?</para>
+     </entry>
      </row>
 
      <row>
@@ -6959,6 +6962,9 @@ FROM pg_stat_get_backend_idset() AS backendid;
    <structname>pg_stat_progress_data_checksums</structname> view will contain
    a row for the launcher process, and one row for each worker process which
    is currently calculating checksums for the data pages in one database.
+   The launcher provides overview of the overall progress (how many database
+   have been processed, how many remain), while the workers track progress for
+   currently processed databases.
   </para>
 
   <table id="pg-stat-progress-data-checksums-view" xreflabel="pg_stat_progress_data_checksums">
@@ -6984,7 +6990,8 @@ FROM pg_stat_get_backend_idset() AS backendid;
         <structfield>pid</structfield> <type>integer</type>
        </para>
        <para>
-        Process ID of a datachecksumworker process.
+        Process ID of a datachecksumsworker process.
+        FIXME Is datachecksumsworker defined somewhere? Does it refer to the launcher too?
        </para>
       </entry>
      </row>
@@ -7127,6 +7134,12 @@ FROM pg_stat_get_backend_idset() AS backendid;
        The command is currently disabling data checksums on the cluster.
       </entry>
      </row>
+     <row>
+      <entry><literal>waiting</literal></entry>
+      <entry>
+       FIXME
+      </entry>
+     </row>
      <row>
       <entry><literal>waiting on temporary tables</literal></entry>
       <entry>
@@ -7141,6 +7154,12 @@ FROM pg_stat_get_backend_idset() AS backendid;
        state before finishing.
       </entry>
      </row>
+     <row>
+      <entry><literal>done</literal></entry>
+      <entry>
+       FIXME
+      </entry>
+     </row>
     </tbody>
    </tgroup>
   </table>
diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgml
index 0ada90ca0b1..8ef16d7769f 100644
--- a/doc/src/sgml/wal.sgml
+++ b/doc/src/sgml/wal.sgml
@@ -284,6 +284,11 @@
     <link linkend="functions-admin-checksum">functions</link>.
    </para>
 
+   <para>
+    Both enabling and disabling checksums happens in two phases, separated by
+    a checkpoint to ensure durability.
+   </para>
+
    <para>
     Enabling checksums will put the cluster checksum mode in
     <literal>inprogress-on</literal> mode.  During this time, checksums will be
@@ -314,6 +319,14 @@
     no support for resuming work from where it was interrupted.
    </para>
 
+   <para>
+    Disabling checksums will put the cluster checksum mode in
+    <literal>inprogress-off</literal> mode.  During this time, checksums will be
+    written but not verified. After all processes acknowledge the change,
+    the mode will automatically switch to <literal>off</literal>. In this case
+    rewriting all blocks is not needed, but checkpoints are still required.
+   </para>
+
    <note>
     <para>
      Enabling checksums can cause significant I/O to the system, as most of the
@@ -324,6 +337,11 @@
     </para>
    </note>
 
+   <para>
+    XXX Maybe this is the place that should mention checkpoints/restartpoints,
+    how it impacts systems/replication and how to mitigate that?
+   </para>
+
   </sect2>
  </sect1>
 
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f7633f47551..807b82eed4f 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -607,7 +607,7 @@ typedef struct ChecksumBarrierCondition
 	int			barrier_ne[MAX_BARRIER_CONDITIONS];
 	/* The number of elements in the barrier_ne set */
 	int			barrier_ne_sz;
-}			ChecksumBarrierCondition;
+} ChecksumBarrierCondition;
 
 static const ChecksumBarrierCondition checksum_barriers[] =
 {
@@ -618,7 +618,6 @@ static const ChecksumBarrierCondition checksum_barriers[] =
 	{-1}
 };
 
-
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -694,10 +693,10 @@ static TimeLineID LocalMinRecoveryPointTLI;
 static bool updateMinRecoveryPoint = true;
 
 /*
- * Local state fror Controlfile data_checksum_version.  After initialization
+ * Local state for Controlfile data_checksum_version.  After initialization
  * this is only updated when absorbing a procsignal barrier during interrupt
  * processing.  The reason for keeping a copy in backend-private memory is to
- * avoid locking for interrogating checksum state.  Possible values are the
+ * avoid locking for interrogating the checksum state.  Possible values are the
  * checksum versions defined in storage/bufpage.h as well as zero when data
  * checksums are disabled.
  */
@@ -712,6 +711,10 @@ static uint32 LocalDataChecksumVersion = 0;
  * processing the barrier.  This may happen if the process is spawned between
  * the update of XLogCtl->data_checksum_version and the barrier being emitted.
  * This can only happen on the very first barrier so mark that with this flag.
+ *
+ * XXX Is PG_DATA_CHECKSUM_ON_VERSION still the only transition with an assert?
+ * I think it was replaced by checking checksum_barriers for every transition,
+ * with elog(ERROR), no?
  */
 static bool InitialDataChecksumTransition = true;
 
@@ -4335,7 +4338,7 @@ InitControlFile(uint64 sysidentifier, uint32 data_checksum_version)
 
 	/*
 	 * Set the data_checksum_version value into XLogCtl, which is where all
-	 * processes get the current value from. (Maybe it should go just there?)
+	 * processes get the current value from.
 	 */
 	XLogCtl->data_checksum_version = data_checksum_version;
 }
@@ -4921,7 +4924,7 @@ SetDataChecksumsOff(bool immediate_checkpoint)
 	uint64		barrier;
 	int			flags;
 
-	Assert(ControlFile);

@@ -5030,7 +5033,7 @@ SetDataChecksumsOff(bool immediate_checkpoint)
 bool
 AbsorbDataChecksumsBarrier(int target_state)
 {
-	const		ChecksumBarrierCondition *condition = checksum_barriers;
+	const ChecksumBarrierCondition *condition = checksum_barriers;
 	int			current = LocalDataChecksumVersion;
 	bool		found = false;
 
diff --git a/src/backend/postmaster/datachecksumsworker.c b/src/backend/postmaster/datachecksumsworker.c
index 3deb57a96de..67045b9014d 100644
--- a/src/backend/postmaster/datachecksumsworker.c
+++ b/src/backend/postmaster/datachecksumsworker.c
@@ -52,6 +52,9 @@
  *    - Data checksums SHALL NOT be considered enabled cluster-wide until all
  *      currently connected backends have the local state "enabled"
  *
+ * FIXME I'm not 100% sure I understand what the two above points say. What does
+ * "violate local data_checksums state" means"?
+ *
  * There are two levels of synchronization required for enabling data checksums
  * in an online cluster: (i) changing state in the active backends ("on",
  * "off", "inprogress-on" and "inprogress-off"), and (ii) ensuring no
@@ -60,6 +63,10 @@
  * latter with ensuring that any concurrent activity cannot break the data
  * checksum contract during processing.
  *
+ * FIXME This para talks about "enabling" but then mentions all four states,
+ * including "inprogress-off" and "off". Maybe it should talk about "changing
+ * data_checksums" instead?
+ *
  * Synchronizing the state change is done with procsignal barriers, where the
  * WAL logging backend updating the global state in the controlfile will wait
  * for all other backends to absorb the barrier. Barrier absorption will happen
@@ -88,6 +95,10 @@
  *   enables data checksums cluster-wide, there are four sets of backends where
  *   Bd shall be an empty set:
  *
+ * FIXME Maybe mention which process initiates the procsignalbarrier?
+ * FIXME Don't we actually wait or the barrier before we start rewriting data?
+ * I think Bd has to be empty prior to that, otherwise it might break checksums.
+ *
  *   Bg: Backend updating the global state and emitting the procsignalbarrier
  *   Bd: Backends in "off" state
  *   Be: Backends in "on" state
@@ -124,6 +135,8 @@
  *   stop writing data checksums as no backend is enforcing data checksum
  *   validation any longer.
  *
+ * XXX Maybe it'd make sense to "visualize" the progress between the states
+ * and barriers in some way. Say, by doing
  *
  * Potential optimizations
  * -----------------------
@@ -148,6 +161,7 @@
  *     to enable checksums on a cluster which is in inprogress-on state and
  *     may have checksummed pages (make pg_checksums be able to resume an
  *     online operation).
+ *   * Restartability (not necessarily with page granularity).
  *
  *
  * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 44213d140ae..9014e90f1c7 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -31,7 +31,6 @@
 #include "postmaster/bgworker_internals.h"
 #include "postmaster/bgwriter.h"
 #include "postmaster/datachecksumsworker.h"
-#include "postmaster/postmaster.h"
 #include "postmaster/walsummarizer.h"
 #include "replication/logicallauncher.h"
 #include "replication/origin.h"
diff --git a/src/include/postmaster/datachecksumsworker.h b/src/include/postmaster/datachecksumsworker.h
index 2cd066fd0fe..0daef709ec8 100644
--- a/src/include/postmaster/datachecksumsworker.h
+++ b/src/include/postmaster/datachecksumsworker.h
@@ -24,7 +24,7 @@ typedef enum DataChecksumsWorkerOperation
 	ENABLE_DATACHECKSUMS,
 	DISABLE_DATACHECKSUMS,
 	/* TODO: VERIFY_DATACHECKSUMS, */
-}			DataChecksumsWorkerOperation;
+} DataChecksumsWorkerOperation;
 
 /*
  * Possible states for a database entry which has been processed. Exported
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9d6b4e57cf3..3049e6018b3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -417,6 +417,7 @@ CheckPointStmt
 CheckpointStatsData
 CheckpointerRequest
 CheckpointerShmemStruct
+ChecksumBarrierCondition
 ChecksumType
 Chromosome
 CkptSortItem
@@ -587,6 +588,7 @@ CustomScan
 CustomScanMethods
 CustomScanState
 CycleCtr
+DataChecksumsWorkerOperation
 DBState
 DCHCacheEntry
 DEADLOCK_INFO
-- 
2.51.0

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 10 Nov 2025 02:26:07 +0100

Hi,

We had some off-list discussions about this patch (me, Daniel and
Andres), and Andres mentioned he suspects the patch may be breaking PITR
in some way. We didn't have any example of that, but PITR seems like a
pretty fundamental feature, so I took it seriously and decided to do
some stress testing. And yeah, there are issues :-(

I did a similar stress testing in the past, which eventually evolved
into the two TAP tests in the current patch. Those TAP tests run either
a single node or primary-standby cluster, and flip checksums on/off
while restarting the instance(s). And verify the cluster behaves OK,
with no checksum failures, unexpected states, etc.

I chose to do something similar to test PITR - run a single node with
pgbench (or some other write activity), flip checksums on/off in a loop,
while taking basebackups. And then validate the basebackups are valid,
and can be used to start a new instance. I planned to extend this to
more elaborate tests with proper PITR using a WAL archive, etc. I didn't
get that far - this simple test already hit a couple issues.

I'm attaching sets of test scripts, and the scripts used to validate
backups. The scripts are fairly simple, but need some changes to run -
adjusting some paths, etc.

1) basebackup-short.tgz - basic basebackup test

2) basebackup-long.tgz - long-runnning basebackup test (throttled)

3) validate.tgz - validate backups from (1) and (2)

The test scripts expect "scale" parameter for pgbench. I used 500, but
that's mostly arbitrary - maybe a different value would hit the issues
more often. Not sure.

While testing I ran into two issues while validating the backups (which
is essentially about performing the usual basebackup redo).

1) assert in a checkpointer

TRAP: failed Assert("TransactionIdIsValid(initial)"), File:
"procarray.c", Line: 1707, PID: 2649754
postgres: checkpointer (ExceptionalCondition+0x56) [0x55d64ec38f96]
postgres: checkpointer (+0x5341d8) [0x55d64eac41d8]
postgres: checkpointer (GetOldestTransactionIdConsideredRunning+0xc)
[0x55d64eac598c]
postgres: checkpointer (CreateRestartPoint+0x725) [0x55d64e7dd2f5]
postgres: checkpointer (CheckpointerMain+0x3ec) [0x55d64ea38a8c]
postgres: checkpointer (postmaster_child_launch+0x102) [0x55d64ea3b592]
postgres: checkpointer (+0x4ad74a) [0x55d64ea3d74a]
postgres: checkpointer (PostmasterMain+0xce7) [0x55d64ea40b97]
postgres: checkpointer (main+0x1ca) [0x55d64e713b1a]
/lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7f55f1a33ca8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f55f1a33d65]
postgres: checkpointer (_start+0x21) [0x55d64e713e81]

I was really confused by this initially, because why would this break
tracking of running transactions in a snapshot, etc.? But that's really
a red herring - the real issue is calling CreateRestartPoint().

This happens because we need to ensure that a standby does not get
confused about checksums state, so redo of XLOG_CHECKSUMS forces
creating a restart point whenever the checksum state changes. Which has
some negative consequences, as discussed in my previous message.

But AFAICS in this case the root cause is calling this during a regular
redo, not just on a standby. Presumably there's a way to distinguish
these two cases, and skip the restart point creation on a simple redo.
Alternatively, maybe the standby should do something different instead
of creating a restart point. (More about that later.)

2) unexpected state during redo of XLOG_CHECKSUMS record

An example of the failure:

    LOG:  redo starts at 36/5B028668
    FATAL:  incorrect data checksum state 3 for target state 1
    CONTEXT:  WAL redo at 37/E136E408 for XLOG/CHECKSUMS: on
    ERROR:  incorrect data checksum state 3 for target state 1
    ERROR:  incorrect data checksum state 3 for target state 1
    ERROR:  incorrect data checksum state 3 for target state 1
    ERROR:  incorrect data checksum state 3 for target state 1
    ERROR:  incorrect data checksum state 3 for target state 1
    LOG:  startup process (PID 2649028) exited with exit code 1

I was really confused about this at first, because I haven't seen this
during earlier testing (which did redo after a crash), so why should it
happen here?

But the reason is very simple - the basebackup can run for a while,
spanning multiple checkpoints, and also multiple changes of checksum
state (enable/disable). Furthermore, basebackup copies the pg_control
file last, and that's where we get the checksum state from.

For the earlier crash/restart test that's not the case, because each
checksum change creates a checkpoint, and the redo starts from that
point. There can't be multiple XLOG_CHECKSUMS records to apply.

So after a crash we start from the last checkpoint (i.e. from the
initial checksum state), and then there can be only a single
XLOG_CHECKSUMS record to reapply.

But with a basebackup we get the checksum state from the pg_control
copied at the end (so likely the final state, not the initial one), and
there may be multiple XLOG_CHECKSUMS in between (and multiple
checkpoints). So what happens is that we start from the final state, and
then try to apply the earlier XLOG_CHECKSUMS states. Hence the failure.

This seems like a pretty fundamental issue - we can't guarantee a backup
won't run too long, or anything like that. A full backup may easily run
for multiple hours or even days.

I suspect it should be possible to hit an issue similar to those we
observed on the standby in earlier testing (which is why we ended up
creating startpoints on a standby) with "future" blocks.

Imagine a basebackup starts, we disable checksums, and one of the blocks
gets written to the disk without a checksum before the basebackup copies
that file. Then during redo of the backup, we start with checksums=on
(assume we don't have the issue with incorrect state). If we attempt to
read the page before the XLOG_CHECKSUMS, that'll fail, because the
on-disk page does not have a valid checksum, and we need that to read
the page LSN.

I only speculate this can happen, I haven't actually seen / reproduced
it. But maybe it's fixed thanks to FPI, or something like that? It
didn't help on the standby, though. And in that case allowing a random
mix of pages with/without checksums in a basebackup seems problematic.

What could we do about the root cause? We discussed this with Daniel and
we've been stuck for quite a while. But then it occurred to us maybe we
can simply "pause" the checksum state change while there's backup in
progress. We already enable/disable FPW based on this, so why couldn't
we check XLogCtl->Insert.runningBackups, and only advance to the next
checksum state if (runningBackups==0)?

That would mean a single backup does not need to worry about seeing a
mix of blocks written with different checksum states, and it also means
the final pg_control file has the correct checksum state, because it is
not allowed to change during the basebackup.

Of course, this would mean checksum changes may take longer. A corner
case is that database with a basebackup running 100% of the time won't
be able to change checksums on-line. But to me that seems acceptable, if
communicated / documented clearly.

It also occurred to me something like this might help with the standby
case too. On the standby, the problem happens when it skips checkpoints
when creating a restart point, in which case redo may go too far back,
with an incompatible checksum state. Maybe if a standby reported LSN of
the last restartpoint, the primary could use that to decide whether it's
safe to update the checksum state. Of course, this is tricky, because
standbys may be disconnected, there's cascading replications, etc.

regards

-- 
Tomas Vondra

---

From: Tomas Vondra <[email-masked]>
Date: Tue, 18 Nov 2025 20:06:40 +0100

> communicated / documented clearly.

After thinking about this approach a bit, I realized the basebackup may
also run on the standby. Which means the checksum process won't see it
by checking XLogCtl->Insert.runningBackups. It will merrily proceed,
breaking the standby backup just as described earlier ...

Not sure what would be a good fix. One option is to "pause" the redo,
which is what the patch already does (by forcing an immediate checkpoint
whenever checksum state changes). We could pause redo until the backup
completes. But of course, that'd be terrible - especially for syncrep. I
hoped we'd find a better approach, and pausing redo for longer goes in
the opposite direction.

On the other hand, we already have similar issue with full_page_writes.
The backup on standby is not allowed to start if fpw=off, and if the
setting changes while the backup is running, the backup fails:

  pg_basebackup: error: backup failed: ERROR:  WAL generated with
"full_page_writes=off" was replayed during online backup
HINT:  This means that the backup being taken on the standby is corrupt
and should not be used. Enable "full_page_writes" and run CHECKPOINT on
the primary, and then try an online backup again.

Maybe this would be acceptable for checksums too ...

It's not exactly the same, of course. We don't really expect people to
change fpw in a running cluster.

regards

-- 
Tomas Vondra

---

From: Andreas Karlsson <[email-masked]>
Date: Wed, 19 Nov 2025 22:03:37 +0100

Hi,

I have been following these discussions but not read the patch in detail.

This patch makes me worried especially with the new issues recently 
uncovered. This was already a quite big patch and to fix these issues it 
will likely have to become even bigger and given how this would become a 
very rarely stressed code paths I wonder if we can actually ever become 
confident that the patch works in all edge cases.

Something like this need to be easy to understand for us to have any 
hope at all to be comfortable in the correctness. Can we actually do that?

Andreas

---

From: Tomas Vondra <[email-masked]>
Date: Thu, 20 Nov 2025 11:34:14 +0100

> hope at all to be comfortable in the correctness. Can we actually do that?

How's this different from any other complex patch? We get more familiar
with the problem during review, identify issues, improve the patch to
address them. And then again and again.

Of course, it'd be great to have a perfect understanding of the problem
from the very beginning, but that's not always possible. And I can't
guarantee we'll find/fix all issues.

regards

-- 
Tomas Vondra

---

From: Andreas Karlsson <[email-masked]>
Date: Fri, 21 Nov 2025 01:44:31 +0100

> address them. And then again and again.

The difference I see is in how rarely anyone actually switches checksum 
state in a production database, especially now that we enabled them by 
default. A complex and rarely stressed code path is a minefield.

Andreas

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 21 Nov 2025 13:17:09 +0100

> default. A complex and rarely stressed code path is a minefield.

True. Hence the stress testing I've been doing - and indeed, that made
us discover the various issues reported in this thread.

Still, isn't that similar to error paths in various other patches? Those
also tend to be rarely exercised in practice. I think the right way to
address that is more testing. Of course, there's a difference between
"regular bugs" and "design problems". Some of the issues are more about
the design/architecture not considering something important.

I don't know if / when this will be ready for commit. Maybe never, who
knows. I prefer going step by step. We know about a couple issues, we
need to figure out what to do about those. Then we can reconsider.

FWIW I'm not sure the number of people currently enabling checksums on
production databases is a good metric of how important the patch is.
Maybe more people would like to do that, but can't accept the downtime.

regards

-- 
Tomas Vondra

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 21 Nov 2025 14:57:35 -0500

> Maybe more people would like to do that, but can't accept the downtime.

I think it is a worth-while feature.  We would have had it years ago
except that people asked for re-start-ability after a crash, and since
we don't have restart logic at the relation level, the patch got too
complex and was abandoned.

---

From: Andres Freund <[email-masked]>
Date: Fri, 21 Nov 2025 15:28:40 -0500

Hi,

> default. A complex and rarely stressed code path is a minefield.

FWIW, I think this is actually a good feature build the infrastructure for
features (i.e. dynamically reconfiguring the cluster while running) like this,
precisely because it isn't *constantly* used.

Greetings,

Andres Freund

---

From: Daniel Gustafsson <[email-masked]>
Date: Mon, 1 Dec 2025 22:51:04 +0100

> On 5 Nov 2025, at 15:05, Tomas Vondra <[email-masked]> wrote:

> comments.

While I am still working on a few ideas on how to handle the PITR issue, I
didn't want to leave this review hanging longer (and it would be nice to get it
out of the way and not mix it with other issues).

> functions (enable/disable).

I had originally omitted them intentionally since they were intended for
testing, but I agree that they should be documented as they might be useful
outside of testng as well.

Fair point.

> in some way. More about the standby stuff later.

Agreed, but I think it's better done in a one central place like the data
checksums section in wal.sgml (which your XXX also mention).  In the preamble
to the function table I've added a mention of the system performance impact
with a link to the aforementioned section.

> processes when disabling, no?

That's very true, the Worker Launcher will receive the operation and if it is
to disable it will proceed without launching any workers since it's cluster
wide.  Thinking about it, maybe DataChecksumsWorkerLauncher isn't a very good
name, DataChecksumsController or DataChecksumsCoordinator might be a better
choice?

Naming seems to be hard, who knew..

> don't hide/reset the counters?

Correct, I've expanded this para to mention this.

+1

> - Do we want to refer to "datachecksumsworker"? Isn't it a bit too

I think so, we should stick to "launcher process" and "worker process" and be
conistent about it.

> the progress view can return it, it should be in the docs.

Nice catch.  The code can't actually return "waiting" since it was broken up
into three different wait phases, but one of them wasn't documented or added to
the view properly.  Fixed.

+1

+1

> impact on standby ... somewhere. Not sure if this is the right place.

I've added a subsection in the main Data Checksums section for this.

> - Some minor corrections (typos, ...).

Thanks, I did some additional minor wordsmithing around these.

> this in AbsorbDataChecksumsBarrier for all transitions, no?

Correct, fixed.

> state in this context"? Should we explain/define that, or would that be

By "local state" I was referring to the data checksum state that a backend
knows about.  I've reworded this to hopefully be a little clearer.

> required to absorb the signal). It kinda is there, but only indirectly.

I've tried to address these, but they might still be off since I am very
Stockholm syndromed into this.

> of the process (with data_checksum states, barriers, ...) e.g. in the

I instead opted for a SVG image in the docs which illustrates the states and
the transitions.  My Graphviz skills aren't that great so it doesn't look all
that pretty yet, but it's something to iterate on at least.

> We should very clearly document this.

Indeed. I've started on such a section.

> to expect the next state change.

Each worker knows how far it has come within its database, but is unaware about
other databases; the launcher knows how far it has come across all databases,
but is unaware about the relative size of each database.  Maybe there still is
a heuristic that can be teased out of imperfect knowledge.

> And we'd ensure the standby starts

That's an interesting idea, do you know if we have processes taking a similar
approach today?

The attached is a rebase with the above fixes along with a few more smaller
fixups and cleanups noticed along the way, nothing which change any
functionality though.

--
Daniel Gustafsson

---

From: Andres Freund <[email-masked]>
Date: Mon, 15 Dec 2025 17:21:46 -0500

Hi,

One high level issue first: I don't think the way this uses checkpoints and
restartpoints is likely to work out.

Synchronously having to wait for restartpoints during recovery seems generally
operationally a huge issue, but actually could also easily lead to undetected
deadlocks, particularly with syncrep.

Using the checksum state from the control file seems very fraught,
particularly with PITR, as the control file can be "from the future". Which
can be a problem e.g. if checksums were disabled, but we start recovery with a
control file with checksums enabled.

Forcing synchronous checkpoints in a bunch of places also will make this a
good bit slower than necessary, particularly for testing.

My suggestion for how to do this instead is to put the checksum state into the
XLOG_CHECKPOINT_* records.  When starting recovery from an online checkpoint,
I think we should use the ChecksumType from the XLOG_CHECKPOINT_REDO record,
that way the standby/recovery environment's assumption about whether checksums
were enabled is the same as it was at the time the WAL was generated. For
shutdown checkpoints, we could either start to emit a XLOG_CHECKPOINT_REDO, or
we can use the information from the checkpoint record itself.

I think if we only ever use the checksum state from the point where we start
recovery, we might not need to force *any* checkpoints.

Daniel and I chatted about that proposal, and couldn't immediately come up
with scenarios where that would be wrong.  For a while I thought there would
be problems when doing PITR from a base backup that had checksums initially
enabled, but where checksums were disabled before the base backup was
completed. My worry was that a later (once checksums were disabled) hint bit
write (which would not necessarily be WAL logged) would corrupt the checksum,
but I don't think that's a problem, because the startup process will only read
data pages in the process of processing WAL records, and if there's a WAL
record, there would also have to be an FPW, which would "cure" the
unchecksummed page.

More comments below, inline - I wrote these first, so it's possible that I
missed updating some of them in light of what I now wrote above.

> +/*
> + * This must match largest number of sets in barrier_eq and barrier_ne in the
> + * below checksum_barriers definition.
> + */
> +#define MAX_BARRIER_CONDITIONS 2
> +
> +/*
> + * Configuration of conditions which must match when absorbing a procsignal
> + * barrier during data checksum enable/disable operations.  A single function
> + * is used for absorbing all barriers, and the set of conditions to use is
> + * looked up in the checksum_barriers struct.  The struct member for the target
> + * state defines which state the backend must currently be in, and which it
> + * must not be in.
> + */
> +typedef struct ChecksumBarrierCondition
> +{
> +	/* The target state of the barrier */
> +	int			target;
> +	/* A set of states in which at least one MUST match the current state */
> +	int			barrier_eq[MAX_BARRIER_CONDITIONS];
> +	/* The number of elements in the barrier_eq set */
> +	int			barrier_eq_sz;
> +	/* A set of states which all MUST NOT match the current state */
> +	int			barrier_ne[MAX_BARRIER_CONDITIONS];
> +	/* The number of elements in the barrier_ne set */
> +	int			barrier_ne_sz;
> +static const ChecksumBarrierCondition checksum_barriers[] =
> +{
> +	{PG_DATA_CHECKSUM_OFF, {PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION, PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION}, 2, {PG_DATA_CHECKSUM_VERSION}, 1},
> +	{PG_DATA_CHECKSUM_VERSION, {PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION}, 1, {0}, 0},
> +	{PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION, {PG_DATA_CHECKSUM_ANY_VERSION}, 1, {PG_DATA_CHECKSUM_VERSION}, 1},
> +	{PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION, {PG_DATA_CHECKSUM_VERSION}, 1, {0}, 0},
> +	{-1}
> +};

The explanation for this doesn't really explain what the purpose of this thing
is...  Perhaps worth referencing datachecksumsworker.c or such?

For a local, constantly sized, array, you shouldn't need a -1 terminator, as
you can instead use lengthof() or such to detect invalid accesses.

> +/*
> + * Flag to remember if the procsignalbarrier being absorbed for checksums is
> + * the first one.  The first procsignalbarrier can in rare cases be for the
> + * state we've initialized, i.e. a duplicate.  This may happen for any
> + * data_checksum_version value when the process is spawned between the update
> + * of XLogCtl->data_checksum_version and the barrier being emitted.  This can
> + * only happen on the very first barrier so mark that with this flag.
> + */

This is pretty hard to understand right now, at the very least it needs an
updated comment. But perhaps we can just get rid of this and accept barriers
that are redundant.

> @@ -830,9 +905,10 @@ XLogInsertRecord(XLogRecData *rdata,
> @@ -845,7 +921,9 @@ XLogInsertRecord(XLogRecData *rdata,

Why do we need to separately check for DataChecksumsNeedWrite() if turning on
checksums also forces Insert->fullPageWrites to on?

> +SetDataChecksumsOnInProgress(bool immediate_checkpoint)
> +{
> +	uint64		barrier;
> +	int			flags;
> +

ISTM that delayChkptFlags ought to only be set once within the critical
section. Obviously we can't fail inbetween as the code stands, but that's not
guaranteed to stay this way.

> +	SpinLockRelease(&XLogCtl->info_lck);

Maybe worth adding an assertion checking that we are currently in an expected
state (off or inprogress, I think?).

> +	MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;

Swap as well.

Think it might be worth mentioning that we rely on the memory ordering implied
by XLogChecksums() and WaitForProcSignalBarrier() for the changes to
delayChkptFlags. Unless we have a different logic around that?

> +	 * force checkpoint to persist the current checksum state in control file
> +	 * etc.
> +	 *
> +	 * XXX is this needed? There's already a checkpoint at the end of
> +	 * ProcessAllDatabases, maybe this is redundant?
> +	 */
> +		flags = flags | CHECKPOINT_FAST;

Why do we need a checkpoint at all?

> + * set the state to "inprogress-on" (done by SetDataChecksumsOnInProgress())
> + * and the second one to set the state to "on" (done here). Below is a short
> + * description of the processing, a more detailed write-up can be found in
> + * datachecksumsworker.c.
> + *
> +SetDataChecksumsOn(bool immediate_checkpoint)
>  {
> +	uint64		barrier;
> +	int			flags;
> +

Seems like a PANIC condition to me...

> +	INJECTION_POINT("datachecksums-enable-checksums-delay", NULL);
> +	START_CRIT_SECTION();

I think it's a really really bad idea to do something fallible, like
INJECTION_POINT, after setting delayChkptFlags, but before entering the crit
section. Any error in the injection point will lead to a corrupted
delayChkptFlags, no?

> +	INJECTION_POINT("datachecksums-enable-checksums-pre-checkpoint", NULL);
> +
> +	/* XXX is this needed? */
> +		flags = flags | CHECKPOINT_FAST;
> +SetDataChecksumsOff(bool immediate_checkpoint)
> +{
> +	[...]
> +	/*

Why no symbolic name here?

> +		flags = flags | CHECKPOINT_FAST;
> + * AbsorbDataChecksumsBarrier
> + *		Generic function for absorbing data checksum state changes
> + *
> + * All procsignalbarriers regarding data checksum state changes are absorbed
> + * with this function.  The set of conditions required for the state change to
> + * be accepted are listed in the checksum_barriers struct, target_state is
> + * used to look up the relevant entry.
> + */
> +bool
> +AbsorbDataChecksumsBarrier(int target_state)
> +{
> +	int			current = LocalDataChecksumVersion;
> +	bool		found = false;
> +
> +	/*
> +	 * Find the barrier condition definition for the target state. Not finding
> +	 * a condition would be a grave programmer error as the states are a
> +	 * discrete set.
> +	 */
> +	while (condition->target != target_state && condition->target != -1)
> +		condition++;
> +	if (unlikely(condition->target == -1))
> +		ereport(ERROR,
> +				errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> +				errmsg("invalid target state %i for data checksum barrier",
> +					   target_state));

FWIW, you don't need unlikely() when the branch does an ereport(ERROR), as
ereports >=ERROR are marked "cold" automatically.

> +	/*
> +	 * The current state MUST be equal to one of the EQ states defined in this
> +	 * barrier condition, or equal to the target_state if - and only if -
> +	 * InitialDataChecksumTransition is true.
> +	 */
> +	for (int i = 0; i < condition->barrier_eq_sz; i++)
> +	{
> +		if (current == condition->barrier_eq[i] ||
> +			condition->barrier_eq[i] == PG_DATA_CHECKSUM_ANY_VERSION)
> +			found = true;
> +	}
> +	if (InitialDataChecksumTransition && current == target_state)
> +		found = true;
> +
> +	/*
> +	 * The current state MUST NOT be equal to any of the NE states defined in
> +	 * this barrier condition.
> +	 */
> +	for (int i = 0; i < condition->barrier_ne_sz; i++)
> +	{
> +		if (current == condition->barrier_ne[i])
> +			found = false;
> +	}
> +
> +	/*
> +	 * If the relevent state criteria aren't satisfied, throw an error which
> +	 * will be caught by the procsignal machinery for a later retry.
> +	 */
> +	if (!found)
> +		ereport(ERROR,
> +				errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> +				errmsg("incorrect data checksum state %i for target state %i",
> +					   current, target_state));
> +
> +	SetLocalDataChecksumVersion(target_state);

> +XLogChecksums(uint32 new_type)
> +{
> +	INJECTION_POINT("datachecksums-xlogchecksums-pre-xloginsert", &new_type);
> +

Why an injection point between XLogBeginInsert() and XLogInsert(), rather than
have the injection point before the XLogBeginInsert()?

> +		 * XXX Could this end up written to the control file prematurely? IIRC
> +		 * that happens during checkpoint, so what if that gets triggered e.g.
> +		 * because someone runs CHECKPOINT? If we then crash (or something
> +		 * like that), could that confuse the instance?
> +		 */

I'd not add a default clause, but add a case each value of the enum. That way
we'll get warnings if the set of states changes.

These WaitForProcSignalBarrier() are one of the scariest bits of the
patchset. If the startup process were to hold any lock that backends need, and
the backends waited for that lock without processing interrupts, we'd have an
undetected deadlock.  This is much more likely to be a problem for the startup
process, as it does the work of many backends on the primary.

We do process interrupts while waiting for heavyweight locks, so that at least
is not an issue. Seems worth to call out rather explicitly.

> +	if (checksumRestartPoint &&
> +		(info == XLOG_CHECKPOINT_ONLINE ||
> +		 info == XLOG_CHECKPOINT_REDO ||
> +		 info == XLOG_CHECKPOINT_SHUTDOWN))
> +	{
> +		int			flags;
> +
> +		elog(LOG, "forcing creation of a restartpoint after XLOG_CHECKSUMS");
> +
> +		/* We explicitly want an immediate checkpoint here */
> +		flags = CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST;
> +		checksumRestartPoint = false;
> +	}

As noted above, I don't think we should rely on starting restartpoints.

> index 2be4e069816..baf6c8cc2cc 100644

Why is DataChecksumsNeedWrite() being tested here?

> index 086c4c8fb6f..6d452b10bce 100644
> @@ -1374,6 +1374,26 @@ CREATE VIEW pg_stat_progress_copy AS
> +                      WHEN 2 THEN 'waiting on temporary tables'
> +                      WHEN 3 THEN 'waiting on checkpoint'
> +					  WHEN 4 THEN 'waiting on barrier'
> +                      WHEN 5 THEN 'done'
> +                      END AS phase,

Not this patch's fault, but I strongly dislike that we do this in SQL. Every
postgres database in the world has ~110kB of pg_stat_progress view definitions
in it.  We should just do this in a C function.

> index 00000000000..57311760b2b
> --- /dev/null
> @@ -0,0 +1,1491 @@
> + * have checksums set, but no reads will fail due to incorrect checksum.

Maybe "... due to not yet set checksums."?  Incorrect checksums sounds like
it's about checksums that are actively wrong, rather than just not set. Except
for the corner case of a torn page in the process of an hint bit write, after
having disabled checksums, there shouldn't be incorrect ones, right?

> The

What about new databases created while checksums are being enabled? They could
be copied before the worker has processed them. At least for the file_copy
strategy, the copy will be verbatim and thus will not necessarily have
checksums set.

> + *    - Backends SHALL NOT violate the data_checksums state they have agreed to
> + *      by acknowledging the procsignalbarrier:  This means that all backends
> + *      MUST calculate and write data checksums during all states except off;
> + *      MUST validate checksums only in the 'on' state.
> + *      currently connected backends have state "on": This means that all
> + *      backends must wait on the procsignalbarrier to be acknowledged by all
> + *      before proceeding to validate data checksums.
> + *
> + * There are two levels of synchronization required for changing data_checksums

Maybe s/levels/steps/?

> + * WAL logging backend updating the global state in the controlfile will wait

It's not entirely obvious what "the WAL logging backend" means.

> + *   When Enabling Data Checksums

Odd change in indentation here.

> + *   will be placed in Bd.

Why not in Bi? Just for simplicities sake? ISTM we already need to be sure
that new backends start in Bi, as they might never observe the barrier...

> Backends transition Bd -> Bi via a procsignalbarrier
> + *   which is emitted by the DataChecksumsLauncher.  When all backends have
> + *   acknowledged the barrier then Bd will be empty and the next phase can
> + *   begin: calculating and writing data checksums with DataChecksumsWorkers.
> + *   When the DataChecksumsWorker processes have finished writing checksums on
> + *   all pages and enables data checksums cluster-wide via another

s/enables/enabled/?

> + *   procsignalbarrier, there are four sets of backends where Bd shall be an
> + *   empty set:
> + *

Hm. I wonder if we ought to zero out old checksums when loading a page into
s_b with checksums disabled... But that's really independent of this patchset.

> + * Potential optimizations

The last sentence might be truncated?

> + *	  src/backend/postmaster/datachecksumsworker.c

Hm. The whole set of interactions with checkpoints/restartpoints aren't explored
here?

> +	 * As of now we only update the block counter for main forks in order to
> +	 * not cause too frequent calls. TODO: investigate whether we should do it
> +	 * more frequent?
> +	 */

That doesn't make much sense to me. Presumably the reason to skip it for the
other forks is that they're small-ish. But if so, there's no point in skipping
the reporting either, as presumably there wouldn't be a lot of reporting?

> +	for (BlockNumber blknum = 0; blknum < numblocks; blknum++)
> +	{
> +		 * off, and then turned on again.  TODO: investigate if this could be
> +		 * avoided if the checksum is calculated to be correct and wal_level
> +		 * is set to "minimal",
> +		 */
> +		START_CRIT_SECTION();
> +		MarkBufferDirty(buf);

Hm. It's pretty annoying to have to pass page_std = false here, that could
increase the write volume noticeably. But there's not a great way to know
what the right value would be :(

> +		LWLockRelease(DataChecksumsWorkerLock);

Could we end up exiting this with the worker still running?

> + * is changed while we're running.

I think either here, or at its callsites, the patch needs to explain *why* we
are waiting for all transactions to finish.  Presumably this is to ensure that
other sessions haven't created relations that we can't see yet?

It actually doesn't seem to wait for all transactions, ust for ones with an
xid?

> +	while (TransactionIdPrecedes(GetOldestActiveTransactionId(false, true), waitforxid))
> +	{
> +		char		activity[64];
> +		int			rc;
> +
> +		/* Retry every 3 seconds */
> +		ResetLatch(MyLatch);
> +					   3000,
> +					errcode(ERRCODE_ADMIN_SHUTDOWN),
> +					errmsg("postmaster exited during data checksum processing"),
> +					errhint("Restart the database and restart data checksum processing by calling pg_enable_data_checksums()."));
> +
> +		if (DataChecksumsWorkerShmem->launch_operation != operation)

I don't like this much - loops with a timeout are generally a really bad idea
and we shouldn't add more instances. Presumably this also makes the tests
slower...

How about collecting the to-be-waited-for virtualxids and then wait for those?

> +	if (operation == ENABLE_DATACHECKSUMS)
> +	{
> +		/*

I don't understand what this interrupt stuff achieves here?

> index 25d13a798d1..0faaac14b1b 100644
> @@ -15,6 +15,21 @@
> + * Checksum version 0 is used for when data checksums are disabled (OFF).
> +	PG_DATA_CHECKSUM_OFF = 0,
> +	PG_DATA_CHECKSUM_VERSION,
> +	PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION,
> +	PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION,
> +	PG_DATA_CHECKSUM_ANY_VERSION
> +} ChecksumType;

Why is there "VERSION" in the name of these?  Feels like that's basically just
vestigial at this point.

>  /*
>   * There also exist several built-in LWLock tranches.  As with the predefined
> index c6f5ebceefd..d90d35b1d6f 100644
> @@ -463,11 +463,11 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;

Aren't they bgworkers now?

> +	PROCSIGNAL_BARRIER_CHECKSUM_ON,

I wonder if these really should be different barriers. What if we just made it
one, and instead drove the transition on the current shmem content?

Other stuff:
- what protects against multiple backends enabling checksums at the same time?

  Afaict there isn't anything, and we just ignore the second request. Which
  seems ok-ish if it's the same request as before, but not great if it's a
  different one.

  Should also have tests for that.

- I think this adds a bit too much of the logic to xlog.c, already an unwieldy
  file. A fair bit of all of this doesn't seem like it needs to be in there.

- the code seems somewhat split brained about bgworkers and auxprocesses

Greetings,

Andres Freund