From: Robert Haas <rhaas@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Add a new WAL summarizer process.
Date: Wed, 20 Dec 2023 13:53:20 +0000
Message-ID: <E1rFx0i-00B5bY-43@gemulon.postgresql.org> (raw)
Add a new WAL summarizer process.
When active, this process writes WAL summary files to
$PGDATA/pg_wal/summaries. Each summary file contains information for a
certain range of LSNs on a certain TLI. For each relation, it stores a
"limit block" which is 0 if a relation is created or destroyed within
a certain range of WAL records, or otherwise the shortest length to
which the relation was truncated during that range of WAL records, or
otherwise InvalidBlockNumber. In addition, it stores a list of blocks
which have been modified during that range of WAL records, but
excluding blocks which were removed by truncation after they were
modified and never subsequently modified again.
In other words, it tells us which blocks need to copied in case of an
incremental backup covering that range of WAL records. But this
doesn't yet add the capability to actually perform an incremental
backup; the next patch will do that.
A new parameter summarize_wal enables or disables this new background
process. The background process also automatically deletes summary
files that are older than wal_summarize_keep_time, if that parameter
has a non-zero value and the summarizer is configured to run.
Patch by me, with some design help from Dilip Kumar and Andres Freund.
Reviewed by Matthias van de Meent, Dilip Kumar, Jakub Wartak, Peter
Eisentraut, and Álvaro Herrera.
Discussion: http://postgr.es/m/CA+TgmoYOYZfMCyOXFyC-P+-mdrZqm5pP2N7S-r0z3_402h9rsA@mail.gmail.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/174c480508ac25568561443e6d4a82d5c1103487
Modified Files
--------------
doc/src/sgml/config.sgml | 61 +
src/backend/access/transam/xlog.c | 101 +-
src/backend/backup/Makefile | 4 +-
src/backend/backup/meson.build | 2 +
src/backend/backup/walsummary.c | 356 ++++++
src/backend/backup/walsummaryfuncs.c | 169 +++
src/backend/postmaster/Makefile | 1 +
src/backend/postmaster/auxprocess.c | 8 +
src/backend/postmaster/meson.build | 1 +
src/backend/postmaster/postmaster.c | 56 +
src/backend/postmaster/walsummarizer.c | 1398 +++++++++++++++++++++++
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/utils/activity/pgstat_io.c | 4 +-
src/backend/utils/activity/wait_event_names.txt | 5 +
src/backend/utils/init/miscinit.c | 3 +
src/backend/utils/misc/guc_tables.c | 26 +
src/backend/utils/misc/postgresql.conf.sample | 5 +
src/bin/initdb/initdb.c | 1 +
src/common/Makefile | 1 +
src/common/blkreftable.c | 1308 +++++++++++++++++++++
src/common/meson.build | 1 +
src/include/access/xlog.h | 1 +
src/include/backup/walsummary.h | 49 +
src/include/catalog/pg_proc.dat | 19 +
src/include/common/blkreftable.h | 116 ++
src/include/miscadmin.h | 3 +
src/include/postmaster/walsummarizer.h | 33 +
src/include/storage/proc.h | 9 +-
src/include/utils/guc_tables.h | 1 +
src/tools/pgindent/typedefs.list | 11 +
30 files changed, 3743 insertions(+), 11 deletions(-)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-committers@postgresql.org
Cc: rhaas@postgresql.org, pgsql-committers@lists.postgresql.org
Subject: Re: pgsql: Add a new WAL summarizer process.
In-Reply-To: <E1rFx0i-00B5bY-43@gemulon.postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox