From: Robert Haas <rhaas@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Introduce 'bbsink' abstraction to modularize base backup code.
Date: Fri, 05 Nov 2021 14:19:25 +0000
Message-ID: <E1mj03x-0007tU-Se@gemulon.postgresql.org> (raw)
Introduce 'bbsink' abstraction to modularize base backup code.
The base backup code has accumulated a healthy number of new
features over the years, but it's becoming increasingly difficult
to maintain and further enhance that code because there's no
real separation of concerns. For example, the code that
understands knows the details of how we send data to the client
using the libpq protocol is scattered throughout basebackup.c,
rather than being centralized in one place.
To try to improve this situation, introduce a new 'bbsink' object
which acts as a recipient for archives generated during the base
backup progress and also for the backup manifest. This commit
introduces three types of bbsink: a 'copytblspc' bbsink forwards the
backup to the client using one COPY OUT operation per tablespace and
another for the manifest, a 'progress' bbsink performs command
progress reporting, and a 'throttle' bbsink performs rate-limiting.
The 'progress' and 'throttle' bbsink types also forward the data to a
successor bbsink; at present, the last bbsink in the chain will
always be of type 'copytblspc'. There are plans to add more types
of 'bbsink' in future commits.
This abstraction is a bit leaky in the case of progress reporting,
but this still seems cleaner than what we had before.
Patch by me, reviewed and tested by Andres Freund, Sumanta Mukherjee,
Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja, Mark Dilger,
and Jeevan Ladhe.
Discussion: https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/bef47ff85df18bf4a3a9b13bd2a54820e27f3614
Modified Files
--------------
src/backend/replication/Makefile | 4 +
src/backend/replication/backup_manifest.c | 28 +-
src/backend/replication/basebackup.c | 692 ++++++++------------------
src/backend/replication/basebackup_copy.c | 335 +++++++++++++
src/backend/replication/basebackup_progress.c | 246 +++++++++
src/backend/replication/basebackup_sink.c | 125 +++++
src/backend/replication/basebackup_throttle.c | 199 ++++++++
src/include/replication/backup_manifest.h | 5 +-
src/include/replication/basebackup_sink.h | 296 +++++++++++
9 files changed, 1414 insertions(+), 516 deletions(-)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-committers@postgresql.org
Cc: rhaas@postgresql.org
Subject: Re: pgsql: Introduce 'bbsink' abstraction to modularize base backup code.
In-Reply-To: <E1mj03x-0007tU-Se@gemulon.postgresql.org>
References: <E1mj03x-0007tU-Se@gemulon.postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox