From: Robert Haas <rhaas@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Allow parallel zstd compression when taking a base backup.
Date: Wed, 30 Mar 2022 13:50:49 +0000
Message-ID: <E1nZYin-0000WJ-E8@gemulon.postgresql.org> (raw)
Allow parallel zstd compression when taking a base backup.
libzstd allows transparent parallel compression just by setting
an option when creating the compression context, so permit that
for both client and server-side backup compression. To use this,
use something like pg_basebackup --compress WHERE-zstd:workers=N
where WHERE is "client" or "server" and N is an integer.
When compression is performed on the server side, this will spawn
threads inside the PostgreSQL backend. While there is almost no
PostgreSQL server code which is thread-safe, the threads here are used
internally by libzstd and touch only data structures controlled by
libzstd.
Patch by me, based in part on earlier work by Dipesh Pandit
and Jeevan Ladhe. Reviewed by Justin Pryzby.
Discussion: http://postgr.es/m/CA+Tgmobj6u-nWF-j=FemygUhobhryLxf9h-wJN7W-2rSsseHNA@mail.gmail.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/51c0d186d99a18e6aae53003f5138f20991e15a6
Modified Files
--------------
doc/src/sgml/protocol.sgml | 12 +++++--
doc/src/sgml/ref/pg_basebackup.sgml | 4 +--
src/backend/replication/basebackup_zstd.c | 45 +++++++++++++++++----------
src/bin/pg_basebackup/bbstreamer_zstd.c | 40 +++++++++++++++++-------
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 5 +++
src/bin/pg_verifybackup/t/009_extract.pl | 29 +++++++++++++++--
src/bin/pg_verifybackup/t/010_client_untar.pl | 33 ++++++++++++++++++--
src/common/backup_compression.c | 16 ++++++++++
src/include/common/backup_compression.h | 2 ++
9 files changed, 147 insertions(+), 39 deletions(-)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-committers@postgresql.org
Cc: rhaas@postgresql.org
Subject: Re: pgsql: Allow parallel zstd compression when taking a base backup.
In-Reply-To: <E1nZYin-0000WJ-E8@gemulon.postgresql.org>
References: <E1nZYin-0000WJ-E8@gemulon.postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox