pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: refactoring basebackup.c (zstd workers)
32+ messages / 4 participants
[nested] [flat]
Commits from this thread:
2022-03-30 8e053dc6df Fix possible NULL-pointer-deference in backup_compression.c.
2022-03-23 ffd53659c4 Replace BASE_BACKUP COMPRESSION_LEVEL option with COMPRESSION_DETAIL.
* Re: refactoring basebackup.c (zstd workers)
@ 2022-03-14 16:35 Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 32+ messages in thread
From: Justin Pryzby @ 2022-03-14 16:35 UTC (permalink / raw)
To: Dipesh Pandit <dipesh.pandit@gmail.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers@postgresql.org, tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 14, 2022 at 09:41:35PM +0530, Dipesh Pandit wrote:
> I tried to implement support for parallel ZSTD compression. The
> library provides an option (ZSTD_c_nbWorkers) to specify the
> number of compression workers. The number of parallel
> workers can be set as part of compression parameter and if this
> option is specified then the library performs parallel compression
> based on the specified number of workers.
>
> User can specify the number of parallel worker as part of
> --compress option by appending an integer value after at sign (@).
> (-Z, --compress=[{client|server}-]{gzip|lz4|zstd}[:LEVEL][@WORKERS])
I suggest to use a syntax that's more general than that, maybe something like
:[level=]N,parallel=N,flag,flag,...
For example, someone may want to use zstd "long" mode or (when it's released)
rsyncable mode, or specify fine-grained compression parameters (strategy,
windowLog, hashLog, etc).
I hope the same syntax will be shared with wal_compression and pg_dump.
And libpq, if that patch progresses.
BTW, I think this may be better left for PG16.
--
Justin
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
@ 2022-03-14 17:02 ` Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 32+ messages in thread
From: Robert Haas @ 2022-03-14 17:02 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 14, 2022 at 12:35 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> I suggest to use a syntax that's more general than that, maybe something like
>
> :[level=]N,parallel=N,flag,flag,...
>
> For example, someone may want to use zstd "long" mode or (when it's released)
> rsyncable mode, or specify fine-grained compression parameters (strategy,
> windowLog, hashLog, etc).
That's an interesting idea. I wonder what the replication protocol
ought to look like in that case. Should we have a COMPRESSION_DETAIL
argument that is just a string, and let the server parse it out? Or
separate protocol-level options? It does feel reasonable to have both
COMPRESSION_LEVEL and COMPRESSION_WORKERS as first-class options, but
I don't know that we want COMPRESSION_HASHLOG true as part of our
first-class grammar.
> I hope the same syntax will be shared with wal_compression and pg_dump.
> And libpq, if that patch progresses.
>
> BTW, I think this may be better left for PG16.
Possibly so ... but if we're thinking of any revisions to the
newly-added grammar, we had better take care of that now, before it's
set in stone.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-14 17:11 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 32+ messages in thread
From: Justin Pryzby @ 2022-03-14 17:11 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 14, 2022 at 01:02:20PM -0400, Robert Haas wrote:
> On Mon, Mar 14, 2022 at 12:35 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> > I suggest to use a syntax that's more general than that, maybe something like
> >
> > :[level=]N,parallel=N,flag,flag,...
> >
> > For example, someone may want to use zstd "long" mode or (when it's released)
> > rsyncable mode, or specify fine-grained compression parameters (strategy,
> > windowLog, hashLog, etc).
>
> That's an interesting idea. I wonder what the replication protocol
> ought to look like in that case. Should we have a COMPRESSION_DETAIL
> argument that is just a string, and let the server parse it out? Or
> separate protocol-level options? It does feel reasonable to have both
> COMPRESSION_LEVEL and COMPRESSION_WORKERS as first-class options, but
> I don't know that we want COMPRESSION_HASHLOG true as part of our
> first-class grammar.
I was only referring to the user-facing grammar.
Internally, I was thinking they'd all be handled as first-class options, with
separate struct fields and separate replication protocol options. If an option
isn't known, it'd be rejected on the client side, rather than causing an error
on the server.
Maybe there'd be an option parser for this in common/ (I think that might
require having new data structure there too, maybe one for each compression
method, or maybe a union{} to handles them all). Most of the ~100 lines to
support wal_compression='zstd:N' are to parse out the N.
--
Justin
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
@ 2022-03-14 17:21 ` Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 32+ messages in thread
From: Robert Haas @ 2022-03-14 17:21 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 14, 2022 at 1:11 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> Internally, I was thinking they'd all be handled as first-class options, with
> separate struct fields and separate replication protocol options. If an option
> isn't known, it'd be rejected on the client side, rather than causing an error
> on the server.
There's some appeal to that, but one downside is that it means that
the client can't be used to fetch data that is compressed in a way
that the server knows about and the client doesn't. I don't think
that's great. Why should, for example, pg_basebackup need to be
compiled with zstd support in order to request zstd compression on the
server side? If the server knows about the brand new
justin-magic-sauce compression algorithm, maybe the client should just
be able to request it and, when given various .jms files by the
server, shrug its shoulders and accept them for what they are. That
doesn't work if -Fp is involved, or similar, but it should work fine
for simple cases if we set things up right.
> Maybe there'd be an option parser for this in common/ (I think that might
> require having new data structure there too, maybe one for each compression
> method, or maybe a union{} to handles them all). Most of the ~100 lines to
> support wal_compression='zstd:N' are to parse out the N.
Yes, it's actually a very simple feature now that we've got the rest
of the infrastructure set up correctly for it.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-17 15:50 ` Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 32+ messages in thread
From: Robert Haas @ 2022-03-17 15:50 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 14, 2022 at 1:21 PM Robert Haas <robertmhaas@gmail.com> wrote:
> There's some appeal to that, but one downside is that it means that
> the client can't be used to fetch data that is compressed in a way
> that the server knows about and the client doesn't. I don't think
> that's great. Why should, for example, pg_basebackup need to be
> compiled with zstd support in order to request zstd compression on the
> server side? If the server knows about the brand new
> justin-magic-sauce compression algorithm, maybe the client should just
> be able to request it and, when given various .jms files by the
> server, shrug its shoulders and accept them for what they are. That
> doesn't work if -Fp is involved, or similar, but it should work fine
> for simple cases if we set things up right.
Concretely, I propose the attached patch for v15. It renames the
newly-added COMPRESSION_LEVEL option to COMPRESSION_DETAIL, introduces
a flexible syntax for options along the lines you proposed, and
adjusts things so that a client that doesn't support a particular type
of compression can still request that type of compression from the
server.
I think it's important to do this for v15 so that we don't end up with
backward-compatibility problems down the road.
--
Robert Haas
EDB: http://www.enterprisedb.com
Attachments:
v1-0001-Replace-BASE_BACKUP-COMPRESSION_LEVEL-option-with.patch (application/octet-stream, 52.5K) download
From dc3e4fb520b6bfe2cdee66a4fa6133b8fae76b1f Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Thu, 17 Mar 2022 11:32:04 -0400
Subject: [PATCH v1] Replace BASE_BACKUP COMPRESSION_LEVEL option with
COMPRESSION_DETAIL.
There are more compression parameters that can be specified than just
an integer compression level, so rename the new COMPRESSION_LEVEL
option to COMPRESSION_DETAIL before it gets released. Introduce a
flexible syntax for that option to allow arbitrary options to be
specified without needing to adjust the main replication grammar,
and common code to parse it that is shared between the client and
the server.
This commit doesn't actually add any new compression parameters,
so the only user-visible change is that you can now type something
like pg_basebackup --compress gzip:level=5 instead of writing just
pg_basebackup --compress gzip:5. However, it should make it easy to
add new options. If for example gzip starts offering fries, we can
support pg_basebackup --compress gzip:level=5,fries=true for the
benefit of users who want fries with that.
Along the way, this fixes a few things in pg_basebackup so that the
pg_basebackup can be used with a server-side compression algorithm
that pg_basebackup itself does not understand. For example,
pg_basebackup --compress server-lz4 could still succeed even if
only the server and not the client has LZ4 support, provided that
the other options to pg_basebackup don't require the client to
decompress the archive.
---
doc/src/sgml/protocol.sgml | 18 +-
src/backend/replication/basebackup.c | 60 +--
src/backend/replication/basebackup_gzip.c | 20 +-
src/backend/replication/basebackup_lz4.c | 19 +-
src/backend/replication/basebackup_zstd.c | 19 +-
src/bin/pg_basebackup/bbstreamer.h | 7 +-
src/bin/pg_basebackup/bbstreamer_gzip.c | 7 +-
src/bin/pg_basebackup/bbstreamer_lz4.c | 4 +-
src/bin/pg_basebackup/bbstreamer_zstd.c | 4 +-
src/bin/pg_basebackup/pg_basebackup.c | 405 ++++++++-----------
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 33 +-
src/common/Makefile | 1 +
src/common/backup_compression.c | 256 ++++++++++++
src/include/common/backup_compression.h | 44 ++
src/include/replication/basebackup_sink.h | 7 +-
src/tools/msvc/Mkvcbuild.pm | 2 +-
src/tools/pgindent/typedefs.list | 2 +
17 files changed, 594 insertions(+), 314 deletions(-)
create mode 100644 src/common/backup_compression.c
create mode 100644 src/include/common/backup_compression.h
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9178c779ba..00c593f1af 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2731,14 +2731,24 @@ The commands accepted in replication mode are:
</varlistentry>
<varlistentry>
- <term><literal>COMPRESSION_LEVEL</literal> <replaceable>level</replaceable></term>
+ <term><literal>COMPRESSION_DETAIL</literal> <replaceable>detail</replaceable></term>
<listitem>
<para>
Specifies the compression level to be used. This should only be
used in conjunction with the <literal>COMPRESSION</literal> option.
- For <literal>gzip</literal> the value should be an integer between 1
- and 9, for <literal>lz4</literal> between 1 and 12, and for
- <literal>zstd</literal> it should be between 1 and 22.
+ If the value is an integer, it specifies the compression level.
+ Otherwise, it should be a comma-separated list of items, each of
+ the form <literal>keyword</literal> or
+ <literal>keyword=value</literal>. Currently, the only supported
+ keyword is <literal>level</literal>, which sets the compression
+ level.
+ </para>
+
+ <para>
+ For <literal>gzip</literal> the compression level should be an
+ integer between 1 and 9, for <literal>lz4</literal> an integer
+ between 1 and 12, and for <literal>zstd</literal> an integer
+ between 1 and 22.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index c2aedc14a2..afaf4f9ce1 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -18,6 +18,7 @@
#include "access/xlog_internal.h" /* for pg_start/stop_backup */
#include "common/file_perm.h"
+#include "common/backup_compression.h"
#include "commands/defrem.h"
#include "lib/stringinfo.h"
#include "miscadmin.h"
@@ -54,14 +55,6 @@
*/
#define SINK_BUFFER_LENGTH Max(32768, BLCKSZ)
-typedef enum
-{
- BACKUP_COMPRESSION_NONE,
- BACKUP_COMPRESSION_GZIP,
- BACKUP_COMPRESSION_LZ4,
- BACKUP_COMPRESSION_ZSTD
-} basebackup_compression_type;
-
typedef struct
{
const char *label;
@@ -75,8 +68,8 @@ typedef struct
bool use_copytblspc;
BaseBackupTargetHandle *target_handle;
backup_manifest_option manifest;
- basebackup_compression_type compression;
- int compression_level;
+ bc_algorithm compression;
+ bc_specification compression_specification;
pg_checksum_type manifest_checksum_type;
} basebackup_options;
@@ -713,12 +706,14 @@ parse_basebackup_options(List *options, basebackup_options *opt)
char *target_str = NULL;
char *target_detail_str = NULL;
bool o_compression = false;
- bool o_compression_level = false;
+ bool o_compression_detail = false;
+ char *compression_detail_str = NULL;
MemSet(opt, 0, sizeof(*opt));
opt->manifest = MANIFEST_OPTION_NO;
opt->manifest_checksum_type = CHECKSUM_TYPE_CRC32C;
opt->compression = BACKUP_COMPRESSION_NONE;
+ opt->compression_specification.algorithm = BACKUP_COMPRESSION_NONE;
foreach(lopt, options)
{
@@ -885,29 +880,21 @@ parse_basebackup_options(List *options, basebackup_options *opt)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("duplicate option \"%s\"", defel->defname)));
- if (strcmp(optval, "none") == 0)
- opt->compression = BACKUP_COMPRESSION_NONE;
- else if (strcmp(optval, "gzip") == 0)
- opt->compression = BACKUP_COMPRESSION_GZIP;
- else if (strcmp(optval, "lz4") == 0)
- opt->compression = BACKUP_COMPRESSION_LZ4;
- else if (strcmp(optval, "zstd") == 0)
- opt->compression = BACKUP_COMPRESSION_ZSTD;
- else
+ if (!parse_bc_algorithm(optval, &opt->compression))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("unrecognized compression algorithm: \"%s\"",
+ errmsg("unrecognized compression algorithm \"%s\"",
optval)));
o_compression = true;
}
- else if (strcmp(defel->defname, "compression_level") == 0)
+ else if (strcmp(defel->defname, "compression_detail") == 0)
{
- if (o_compression_level)
+ if (o_compression_detail)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("duplicate option \"%s\"", defel->defname)));
- opt->compression_level = defGetInt32(defel);
- o_compression_level = true;
+ compression_detail_str = defGetString(defel);
+ o_compression_detail = true;
}
else
ereport(ERROR,
@@ -949,10 +936,25 @@ parse_basebackup_options(List *options, basebackup_options *opt)
opt->target_handle =
BaseBackupGetTargetHandle(target_str, target_detail_str);
- if (o_compression_level && !o_compression)
+ if (o_compression_detail && !o_compression)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("compression level requires compression")));
+
+ if (o_compression)
+ {
+ char *error_detail;
+
+ parse_bc_specification(opt->compression, compression_detail_str,
+ &opt->compression_specification);
+ error_detail =
+ validate_bc_specification(&opt->compression_specification);
+ if (error_detail != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("invalid compression specification: %s",
+ error_detail));
+ }
}
@@ -998,11 +1000,11 @@ SendBaseBackup(BaseBackupCmd *cmd)
/* Set up server-side compression, if client requested it */
if (opt.compression == BACKUP_COMPRESSION_GZIP)
- sink = bbsink_gzip_new(sink, opt.compression_level);
+ sink = bbsink_gzip_new(sink, &opt.compression_specification);
else if (opt.compression == BACKUP_COMPRESSION_LZ4)
- sink = bbsink_lz4_new(sink, opt.compression_level);
+ sink = bbsink_lz4_new(sink, &opt.compression_specification);
else if (opt.compression == BACKUP_COMPRESSION_ZSTD)
- sink = bbsink_zstd_new(sink, opt.compression_level);
+ sink = bbsink_zstd_new(sink, &opt.compression_specification);
/* Set up progress reporting. */
sink = bbsink_progress_new(sink, opt.progress);
diff --git a/src/backend/replication/basebackup_gzip.c b/src/backend/replication/basebackup_gzip.c
index b66d3da7a3..703a91ba77 100644
--- a/src/backend/replication/basebackup_gzip.c
+++ b/src/backend/replication/basebackup_gzip.c
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_gzip_ops = {
#endif
/*
- * Create a new basebackup sink that performs gzip compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs gzip compression.
*/
bbsink *
-bbsink_gzip_new(bbsink *next, int compresslevel)
+bbsink_gzip_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef HAVE_LIBZ
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,15 +72,14 @@ bbsink_gzip_new(bbsink *next, int compresslevel)
bbsink_gzip *sink;
Assert(next != NULL);
- Assert(compresslevel >= 0 && compresslevel <= 9);
- if (compresslevel == 0)
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
compresslevel = Z_DEFAULT_COMPRESSION;
- else if (compresslevel < 0 || compresslevel > 9)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("gzip compression level %d is out of range",
- compresslevel)));
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 9);
+ }
sink = palloc0(sizeof(bbsink_gzip));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_gzip_ops;
diff --git a/src/backend/replication/basebackup_lz4.c b/src/backend/replication/basebackup_lz4.c
index d838f723d0..06c161ddc4 100644
--- a/src/backend/replication/basebackup_lz4.c
+++ b/src/backend/replication/basebackup_lz4.c
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_lz4_ops = {
#endif
/*
- * Create a new basebackup sink that performs lz4 compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs lz4 compression.
*/
bbsink *
-bbsink_lz4_new(bbsink *next, int compresslevel)
+bbsink_lz4_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef USE_LZ4
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -72,11 +73,13 @@ bbsink_lz4_new(bbsink *next, int compresslevel)
Assert(next != NULL);
- if (compresslevel < 0 || compresslevel > 12)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("lz4 compression level %d is out of range",
- compresslevel)));
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
+ compresslevel = 0;
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 12);
+ }
sink = palloc0(sizeof(bbsink_lz4));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_lz4_ops;
diff --git a/src/backend/replication/basebackup_zstd.c b/src/backend/replication/basebackup_zstd.c
index c0e2be6e27..96b7985693 100644
--- a/src/backend/replication/basebackup_zstd.c
+++ b/src/backend/replication/basebackup_zstd.c
@@ -55,12 +55,13 @@ const bbsink_ops bbsink_zstd_ops = {
#endif
/*
- * Create a new basebackup sink that performs zstd compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs zstd compression.
*/
bbsink *
-bbsink_zstd_new(bbsink *next, int compresslevel)
+bbsink_zstd_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef USE_ZSTD
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,11 +72,13 @@ bbsink_zstd_new(bbsink *next, int compresslevel)
Assert(next != NULL);
- if (compresslevel < 0 || compresslevel > 22)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("zstd compression level %d is out of range",
- compresslevel)));
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
+ compresslevel = 0;
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 22);
+ }
sink = palloc0(sizeof(bbsink_zstd));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_zstd_ops;
diff --git a/src/bin/pg_basebackup/bbstreamer.h b/src/bin/pg_basebackup/bbstreamer.h
index 02d4c05df6..dfa3f77af4 100644
--- a/src/bin/pg_basebackup/bbstreamer.h
+++ b/src/bin/pg_basebackup/bbstreamer.h
@@ -22,6 +22,7 @@
#ifndef BBSTREAMER_H
#define BBSTREAMER_H
+#include "common/backup_compression.h"
#include "lib/stringinfo.h"
#include "pqexpbuffer.h"
@@ -200,17 +201,17 @@ bbstreamer_buffer_until(bbstreamer *streamer, const char **data, int *len,
*/
extern bbstreamer *bbstreamer_plain_writer_new(char *pathname, FILE *file);
extern bbstreamer *bbstreamer_gzip_writer_new(char *pathname, FILE *file,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_extractor_new(const char *basepath,
const char *(*link_map) (const char *),
void (*report_output_file) (const char *));
extern bbstreamer *bbstreamer_gzip_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_lz4_compressor_new(bbstreamer *next,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_lz4_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_zstd_compressor_new(bbstreamer *next,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_zstd_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_tar_parser_new(bbstreamer *next);
extern bbstreamer *bbstreamer_tar_terminator_new(bbstreamer *next);
diff --git a/src/bin/pg_basebackup/bbstreamer_gzip.c b/src/bin/pg_basebackup/bbstreamer_gzip.c
index 894f857103..1979e95639 100644
--- a/src/bin/pg_basebackup/bbstreamer_gzip.c
+++ b/src/bin/pg_basebackup/bbstreamer_gzip.c
@@ -76,7 +76,8 @@ const bbstreamer_ops bbstreamer_gzip_decompressor_ops = {
* closed so that the data may be written there.
*/
bbstreamer *
-bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
+bbstreamer_gzip_writer_new(char *pathname, FILE *file,
+ bc_specification *compress)
{
#ifdef HAVE_LIBZ
bbstreamer_gzip_writer *streamer;
@@ -115,11 +116,11 @@ bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
}
}
- if (gzsetparams(streamer->gzfile, compresslevel,
+ if (gzsetparams(streamer->gzfile, compress->level,
Z_DEFAULT_STRATEGY) != Z_OK)
{
pg_log_error("could not set compression level %d: %s",
- compresslevel, get_gz_error(streamer->gzfile));
+ compress->level, get_gz_error(streamer->gzfile));
exit(1);
}
diff --git a/src/bin/pg_basebackup/bbstreamer_lz4.c b/src/bin/pg_basebackup/bbstreamer_lz4.c
index 810052e4e3..a6ec317e2b 100644
--- a/src/bin/pg_basebackup/bbstreamer_lz4.c
+++ b/src/bin/pg_basebackup/bbstreamer_lz4.c
@@ -67,7 +67,7 @@ const bbstreamer_ops bbstreamer_lz4_decompressor_ops = {
* blocks.
*/
bbstreamer *
-bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
+bbstreamer_lz4_compressor_new(bbstreamer *next, bc_specification *compress)
{
#ifdef USE_LZ4
bbstreamer_lz4_frame *streamer;
@@ -89,7 +89,7 @@ bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
prefs = &streamer->prefs;
memset(prefs, 0, sizeof(LZ4F_preferences_t));
prefs->frameInfo.blockSizeID = LZ4F_max256KB;
- prefs->compressionLevel = compresslevel;
+ prefs->compressionLevel = compress->level;
/*
* Find out the compression bound, it specifies the minimum destination
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index e86749a8fb..caa5edcaf1 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -63,7 +63,7 @@ const bbstreamer_ops bbstreamer_zstd_decompressor_ops = {
* blocks.
*/
bbstreamer *
-bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
+bbstreamer_zstd_compressor_new(bbstreamer *next, bc_specification *compress)
{
#ifdef USE_ZSTD
bbstreamer_zstd_frame *streamer;
@@ -85,7 +85,7 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
/* Initialize stream compression preferences */
ZSTD_CCtx_setParameter(streamer->cctx, ZSTD_c_compressionLevel,
- compresslevel);
+ compress->level);
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 2943d9ec1a..88635ad1b0 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -29,6 +29,7 @@
#include "access/xlog_internal.h"
#include "bbstreamer.h"
+#include "common/backup_compression.h"
#include "common/file_perm.h"
#include "common/file_utils.h"
#include "common/logging.h"
@@ -57,6 +58,7 @@ typedef struct TablespaceList
typedef struct ArchiveStreamState
{
int tablespacenum;
+ bc_specification *compress;
bbstreamer *streamer;
bbstreamer *manifest_inject_streamer;
PQExpBuffer manifest_buffer;
@@ -132,9 +134,6 @@ static bool checksum_failure = false;
static bool showprogress = false;
static bool estimatesize = true;
static int verbose = 0;
-static int compresslevel = 0;
-static WalCompressionMethod compressmethod = COMPRESSION_NONE;
-static CompressionLocation compressloc = COMPRESS_LOCATION_UNSPECIFIED;
static IncludeWal includewal = STREAM_WAL;
static bool fastcheckpoint = false;
static bool writerecoveryconf = false;
@@ -198,7 +197,8 @@ static void progress_report(int tablespacenum, bool force, bool finished);
static bbstreamer *CreateBackupStreamer(char *archive_name, char *spclocation,
bbstreamer **manifest_inject_streamer_p,
bool is_recovery_guc_supported,
- bool expect_unterminated_tarfile);
+ bool expect_unterminated_tarfile,
+ bc_specification *compress);
static void ReceiveArchiveStreamChunk(size_t r, char *copybuf,
void *callback_data);
static char GetCopyDataByte(size_t r, char *copybuf, size_t *cursor);
@@ -207,7 +207,7 @@ static uint64 GetCopyDataUInt64(size_t r, char *copybuf, size_t *cursor);
static void GetCopyDataEnd(size_t r, char *copybuf, size_t cursor);
static void ReportCopyDataParseError(size_t r, char *copybuf);
static void ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
- bool tablespacenum);
+ bool tablespacenum, bc_specification *compress);
static void ReceiveTarCopyChunk(size_t r, char *copybuf, void *callback_data);
static void ReceiveBackupManifest(PGconn *conn);
static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
@@ -215,7 +215,9 @@ static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
static void ReceiveBackupManifestInMemory(PGconn *conn, PQExpBuffer buf);
static void ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
void *callback_data);
-static void BaseBackup(void);
+static void BaseBackup(char *compression_algorithm, char *compression_detail,
+ CompressionLocation compressloc,
+ bc_specification *client_compress);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
bool segment_finished);
@@ -542,7 +544,9 @@ typedef struct
} logstreamer_param;
static int
-LogStreamerMain(logstreamer_param *param)
+LogStreamerMain(logstreamer_param *param,
+ WalCompressionMethod wal_compress_method,
+ int wal_compress_level)
{
StreamCtl stream;
@@ -565,25 +569,14 @@ LogStreamerMain(logstreamer_param *param)
stream.mark_done = true;
stream.partial_suffix = NULL;
stream.replication_slot = replication_slot;
-
if (format == 'p')
stream.walmethod = CreateWalDirectoryMethod(param->xlog,
COMPRESSION_NONE, 0,
stream.do_sync);
- else if (compressloc != COMPRESS_LOCATION_CLIENT)
- stream.walmethod = CreateWalTarMethod(param->xlog,
- COMPRESSION_NONE,
- compresslevel,
- stream.do_sync);
- else if (compressmethod == COMPRESSION_GZIP)
- stream.walmethod = CreateWalTarMethod(param->xlog,
- compressmethod,
- compresslevel,
- stream.do_sync);
else
stream.walmethod = CreateWalTarMethod(param->xlog,
- COMPRESSION_NONE,
- compresslevel,
+ wal_compress_method,
+ wal_compress_level,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
@@ -629,7 +622,9 @@ LogStreamerMain(logstreamer_param *param)
* stream the logfile in parallel with the backups.
*/
static void
-StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
+StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier,
+ WalCompressionMethod wal_compress_method,
+ int wal_compress_level)
{
logstreamer_param *param;
uint32 hi,
@@ -729,7 +724,7 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
int ret;
/* in child process */
- ret = LogStreamerMain(param);
+ ret = LogStreamerMain(param, wal_compress_method, wal_compress_level);
/* temp debugging aid to analyze 019_replslot_limit failures */
if (verbose)
@@ -1004,136 +999,81 @@ parse_max_rate(char *src)
}
/*
- * Utility wrapper to parse the values specified for -Z/--compress.
- * *methodres and *levelres will be optionally filled with values coming
- * from the parsed results.
+ * Basic parsing of a value specified for -Z/--compress.
+ *
+ * We're not concerned here with understanding exactly what behavior the
+ * user wants, but we do need to know whether the user is requesting client
+ * or server side compression or leaving it unspecified, and we need to
+ * separate the name of the compression algorithm from the detail string.
+ *
+ * For instance, if the user writes --compress client-lz4:6, we want to
+ * separate that into (a) client-side compression, (b) algorithm "lz4",
+ * and (c) detail "6". Note, however, that all the client/server prefix is
+ * optional, and so is the detail. The algorithm name is required, unless
+ * the whole string is an integer, in which case we assume "gzip" as the
+ * algorithm and use the integer as the detail.
+ *
+ * We're not concerned with validation at this stage, so if the user writes
+ * --compress client-turkey:sandwhich, the requested algorithm is "turkey"
+ * and the detail string is "sandwhich". We'll sort out whether that's legal
+ * at a later stage.
*/
static void
-parse_compress_options(char *src, WalCompressionMethod *methodres,
- CompressionLocation *locationres, int *levelres)
+parse_compress_options(char *option, char **algorithm, char **detail,
+ CompressionLocation *locationres)
{
char *sep;
- int firstlen;
- char *firstpart;
+ char *endp;
/*
- * clear 'levelres' so that if there are multiple compression options,
- * the last one fully overrides the earlier ones
- */
- *levelres = 0;
-
- /* check if the option is split in two */
- sep = strchr(src, ':');
-
- /*
- * The first part of the option value could be a method name, or just a
- * level value.
- */
- firstlen = (sep != NULL) ? (sep - src) : strlen(src);
- firstpart = pg_malloc(firstlen + 1);
- memcpy(firstpart, src, firstlen);
- firstpart[firstlen] = '\0';
-
- /*
- * Check if the first part of the string matches with a supported
- * compression method.
+ * Check whether the compression specification consists of a bare integer.
+ *
+ * If so, for backward compatibility, assume gzip.
*/
- if (pg_strcasecmp(firstpart, "gzip") == 0)
+ (void) strtol(option, &endp, 10);
+ if (*endp == '\0')
{
- *methodres = COMPRESSION_GZIP;
*locationres = COMPRESS_LOCATION_UNSPECIFIED;
+ *algorithm = pstrdup("gzip");
+ *detail = pstrdup(option);
+ return;
}
- else if (pg_strcasecmp(firstpart, "client-gzip") == 0)
- {
- *methodres = COMPRESSION_GZIP;
- *locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-gzip") == 0)
+
+ /* Strip off any "client-" or "server-" prefix. */
+ if (strncmp(option, "server-", 7) == 0)
{
- *methodres = COMPRESSION_GZIP;
*locationres = COMPRESS_LOCATION_SERVER;
+ option += 7;
}
- else if (pg_strcasecmp(firstpart, "lz4") == 0)
- {
- *methodres = COMPRESSION_LZ4;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
- }
- else if (pg_strcasecmp(firstpart, "client-lz4") == 0)
+ else if (strncmp(option, "client-", 7) == 0)
{
- *methodres = COMPRESSION_LZ4;
*locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-lz4") == 0)
- {
- *methodres = COMPRESSION_LZ4;
- *locationres = COMPRESS_LOCATION_SERVER;
- }
- else if (pg_strcasecmp(firstpart, "zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
- }
- else if (pg_strcasecmp(firstpart, "client-zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_SERVER;
- }
- else if (pg_strcasecmp(firstpart, "none") == 0)
- {
- *methodres = COMPRESSION_NONE;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
+ option += 7;
}
else
- {
- /*
- * It does not match anything known, so check for the
- * backward-compatible case of only an integer where the implied
- * compression method changes depending on the level value.
- */
- if (!option_parse_int(firstpart, "-Z/--compress", 0,
- INT_MAX, levelres))
- exit(1);
-
- *methodres = (*levelres > 0) ?
- COMPRESSION_GZIP : COMPRESSION_NONE;
*locationres = COMPRESS_LOCATION_UNSPECIFIED;
- free(firstpart);
- return;
- }
-
+ /*
+ * Check whether there is a compression detail following the algorithm
+ * name.
+ */
+ sep = strchr(option, ':');
if (sep == NULL)
{
- /*
- * The caller specified a method without a colon separator, so let any
- * subsequent checks assign a default level.
- */
- free(firstpart);
- return;
+ *algorithm = pstrdup(option);
+ *detail = NULL;
}
-
- /* Check the contents after the colon separator. */
- sep++;
- if (*sep == '\0')
+ else
{
- pg_log_error("no compression level defined for method %s", firstpart);
- exit(1);
- }
+ char *alg;
- /*
- * For any of the methods currently supported, the data after the
- * separator can just be an integer.
- */
- if (!option_parse_int(sep, "-Z/--compress", 0, INT_MAX,
- levelres))
- exit(1);
+ alg = palloc((sep - option) + 1);
+ memcpy(alg, option, sep - option);
+ alg[sep - option] = '\0';
- free(firstpart);
+ *algorithm = alg;
+ *detail = pstrdup(sep + 1);
+ }
}
/*
@@ -1200,7 +1140,8 @@ static bbstreamer *
CreateBackupStreamer(char *archive_name, char *spclocation,
bbstreamer **manifest_inject_streamer_p,
bool is_recovery_guc_supported,
- bool expect_unterminated_tarfile)
+ bool expect_unterminated_tarfile,
+ bc_specification *compress)
{
bbstreamer *streamer = NULL;
bbstreamer *manifest_inject_streamer = NULL;
@@ -1316,32 +1257,28 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
archive_file = NULL;
}
- if (compressmethod == COMPRESSION_NONE ||
- compressloc != COMPRESS_LOCATION_CLIENT)
+ if (compress->algorithm == BACKUP_COMPRESSION_NONE)
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- else if (compressmethod == COMPRESSION_GZIP)
+ else if (compress->algorithm == BACKUP_COMPRESSION_GZIP)
{
strlcat(archive_filename, ".gz", sizeof(archive_filename));
streamer = bbstreamer_gzip_writer_new(archive_filename,
- archive_file,
- compresslevel);
+ archive_file, compress);
}
- else if (compressmethod == COMPRESSION_LZ4)
+ else if (compress->algorithm == BACKUP_COMPRESSION_LZ4)
{
strlcat(archive_filename, ".lz4", sizeof(archive_filename));
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- streamer = bbstreamer_lz4_compressor_new(streamer,
- compresslevel);
+ streamer = bbstreamer_lz4_compressor_new(streamer, compress);
}
- else if (compressmethod == COMPRESSION_ZSTD)
+ else if (compress->algorithm == BACKUP_COMPRESSION_ZSTD)
{
strlcat(archive_filename, ".zst", sizeof(archive_filename));
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- streamer = bbstreamer_zstd_compressor_new(streamer,
- compresslevel);
+ streamer = bbstreamer_zstd_compressor_new(streamer, compress);
}
else
{
@@ -1395,13 +1332,13 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
* If the user has requested a server compressed archive along with archive
* extraction at client then we need to decompress it.
*/
- if (format == 'p' && compressloc == COMPRESS_LOCATION_SERVER)
+ if (format == 'p')
{
- if (compressmethod == COMPRESSION_GZIP)
+ if (is_tar_gz)
streamer = bbstreamer_gzip_decompressor_new(streamer);
- else if (compressmethod == COMPRESSION_LZ4)
+ else if (is_tar_lz4)
streamer = bbstreamer_lz4_decompressor_new(streamer);
- else if (compressmethod == COMPRESSION_ZSTD)
+ else if (is_tar_zstd)
streamer = bbstreamer_zstd_decompressor_new(streamer);
}
@@ -1415,13 +1352,14 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
* manifest if present - as a single COPY stream.
*/
static void
-ReceiveArchiveStream(PGconn *conn)
+ReceiveArchiveStream(PGconn *conn, bc_specification *compress)
{
ArchiveStreamState state;
/* Set up initial state. */
memset(&state, 0, sizeof(state));
state.tablespacenum = -1;
+ state.compress = compress;
/* All the real work happens in ReceiveArchiveStreamChunk. */
ReceiveCopyData(conn, ReceiveArchiveStreamChunk, &state);
@@ -1542,7 +1480,8 @@ ReceiveArchiveStreamChunk(size_t r, char *copybuf, void *callback_data)
CreateBackupStreamer(archive_name,
spclocation,
&state->manifest_inject_streamer,
- true, false);
+ true, false,
+ state->compress);
}
break;
}
@@ -1743,7 +1682,7 @@ ReportCopyDataParseError(size_t r, char *copybuf)
*/
static void
ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
- bool tablespacenum)
+ bool tablespacenum, bc_specification *compress)
{
WriteTarState state;
bbstreamer *manifest_inject_streamer;
@@ -1759,7 +1698,8 @@ ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
state.streamer = CreateBackupStreamer(archive_name, spclocation,
&manifest_inject_streamer,
is_recovery_guc_supported,
- expect_unterminated_tarfile);
+ expect_unterminated_tarfile,
+ compress);
state.tablespacenum = tablespacenum;
ReceiveCopyData(conn, ReceiveTarCopyChunk, &state);
progress_update_filename(NULL);
@@ -1902,7 +1842,8 @@ ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
}
static void
-BaseBackup(void)
+BaseBackup(char *compression_algorithm, char *compression_detail,
+ CompressionLocation compressloc, bc_specification *client_compress)
{
PGresult *res;
char *sysidentifier;
@@ -2055,33 +1996,17 @@ BaseBackup(void)
if (compressloc == COMPRESS_LOCATION_SERVER)
{
- char *compressmethodstr = NULL;
-
if (!use_new_option_syntax)
{
pg_log_error("server does not support server-side compression");
exit(1);
}
- switch (compressmethod)
- {
- case COMPRESSION_GZIP:
- compressmethodstr = "gzip";
- break;
- case COMPRESSION_LZ4:
- compressmethodstr = "lz4";
- break;
- case COMPRESSION_ZSTD:
- compressmethodstr = "zstd";
- break;
- default:
- Assert(false);
- break;
- }
AppendStringCommandOption(&buf, use_new_option_syntax,
- "COMPRESSION", compressmethodstr);
- if (compresslevel >= 1) /* not 0 or Z_DEFAULT_COMPRESSION */
- AppendIntegerCommandOption(&buf, use_new_option_syntax,
- "COMPRESSION_LEVEL", compresslevel);
+ "COMPRESSION", compression_algorithm);
+ if (compression_detail != NULL)
+ AppendStringCommandOption(&buf, use_new_option_syntax,
+ "COMPRESSION_DETAIL",
+ compression_detail);
}
if (verbose)
@@ -2207,15 +2132,33 @@ BaseBackup(void)
*/
if (includewal == STREAM_WAL)
{
+ WalCompressionMethod wal_compress_method;
+ int wal_compress_level;
+
if (verbose)
pg_log_info("starting background WAL receiver");
- StartLogStreamer(xlogstart, starttli, sysidentifier);
+
+ if (client_compress->algorithm == BACKUP_COMPRESSION_GZIP)
+ {
+ wal_compress_method = COMPRESSION_GZIP;
+ wal_compress_level =
+ (client_compress->options & BACKUP_COMPRESSION_OPTION_LEVEL)
+ != 0 ? client_compress->level : 0;
+ }
+ else
+ {
+ wal_compress_method = COMPRESSION_NONE;
+ wal_compress_level = 0;
+ }
+
+ StartLogStreamer(xlogstart, starttli, sysidentifier,
+ wal_compress_method, wal_compress_level);
}
if (serverMajor >= 1500)
{
/* Receive a single tar stream with everything. */
- ReceiveArchiveStream(conn);
+ ReceiveArchiveStream(conn, client_compress);
}
else
{
@@ -2244,7 +2187,8 @@ BaseBackup(void)
spclocation = PQgetvalue(res, i, 1);
}
- ReceiveTarFile(conn, archive_name, spclocation, i);
+ ReceiveTarFile(conn, archive_name, spclocation, i,
+ client_compress);
}
/*
@@ -2511,6 +2455,10 @@ main(int argc, char **argv)
int c;
int option_index;
+ char *compression_algorithm = "none";
+ char *compression_detail = NULL;
+ CompressionLocation compressloc = COMPRESS_LOCATION_UNSPECIFIED;
+ bc_specification client_compress;
pg_logging_init(argv[0]);
progname = get_progname(argv[0]);
@@ -2616,17 +2564,13 @@ main(int argc, char **argv)
do_sync = false;
break;
case 'z':
-#ifdef HAVE_LIBZ
- compresslevel = Z_DEFAULT_COMPRESSION;
-#else
- compresslevel = 1; /* will be rejected below */
-#endif
- compressmethod = COMPRESSION_GZIP;
+ compression_algorithm = "gzip";
+ compression_detail = NULL;
compressloc = COMPRESS_LOCATION_UNSPECIFIED;
break;
case 'Z':
- parse_compress_options(optarg, &compressmethod,
- &compressloc, &compresslevel);
+ parse_compress_options(optarg, &compression_algorithm,
+ &compression_detail, &compressloc);
break;
case 'c':
if (pg_strcasecmp(optarg, "fast") == 0)
@@ -2753,12 +2697,11 @@ main(int argc, char **argv)
}
/*
- * If we're compressing the backup and the user has not said where to
- * perform the compression, do it on the client, unless they specified
- * --target, in which case the server is the only choice.
+ * If the user has not specified where to perform backup compression,
+ * default to the client, unless the user specified --target, in which case
+ * the server is the only choice.
*/
- if (compressmethod != COMPRESSION_NONE &&
- compressloc == COMPRESS_LOCATION_UNSPECIFIED)
+ if (compressloc == COMPRESS_LOCATION_UNSPECIFIED)
{
if (backup_target == NULL)
compressloc = COMPRESS_LOCATION_CLIENT;
@@ -2766,6 +2709,40 @@ main(int argc, char **argv)
compressloc = COMPRESS_LOCATION_SERVER;
}
+ /*
+ * If any compression that we're doing is happening on the client side,
+ * we must try to parse the compression algorithm and detail, but if it's
+ * all on the server side, then we're just going to pass through whatever
+ * was requested and let the server decide what to do.
+ */
+ if (compressloc == COMPRESS_LOCATION_CLIENT)
+ {
+ bc_algorithm alg;
+ char *error_detail;
+
+ if (!parse_bc_algorithm(compression_algorithm, &alg))
+ {
+ pg_log_error("unrecognized compression algorithm \"%s\"",
+ compression_algorithm);
+ exit(1);
+ }
+
+ parse_bc_specification(alg, compression_detail, &client_compress);
+ error_detail = validate_bc_specification(&client_compress);
+ if (error_detail != NULL)
+ {
+ pg_log_error("invalid compression specification: %s",
+ error_detail);
+ exit(1);
+ }
+ }
+ else
+ {
+ Assert(compressloc = COMPRESS_LOCATION_SERVER);
+ client_compress.algorithm = BACKUP_COMPRESSION_NONE;
+ client_compress.options = 0;
+ }
+
/*
* Can't perform client-side compression if the backup is not being
* sent to the client.
@@ -2779,9 +2756,10 @@ main(int argc, char **argv)
}
/*
- * Compression doesn't make sense unless tar format is in use.
+ * Client-side compression doesn't make sense unless tar format is in use.
*/
- if (format == 'p' && compressloc == COMPRESS_LOCATION_CLIENT)
+ if (format == 'p' && compressloc == COMPRESS_LOCATION_CLIENT &&
+ client_compress.algorithm != BACKUP_COMPRESSION_NONE)
{
pg_log_error("only tar mode backups can be compressed");
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
@@ -2882,56 +2860,6 @@ main(int argc, char **argv)
}
}
- /* Sanity checks for compression-related options. */
- switch (compressmethod)
- {
- case COMPRESSION_NONE:
- if (compresslevel != 0)
- {
- pg_log_error("cannot use compression level with method %s",
- "none");
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
- progname);
- exit(1);
- }
- break;
- case COMPRESSION_GZIP:
- if (compresslevel > 9)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 9",
- compresslevel, "gzip");
- exit(1);
- }
- if (compressloc == COMPRESS_LOCATION_CLIENT)
- {
-#ifdef HAVE_LIBZ
- if (compresslevel == 0)
- compresslevel = Z_DEFAULT_COMPRESSION;
-#else
- pg_log_error("this build does not support compression with %s",
- "gzip");
- exit(1);
-#endif
- }
- break;
- case COMPRESSION_LZ4:
- if (compresslevel > 12)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 12",
- compresslevel, "lz4");
- exit(1);
- }
- break;
- case COMPRESSION_ZSTD:
- if (compresslevel > 22)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 22",
- compresslevel, "zstd");
- exit(1);
- }
- break;
- }
-
/*
* Sanity checks for progress reporting options.
*/
@@ -3040,7 +2968,8 @@ main(int argc, char **argv)
free(linkloc);
}
- BaseBackup();
+ BaseBackup(compression_algorithm, compression_detail, compressloc,
+ &client_compress);
success = true;
return 0;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index efefe947d9..a05abdadbe 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -42,16 +42,24 @@ $node->command_fails(['pg_basebackup'],
# Sanity checks for options
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none:1' ],
- qr/\Qpg_basebackup: error: cannot use compression level with method none/,
+ qr/\Qcompression algorithm "none" does not accept a compression level/,
'failure if method "none" specified with compression level');
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none+' ],
- qr/\Qpg_basebackup: error: invalid value "none+" for option/,
+ qr/\Qunrecognized compression algorithm "none+"/,
'failure on incorrect separator to define compression level');
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none:' ],
- qr/\Qpg_basebackup: error: no compression level defined for method none/,
+ qr/\Qcompression algorithm "none" does not accept a compression level/,
'failure on missing compression level value');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'gzip:thunk' ],
+ qr/\Qunknown compression option "thunk"/,
+ 'failure on missing compression level value');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'gzip:level=high' ],
+ qr/\Qvalue for compression option "level" must be an integer/,
+ 'failure on non-numeric compression level');
# Some Windows ANSI code pages may reject this filename, in which case we
# quietly proceed without this bit of test coverage.
@@ -89,6 +97,25 @@ print $conf "wal_level = replica\n";
close $conf;
$node->restart;
+# Now that we have a server that supports replication commands, test whether
+# certain scenarios fail on the client side or on the server side.
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'gzip:level=236' ],
+ qr/\Qpg_basebackup: error: invalid compression specification: compression algorithm "gzip" expects a compression level between 1 and 9/,
+ 'client failure on out-of-range compression level');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'server-gzip:level=236' ],
+ qr/\Qpg_basebackup: error: could not initiate base backup: ERROR: invalid compression specification: compression algorithm "gzip" expects a compression level between 1 and 9/,
+ 'server failure on out-of-range compression level');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'client-extrasquishy' ],
+ qr/\Qpg_basebackup: error: unrecognized compression algorithm "extrasquishy"/,
+ 'client failure on invalid compression algorithm');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'server-extrasquishy' ],
+ qr/\Qpg_basebackup: error: could not initiate base backup: ERROR: unrecognized compression algorithm "extrasquishy"/,
+ 'server failure on invalid compression algorithm');
+
# Write some files to test that they are not copied.
foreach my $filename (
qw(backup_label tablespace_map postgresql.auto.conf.tmp
diff --git a/src/common/Makefile b/src/common/Makefile
index 31c0dd366d..f627349835 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -47,6 +47,7 @@ LIBS += $(PTHREAD_LIBS)
OBJS_COMMON = \
archive.o \
+ backup_compression.o \
base64.o \
checksum_helper.o \
config_info.o \
diff --git a/src/common/backup_compression.c b/src/common/backup_compression.c
new file mode 100644
index 0000000000..f943345421
--- /dev/null
+++ b/src/common/backup_compression.c
@@ -0,0 +1,256 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_compression.c
+ *
+ * Shared code for backup compression methods and specifications.
+ *
+ * A compression specification specifies the parameters that should be used
+ * when * performing compression with a specific algorithm. The simplest
+ * possible compression specification is an integer, which sets the
+ * compression level.
+ *
+ * Otherwise, a compression specification is a comma-separated list of items,
+ * each having the form keyword or keyword=value.
+ *
+ * Currently, the only supported keyword is "level".
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/backup_compression.c
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#include "common/backup_compression.h"
+
+static int expect_integer_value(char *keyword, char *value,
+ bc_specification *result);
+
+/*
+ * Look up a compression algorithm by name. Returns true and sets *algorithm
+ * if the name is recognized. Otherwise returns false.
+ */
+bool
+parse_bc_algorithm(char *name, bc_algorithm *algorithm)
+{
+ if (strcmp(name, "none") == 0)
+ *algorithm = BACKUP_COMPRESSION_NONE;
+ else if (strcmp(name, "gzip") == 0)
+ *algorithm = BACKUP_COMPRESSION_GZIP;
+ else if (strcmp(name, "lz4") == 0)
+ *algorithm = BACKUP_COMPRESSION_LZ4;
+ else if (strcmp(name, "zstd") == 0)
+ *algorithm = BACKUP_COMPRESSION_ZSTD;
+ else
+ return false;
+ return true;
+}
+
+/*
+ * Get the human-readable name corresponding to a particular compression
+ * algorithm.
+ */
+char *
+get_bc_algorithm_name(bc_algorithm algorithm)
+{
+ switch (algorithm)
+ {
+ case BACKUP_COMPRESSION_NONE:
+ return "none";
+ case BACKUP_COMPRESSION_GZIP:
+ return "gzip";
+ case BACKUP_COMPRESSION_LZ4:
+ return "lz4";
+ case BACKUP_COMPRESSION_ZSTD:
+ return "zstd";
+ /* no default, to provoke compiler warnings if values are added */
+ }
+ Assert(false);
+}
+
+/*
+ * Parse a compression specification for a specified algorithm.
+ *
+ * See the file header comments for a brief description of what a compression
+ * specification is expected to look like.
+ *
+ * On return, all fields of the result object will be initialized.
+ * In particular, result->parse_error will contain an appropriate error message
+ * if errors were found during parsing, and will be NULL otherwise. However,
+ * even if there's no parse error, the string might not make sense: e.g.
+ * for gzip, level=12 is not sensible, but it does parse OK.
+ *
+ * Use validate_bc_specification() to find out whether a compression
+ * specification is semantically sensible.
+ */
+void
+parse_bc_specification(bc_algorithm algorithm, char *specification,
+ bc_specification *result)
+{
+ int bare_level;
+ char *bare_level_endp;
+
+ /* Initial setup of result object. */
+ result->algorithm = algorithm;
+ result->options = 0;
+ result->level = -1;
+ result->parse_error = NULL;
+
+ /* If there is no specification, we're done already. */
+ if (specification == NULL)
+ return;
+
+ /* As a special case, the specification can be a bare integer. */
+ bare_level = strtol(specification, &bare_level_endp, 10);
+ if (*bare_level_endp == '\0')
+ {
+ result->level = bare_level;
+ result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
+ return;
+ }
+
+ /* Look for comma-separated keyword or keyword=value entries. */
+ while (1)
+ {
+ char *kwstart;
+ char *kwend;
+ char *vstart;
+ char *vend;
+ int kwlen;
+ int vlen;
+ char *keyword;
+ char *value;
+
+ /* Figure start, end, and length of next keyword and any value. */
+ kwstart = kwend = specification;
+ while (*kwend != '\0' && *kwend != ',' && *kwend != '=')
+ ++kwend;
+ kwlen = kwend - kwstart;
+ if (*kwend != '=')
+ {
+ vstart = vend = NULL;
+ vlen = 0;
+ }
+ else
+ {
+ vstart = vend = kwend + 1;
+ while (*vend != '\0' && *vend != ',')
+ ++vend;
+ vlen = vend - vstart;
+ }
+
+ /* Reject empty keyword. */
+ if (kwlen == 0)
+ {
+ result->parse_error =
+ pstrdup("found empty string where a compression option was expected");
+ break;
+ }
+
+ /* Extract keyword and value as separate C strings. */
+ keyword = palloc(kwlen + 1);
+ memcpy(keyword, kwstart, kwlen);
+ keyword[kwlen] = '\0';
+ if (vlen == 0)
+ value = NULL;
+ else
+ {
+ value = palloc(vlen + 1);
+ memcpy(value, vstart, vlen);
+ value[vlen] = '\0';
+ }
+
+ /* Handle whatever keyword we found. */
+ if (strcmp(keyword, "level") == 0)
+ {
+ result->level = expect_integer_value(keyword, value, result);
+ result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
+ }
+ else
+ result->parse_error =
+ psprintf("unknown compression option \"%s\"", keyword);
+
+ /* Release memory, just to be tidy. */
+ pfree(keyword);
+ pfree(value);
+
+ /* If we got an error or have reached the end of the string, stop. */
+ if (result->parse_error != NULL || *kwend == '\0' || *vend == '\0')
+ break;
+
+ /* Advance to next entry and loop around. */
+ specification = vend == NULL ? kwend + 1 : vend + 1;
+ }
+}
+
+/*
+ * Parse 'value' as an integer and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1.
+ */
+static int
+expect_integer_value(char *keyword, char *value, bc_specification *result)
+{
+ int ivalue;
+ char *ivalue_endp;
+
+ ivalue = strtol(value, &ivalue_endp, 10);
+ if (*ivalue_endp != '\0')
+ {
+ result->parse_error =
+ psprintf("value for compression option \"%s\" must be an integer",
+ keyword);
+ return -1;
+ }
+ return ivalue;
+}
+
+/*
+ * Returns NULL if the compression specification string was syntactically
+ * valid and semantically sensible. Otherwise, returns an error message.
+ *
+ * Does not test whether this build of PostgreSQL supports the requested
+ * compression method.
+ */
+char *
+validate_bc_specification(bc_specification *spec)
+{
+ /* If it didn't even parse OK, it's definitely no good. */
+ if (spec->parse_error != NULL)
+ return spec->parse_error;
+
+ /*
+ * If a compression level was specified, check that the algorithm expects
+ * a compression level and that the level is within the legal range for
+ * the algorithm.
+ */
+ if ((spec->options & BACKUP_COMPRESSION_OPTION_LEVEL) != 0)
+ {
+ int min_level = 1;
+ int max_level;
+
+ if (spec->algorithm == BACKUP_COMPRESSION_GZIP)
+ max_level = 9;
+ else if (spec->algorithm == BACKUP_COMPRESSION_LZ4)
+ max_level = 12;
+ else if (spec->algorithm == BACKUP_COMPRESSION_ZSTD)
+ max_level = 22;
+ else
+ return psprintf("compression algorithm \"%s\" does not accept a compression level",
+ get_bc_algorithm_name(spec->algorithm));
+
+ if (spec->level < min_level || spec->level > max_level)
+ return psprintf("compression algorithm \"%s\" expects a compression level between %d and %d",
+ get_bc_algorithm_name(spec->algorithm),
+ min_level, max_level);
+ }
+
+ return NULL;
+}
diff --git a/src/include/common/backup_compression.h b/src/include/common/backup_compression.h
new file mode 100644
index 0000000000..98c1e75a80
--- /dev/null
+++ b/src/include/common/backup_compression.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_compression.h
+ *
+ * Shared definitions for backup compression methods and specifications.
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/backup_compression.h
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef BACKUP_COMPRESSION_H
+#define BACKUP_COMPRESSION_H
+
+typedef enum bc_algorithm
+{
+ BACKUP_COMPRESSION_NONE,
+ BACKUP_COMPRESSION_GZIP,
+ BACKUP_COMPRESSION_LZ4,
+ BACKUP_COMPRESSION_ZSTD
+} bc_algorithm;
+
+#define BACKUP_COMPRESSION_OPTION_LEVEL (1 << 0)
+
+typedef struct bc_specification
+{
+ bc_algorithm algorithm;
+ unsigned options; /* OR of BACKUP_COMPRESSION_OPTION constants */
+ int level;
+ char *parse_error; /* NULL if parsing was OK, else message */
+} bc_specification;
+
+extern bool parse_bc_algorithm(char *name, bc_algorithm *algorithm);
+extern char *get_bc_algorithm_name(bc_algorithm algorithm);
+
+extern void parse_bc_specification(bc_algorithm algorithm,
+ char *specification,
+ bc_specification *result);
+
+extern char *validate_bc_specification(bc_specification *);
+
+#endif
diff --git a/src/include/replication/basebackup_sink.h b/src/include/replication/basebackup_sink.h
index a7f16758a4..654df28576 100644
--- a/src/include/replication/basebackup_sink.h
+++ b/src/include/replication/basebackup_sink.h
@@ -27,6 +27,7 @@
#define BASEBACKUP_SINK_H
#include "access/xlog_internal.h"
+#include "common/backup_compression.h"
#include "nodes/pg_list.h"
/* Forward declarations. */
@@ -283,9 +284,9 @@ extern void bbsink_forward_cleanup(bbsink *sink);
/* Constructors for various types of sinks. */
extern bbsink *bbsink_copystream_new(bool send_to_client);
-extern bbsink *bbsink_gzip_new(bbsink *next, int compresslevel);
-extern bbsink *bbsink_lz4_new(bbsink *next, int compresslevel);
-extern bbsink *bbsink_zstd_new(bbsink *next, int compresslevel);
+extern bbsink *bbsink_gzip_new(bbsink *next, bc_specification *);
+extern bbsink *bbsink_lz4_new(bbsink *next, bc_specification *);
+extern bbsink *bbsink_zstd_new(bbsink *next, bc_specification *);
extern bbsink *bbsink_progress_new(bbsink *next, bool estimate_backup_size);
extern bbsink *bbsink_server_new(bbsink *next, char *pathname);
extern bbsink *bbsink_throttle_new(bbsink *next, uint32 maxrate);
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 441d6ae6bf..de8676d339 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -124,7 +124,7 @@ sub mkvcbuild
}
our @pgcommonallfiles = qw(
- archive.c base64.c checksum_helper.c
+ archive.c backup_compression.c base64.c checksum_helper.c
config_info.c controldata_utils.c d2s.c encnames.c exec.c
f2s.c file_perm.c file_utils.c hashfn.c ip.c jsonapi.c
keywords.c kwlookup.c link-canary.c md5_common.c
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index eaf3e7a8d4..01748cac07 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3788,3 +3788,5 @@ yyscan_t
z_stream
z_streamp
zic_t
+bc_algorithm
+bc_specification
--
2.24.3 (Apple Git-128)
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-17 19:41 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-17 20:29 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
0 siblings, 2 replies; 32+ messages in thread
From: Justin Pryzby @ 2022-03-17 19:41 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers@postgresql.org, tushar <tushar.ahuja@enterprisedb.com>
inline patch 1: doc/src/sgml/protocol.sgml (+2/-0 lines) download inline-1.patch
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9178c779ba..00c593f1af 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2731,14 +2731,24 @@ The commands accepted in replication mode are:
+ <para>
+ For <literal>gzip</literal> the compression level should be an
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
@ 2022-03-17 20:29 ` Robert Haas <robertmhaas@gmail.com>
1 sibling, 0 replies; 32+ messages in thread
From: Robert Haas @ 2022-03-17 20:29 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
Thanks for the review!
I'll address most of these comments later, but quickly for right now...
On Thu, Mar 17, 2022 at 3:41 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> It'd be great if this were re-usable for wal_compression, which I hope in pg16 will
> support at least level=N. And eventually pg_dump. But those clients shouldn't
> accept a client/server prefix. Maybe the way to handle that is for those tools
> to check locationres and reject it if it was specified.
> [...]
> This is confusingly similar to src/include/access/xlog.h:WalCompression.
> I think someone else mentioned this before ?
A couple of people before me have had delusions of grandeur in this
area. We have the WalCompression enum, which has values of the form
COMPRESSION_*, instead of WAL_COMPRESSION_*, as if the WAL were going
to be the only thing that ever got compressed. And pg_dump.h also has
a CompressionAlgorithm enum, with values like COMPR_ALG_*, which isn't
great naming either. Clearly there's some cleanup needed here: if we
can use the same enum for multiple systems, then it can have a name
implying that it's the only game in town, but otherwise both the enum
name and the corresponding value need to use a suitable prefix. I
think that's a job for another patch, probably post-v15. For now I
plan to do the right thing with the new names I'm adding, and leave
the existing names alone. That can be changed in the future, if and
when it seems sensible.
As I said elsewhere, I think the WAL compression stuff is badly
designed and should probably be rewritten completely, maybe to reuse
the bbstreamer stuff. In that case, WalCompressionMethod would
probably go away entirely, making the naming confusion moot, and
picking up zstd and lz4 compression support for free. If that doesn't
happen, we can probably find some way to at least make them share an
enum, but I think that's too hairy to try to clean up right now with
feature freeze pending.
> The server crashes if I send an unknown option - you should hit that in the
> regression tests.
>
> $ src/bin/pg_basebackup/pg_basebackup --wal-method fetch -Ft -D - -h /tmp --no-sync --no-manifest --compress=server-lz4:a |wc -c
> TRAP: FailedAssertion("pointer != NULL", File: "../../../../src/include/utils/memutils.h", Line: 123, PID: 8627)
> postgres: walsender pryzbyj [local] BASE_BACKUP(ExceptionalCondition+0xa0)[0x560b45d7b64b]
> postgres: walsender pryzbyj [local] BASE_BACKUP(pfree+0x5d)[0x560b45dad1ea]
> postgres: walsender pryzbyj [local] BASE_BACKUP(parse_bc_specification+0x154)[0x560b45dc5d4f]
> postgres: walsender pryzbyj [local] BASE_BACKUP(+0x43d56c)[0x560b45bc556c]
> postgres: walsender pryzbyj [local] BASE_BACKUP(SendBaseBackup+0x2d)[0x560b45bc85ca]
> postgres: walsender pryzbyj [local] BASE_BACKUP(exec_replication_command+0x3a2)[0x560b45bdddb2]
> postgres: walsender pryzbyj [local] BASE_BACKUP(PostgresMain+0x6b2)[0x560b45c39131]
> postgres: walsender pryzbyj [local] BASE_BACKUP(+0x40530e)[0x560b45b8d30e]
> postgres: walsender pryzbyj [local] BASE_BACKUP(+0x408572)[0x560b45b90572]
> postgres: walsender pryzbyj [local] BASE_BACKUP(+0x4087b9)[0x560b45b907b9]
> postgres: walsender pryzbyj [local] BASE_BACKUP(PostmasterMain+0x1135)[0x560b45b91d9b]
> postgres: walsender pryzbyj [local] BASE_BACKUP(main+0x229)[0x560b45ad0f78]
That's odd - I thought I had tested that case. Will double-check.
> This is interpreted like client-gzip-1; should multiple specifications of
> compress be prohibited ?
>
> | src/bin/pg_basebackup/pg_basebackup --wal-method fetch -Ft -D - -h /tmp --no-sync --no-manifest --compress=server-lz4 --compress=1
They're not now and haven't been in the past. I think the last one
should just win (as it apparently does, here). We do that in some
places and throw an error in others and I'm not sure if we have a 100%
consistent rule for it, but flipping one location between one behavior
and the other isn't going to make things more consistent overall.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
@ 2022-03-20 19:05 ` Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:11 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
1 sibling, 2 replies; 32+ messages in thread
From: Robert Haas @ 2022-03-20 19:05 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Thu, Mar 17, 2022 at 3:41 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> gzip comma
I think it's fine the way it's written. If we made that change, then
we'd have a comma for gzip and not for the other two algorithms. Also,
I'm just moving that sentence, so any change that there is to be made
here is a job for some other patch.
> alphabetical
Fixed.
> - errmsg("unrecognized compression algorithm: \"%s\"",
> + errmsg("unrecognized compression algorithm \"%s\"",
>
> Most other places seem to say "compression method". So I'd suggest to change
> that here, and in doc/src/sgml/ref/pg_basebackup.sgml.
I'm not sure that's really better, and I don't think this patch is
introducing an altogether novel usage. I think I would probably try to
standardize on algorithm rather than method if I were standardizing
the whole source tree, but I think we can leave that discussion for
another time.
> - if (o_compression_level && !o_compression)
> + if (o_compression_detail && !o_compression)
> ereport(ERROR,
> (errcode(ERRCODE_SYNTAX_ERROR),
> errmsg("compression level requires compression")));
>
> s/level/detail/
Fixed.
> It'd be great if this were re-usable for wal_compression, which I hope in pg16 will
> support at least level=N. And eventually pg_dump. But those clients shouldn't
> accept a client/server prefix. Maybe the way to handle that is for those tools
> to check locationres and reject it if it was specified.
One thing I forgot to mention in my previous response is that I think
the parsing code is actually well set up for this the way I have it.
server- and client- gets parsed off in a different place than we
interpret the rest, which fits well with your observation that other
cases wouldn't have a client or server prefix.
> sp: sandwich
Fixed.
> star
Fixed.
> should be const ?
OK.
>
> + /* As a special case, the specification can be a bare integer. */
> + bare_level = strtol(specification, &bare_level_endp, 10);
>
> Should this call expect_integer_value()?
> See below.
I don't think that would be useful. We have no keyword to pass for the
error message, nor would we use the error message if one got
constructed.
> + result->parse_error =
> + pstrdup("found empty string where a compression option was expected");
>
> Needs to be localized with _() ?
> Also, document that it's pstrdup'd.
Did the latter. The former would need to be fixed in a bunch of places
and while I'm happy to accept an expert opinion on exactly what needs
to be done here, I don't want to try to do it and do it wrong. Better
to let someone with good knowledge of the subject matter patch it up
later than do a crummy job now.
> -1 isn't great, since it's also an integer, and, also a valid compression level
> for zstd (did you see my message about that?). Maybe INT_MIN is ok.
It really doesn't matter. Could just return 42. The client shouldn't
use the value if there's an error.
> +{
> + int ivalue;
> + char *ivalue_endp;
> +
> + ivalue = strtol(value, &ivalue_endp, 10);
>
> Should this also set/check errno ?
> And check if value != ivalue_endp ?
> See strtol(3)
Even after reading the man page for strtol, it's not clear to me that
this is needed. That page represents checking *endptr != '\0' as
sufficient to tell whether an error occurred. Maybe it wouldn't catch
an out of range value, but in practice all of the algorithms we
support now and any we support in the future are going to catch
something clamped to LONG_MIN or LONG_MAX as out of range and display
the correct error message. What's your specific thinking here?
> + unsigned options; /* OR of BACKUP_COMPRESSION_OPTION constants */
>
> Should be "unsigned int" or "bits32" ?
I do not see why either of those would be better.
> The server crashes if I send an unknown option - you should hit that in the
> regression tests.
Turns out I was testing this on the client side but not the server
side. Fixed and added more tests.
v2 attached.
--
Robert Haas
EDB: http://www.enterprisedb.com
Attachments:
v2-0001-Replace-BASE_BACKUP-COMPRESSION_LEVEL-option-with.patch (application/octet-stream, 53.3K) download
From d157b30adadb26ffd4f0262d0bc53119eec6e8e7 Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Sun, 20 Mar 2022 14:48:19 -0400
Subject: [PATCH v2] Replace BASE_BACKUP COMPRESSION_LEVEL option with
COMPRESSION_DETAIL.
There are more compression parameters that can be specified than just
an integer compression level, so rename the new COMPRESSION_LEVEL
option to COMPRESSION_DETAIL before it gets released. Introduce a
flexible syntax for that option to allow arbitrary options to be
specified without needing to adjust the main replication grammar,
and common code to parse it that is shared between the client and
the server.
This commit doesn't actually add any new compression parameters,
so the only user-visible change is that you can now type something
like pg_basebackup --compress gzip:level=5 instead of writing just
pg_basebackup --compress gzip:5. However, it should make it easy to
add new options. If for example gzip starts offering fries, we can
support pg_basebackup --compress gzip:level=5,fries=true for the
benefit of users who want fries with that.
Along the way, this fixes a few things in pg_basebackup so that the
pg_basebackup can be used with a server-side compression algorithm
that pg_basebackup itself does not understand. For example,
pg_basebackup --compress server-lz4 could still succeed even if
only the server and not the client has LZ4 support, provided that
the other options to pg_basebackup don't require the client to
decompress the archive.
Patch by me. Reviewed by Justin Pryzby.
---
doc/src/sgml/protocol.sgml | 18 +-
src/backend/replication/basebackup.c | 62 +--
src/backend/replication/basebackup_gzip.c | 20 +-
src/backend/replication/basebackup_lz4.c | 19 +-
src/backend/replication/basebackup_zstd.c | 19 +-
src/bin/pg_basebackup/bbstreamer.h | 7 +-
src/bin/pg_basebackup/bbstreamer_gzip.c | 7 +-
src/bin/pg_basebackup/bbstreamer_lz4.c | 4 +-
src/bin/pg_basebackup/bbstreamer_zstd.c | 4 +-
src/bin/pg_basebackup/pg_basebackup.c | 405 ++++++++-----------
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 41 +-
src/common/Makefile | 1 +
src/common/backup_compression.c | 258 ++++++++++++
src/include/common/backup_compression.h | 44 ++
src/include/replication/basebackup_sink.h | 7 +-
src/tools/msvc/Mkvcbuild.pm | 2 +-
src/tools/pgindent/typedefs.list | 2 +
17 files changed, 605 insertions(+), 315 deletions(-)
create mode 100644 src/common/backup_compression.c
create mode 100644 src/include/common/backup_compression.h
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9178c779ba..00c593f1af 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2731,14 +2731,24 @@ The commands accepted in replication mode are:
</varlistentry>
<varlistentry>
- <term><literal>COMPRESSION_LEVEL</literal> <replaceable>level</replaceable></term>
+ <term><literal>COMPRESSION_DETAIL</literal> <replaceable>detail</replaceable></term>
<listitem>
<para>
Specifies the compression level to be used. This should only be
used in conjunction with the <literal>COMPRESSION</literal> option.
- For <literal>gzip</literal> the value should be an integer between 1
- and 9, for <literal>lz4</literal> between 1 and 12, and for
- <literal>zstd</literal> it should be between 1 and 22.
+ If the value is an integer, it specifies the compression level.
+ Otherwise, it should be a comma-separated list of items, each of
+ the form <literal>keyword</literal> or
+ <literal>keyword=value</literal>. Currently, the only supported
+ keyword is <literal>level</literal>, which sets the compression
+ level.
+ </para>
+
+ <para>
+ For <literal>gzip</literal> the compression level should be an
+ integer between 1 and 9, for <literal>lz4</literal> an integer
+ between 1 and 12, and for <literal>zstd</literal> an integer
+ between 1 and 22.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index c2aedc14a2..49deead091 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h" /* for pg_start/stop_backup */
+#include "common/backup_compression.h"
#include "common/file_perm.h"
#include "commands/defrem.h"
#include "lib/stringinfo.h"
@@ -54,14 +55,6 @@
*/
#define SINK_BUFFER_LENGTH Max(32768, BLCKSZ)
-typedef enum
-{
- BACKUP_COMPRESSION_NONE,
- BACKUP_COMPRESSION_GZIP,
- BACKUP_COMPRESSION_LZ4,
- BACKUP_COMPRESSION_ZSTD
-} basebackup_compression_type;
-
typedef struct
{
const char *label;
@@ -75,8 +68,8 @@ typedef struct
bool use_copytblspc;
BaseBackupTargetHandle *target_handle;
backup_manifest_option manifest;
- basebackup_compression_type compression;
- int compression_level;
+ bc_algorithm compression;
+ bc_specification compression_specification;
pg_checksum_type manifest_checksum_type;
} basebackup_options;
@@ -713,12 +706,14 @@ parse_basebackup_options(List *options, basebackup_options *opt)
char *target_str = NULL;
char *target_detail_str = NULL;
bool o_compression = false;
- bool o_compression_level = false;
+ bool o_compression_detail = false;
+ char *compression_detail_str = NULL;
MemSet(opt, 0, sizeof(*opt));
opt->manifest = MANIFEST_OPTION_NO;
opt->manifest_checksum_type = CHECKSUM_TYPE_CRC32C;
opt->compression = BACKUP_COMPRESSION_NONE;
+ opt->compression_specification.algorithm = BACKUP_COMPRESSION_NONE;
foreach(lopt, options)
{
@@ -885,29 +880,21 @@ parse_basebackup_options(List *options, basebackup_options *opt)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("duplicate option \"%s\"", defel->defname)));
- if (strcmp(optval, "none") == 0)
- opt->compression = BACKUP_COMPRESSION_NONE;
- else if (strcmp(optval, "gzip") == 0)
- opt->compression = BACKUP_COMPRESSION_GZIP;
- else if (strcmp(optval, "lz4") == 0)
- opt->compression = BACKUP_COMPRESSION_LZ4;
- else if (strcmp(optval, "zstd") == 0)
- opt->compression = BACKUP_COMPRESSION_ZSTD;
- else
+ if (!parse_bc_algorithm(optval, &opt->compression))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("unrecognized compression algorithm: \"%s\"",
+ errmsg("unrecognized compression algorithm \"%s\"",
optval)));
o_compression = true;
}
- else if (strcmp(defel->defname, "compression_level") == 0)
+ else if (strcmp(defel->defname, "compression_detail") == 0)
{
- if (o_compression_level)
+ if (o_compression_detail)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("duplicate option \"%s\"", defel->defname)));
- opt->compression_level = defGetInt32(defel);
- o_compression_level = true;
+ compression_detail_str = defGetString(defel);
+ o_compression_detail = true;
}
else
ereport(ERROR,
@@ -949,10 +936,25 @@ parse_basebackup_options(List *options, basebackup_options *opt)
opt->target_handle =
BaseBackupGetTargetHandle(target_str, target_detail_str);
- if (o_compression_level && !o_compression)
+ if (o_compression_detail && !o_compression)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("compression level requires compression")));
+ errmsg("compression detail requires compression")));
+
+ if (o_compression)
+ {
+ char *error_detail;
+
+ parse_bc_specification(opt->compression, compression_detail_str,
+ &opt->compression_specification);
+ error_detail =
+ validate_bc_specification(&opt->compression_specification);
+ if (error_detail != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("invalid compression specification: %s",
+ error_detail));
+ }
}
@@ -998,11 +1000,11 @@ SendBaseBackup(BaseBackupCmd *cmd)
/* Set up server-side compression, if client requested it */
if (opt.compression == BACKUP_COMPRESSION_GZIP)
- sink = bbsink_gzip_new(sink, opt.compression_level);
+ sink = bbsink_gzip_new(sink, &opt.compression_specification);
else if (opt.compression == BACKUP_COMPRESSION_LZ4)
- sink = bbsink_lz4_new(sink, opt.compression_level);
+ sink = bbsink_lz4_new(sink, &opt.compression_specification);
else if (opt.compression == BACKUP_COMPRESSION_ZSTD)
- sink = bbsink_zstd_new(sink, opt.compression_level);
+ sink = bbsink_zstd_new(sink, &opt.compression_specification);
/* Set up progress reporting. */
sink = bbsink_progress_new(sink, opt.progress);
diff --git a/src/backend/replication/basebackup_gzip.c b/src/backend/replication/basebackup_gzip.c
index b66d3da7a3..703a91ba77 100644
--- a/src/backend/replication/basebackup_gzip.c
+++ b/src/backend/replication/basebackup_gzip.c
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_gzip_ops = {
#endif
/*
- * Create a new basebackup sink that performs gzip compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs gzip compression.
*/
bbsink *
-bbsink_gzip_new(bbsink *next, int compresslevel)
+bbsink_gzip_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef HAVE_LIBZ
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,15 +72,14 @@ bbsink_gzip_new(bbsink *next, int compresslevel)
bbsink_gzip *sink;
Assert(next != NULL);
- Assert(compresslevel >= 0 && compresslevel <= 9);
- if (compresslevel == 0)
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
compresslevel = Z_DEFAULT_COMPRESSION;
- else if (compresslevel < 0 || compresslevel > 9)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("gzip compression level %d is out of range",
- compresslevel)));
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 9);
+ }
sink = palloc0(sizeof(bbsink_gzip));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_gzip_ops;
diff --git a/src/backend/replication/basebackup_lz4.c b/src/backend/replication/basebackup_lz4.c
index d838f723d0..06c161ddc4 100644
--- a/src/backend/replication/basebackup_lz4.c
+++ b/src/backend/replication/basebackup_lz4.c
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_lz4_ops = {
#endif
/*
- * Create a new basebackup sink that performs lz4 compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs lz4 compression.
*/
bbsink *
-bbsink_lz4_new(bbsink *next, int compresslevel)
+bbsink_lz4_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef USE_LZ4
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -72,11 +73,13 @@ bbsink_lz4_new(bbsink *next, int compresslevel)
Assert(next != NULL);
- if (compresslevel < 0 || compresslevel > 12)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("lz4 compression level %d is out of range",
- compresslevel)));
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
+ compresslevel = 0;
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 12);
+ }
sink = palloc0(sizeof(bbsink_lz4));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_lz4_ops;
diff --git a/src/backend/replication/basebackup_zstd.c b/src/backend/replication/basebackup_zstd.c
index c0e2be6e27..96b7985693 100644
--- a/src/backend/replication/basebackup_zstd.c
+++ b/src/backend/replication/basebackup_zstd.c
@@ -55,12 +55,13 @@ const bbsink_ops bbsink_zstd_ops = {
#endif
/*
- * Create a new basebackup sink that performs zstd compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs zstd compression.
*/
bbsink *
-bbsink_zstd_new(bbsink *next, int compresslevel)
+bbsink_zstd_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef USE_ZSTD
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,11 +72,13 @@ bbsink_zstd_new(bbsink *next, int compresslevel)
Assert(next != NULL);
- if (compresslevel < 0 || compresslevel > 22)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("zstd compression level %d is out of range",
- compresslevel)));
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
+ compresslevel = 0;
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 22);
+ }
sink = palloc0(sizeof(bbsink_zstd));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_zstd_ops;
diff --git a/src/bin/pg_basebackup/bbstreamer.h b/src/bin/pg_basebackup/bbstreamer.h
index 02d4c05df6..dfa3f77af4 100644
--- a/src/bin/pg_basebackup/bbstreamer.h
+++ b/src/bin/pg_basebackup/bbstreamer.h
@@ -22,6 +22,7 @@
#ifndef BBSTREAMER_H
#define BBSTREAMER_H
+#include "common/backup_compression.h"
#include "lib/stringinfo.h"
#include "pqexpbuffer.h"
@@ -200,17 +201,17 @@ bbstreamer_buffer_until(bbstreamer *streamer, const char **data, int *len,
*/
extern bbstreamer *bbstreamer_plain_writer_new(char *pathname, FILE *file);
extern bbstreamer *bbstreamer_gzip_writer_new(char *pathname, FILE *file,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_extractor_new(const char *basepath,
const char *(*link_map) (const char *),
void (*report_output_file) (const char *));
extern bbstreamer *bbstreamer_gzip_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_lz4_compressor_new(bbstreamer *next,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_lz4_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_zstd_compressor_new(bbstreamer *next,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_zstd_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_tar_parser_new(bbstreamer *next);
extern bbstreamer *bbstreamer_tar_terminator_new(bbstreamer *next);
diff --git a/src/bin/pg_basebackup/bbstreamer_gzip.c b/src/bin/pg_basebackup/bbstreamer_gzip.c
index 894f857103..1979e95639 100644
--- a/src/bin/pg_basebackup/bbstreamer_gzip.c
+++ b/src/bin/pg_basebackup/bbstreamer_gzip.c
@@ -76,7 +76,8 @@ const bbstreamer_ops bbstreamer_gzip_decompressor_ops = {
* closed so that the data may be written there.
*/
bbstreamer *
-bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
+bbstreamer_gzip_writer_new(char *pathname, FILE *file,
+ bc_specification *compress)
{
#ifdef HAVE_LIBZ
bbstreamer_gzip_writer *streamer;
@@ -115,11 +116,11 @@ bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
}
}
- if (gzsetparams(streamer->gzfile, compresslevel,
+ if (gzsetparams(streamer->gzfile, compress->level,
Z_DEFAULT_STRATEGY) != Z_OK)
{
pg_log_error("could not set compression level %d: %s",
- compresslevel, get_gz_error(streamer->gzfile));
+ compress->level, get_gz_error(streamer->gzfile));
exit(1);
}
diff --git a/src/bin/pg_basebackup/bbstreamer_lz4.c b/src/bin/pg_basebackup/bbstreamer_lz4.c
index 810052e4e3..a6ec317e2b 100644
--- a/src/bin/pg_basebackup/bbstreamer_lz4.c
+++ b/src/bin/pg_basebackup/bbstreamer_lz4.c
@@ -67,7 +67,7 @@ const bbstreamer_ops bbstreamer_lz4_decompressor_ops = {
* blocks.
*/
bbstreamer *
-bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
+bbstreamer_lz4_compressor_new(bbstreamer *next, bc_specification *compress)
{
#ifdef USE_LZ4
bbstreamer_lz4_frame *streamer;
@@ -89,7 +89,7 @@ bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
prefs = &streamer->prefs;
memset(prefs, 0, sizeof(LZ4F_preferences_t));
prefs->frameInfo.blockSizeID = LZ4F_max256KB;
- prefs->compressionLevel = compresslevel;
+ prefs->compressionLevel = compress->level;
/*
* Find out the compression bound, it specifies the minimum destination
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index e86749a8fb..caa5edcaf1 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -63,7 +63,7 @@ const bbstreamer_ops bbstreamer_zstd_decompressor_ops = {
* blocks.
*/
bbstreamer *
-bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
+bbstreamer_zstd_compressor_new(bbstreamer *next, bc_specification *compress)
{
#ifdef USE_ZSTD
bbstreamer_zstd_frame *streamer;
@@ -85,7 +85,7 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
/* Initialize stream compression preferences */
ZSTD_CCtx_setParameter(streamer->cctx, ZSTD_c_compressionLevel,
- compresslevel);
+ compress->level);
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 2943d9ec1a..ec0abdd4d3 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -29,6 +29,7 @@
#include "access/xlog_internal.h"
#include "bbstreamer.h"
+#include "common/backup_compression.h"
#include "common/file_perm.h"
#include "common/file_utils.h"
#include "common/logging.h"
@@ -57,6 +58,7 @@ typedef struct TablespaceList
typedef struct ArchiveStreamState
{
int tablespacenum;
+ bc_specification *compress;
bbstreamer *streamer;
bbstreamer *manifest_inject_streamer;
PQExpBuffer manifest_buffer;
@@ -132,9 +134,6 @@ static bool checksum_failure = false;
static bool showprogress = false;
static bool estimatesize = true;
static int verbose = 0;
-static int compresslevel = 0;
-static WalCompressionMethod compressmethod = COMPRESSION_NONE;
-static CompressionLocation compressloc = COMPRESS_LOCATION_UNSPECIFIED;
static IncludeWal includewal = STREAM_WAL;
static bool fastcheckpoint = false;
static bool writerecoveryconf = false;
@@ -198,7 +197,8 @@ static void progress_report(int tablespacenum, bool force, bool finished);
static bbstreamer *CreateBackupStreamer(char *archive_name, char *spclocation,
bbstreamer **manifest_inject_streamer_p,
bool is_recovery_guc_supported,
- bool expect_unterminated_tarfile);
+ bool expect_unterminated_tarfile,
+ bc_specification *compress);
static void ReceiveArchiveStreamChunk(size_t r, char *copybuf,
void *callback_data);
static char GetCopyDataByte(size_t r, char *copybuf, size_t *cursor);
@@ -207,7 +207,7 @@ static uint64 GetCopyDataUInt64(size_t r, char *copybuf, size_t *cursor);
static void GetCopyDataEnd(size_t r, char *copybuf, size_t cursor);
static void ReportCopyDataParseError(size_t r, char *copybuf);
static void ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
- bool tablespacenum);
+ bool tablespacenum, bc_specification *compress);
static void ReceiveTarCopyChunk(size_t r, char *copybuf, void *callback_data);
static void ReceiveBackupManifest(PGconn *conn);
static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
@@ -215,7 +215,9 @@ static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
static void ReceiveBackupManifestInMemory(PGconn *conn, PQExpBuffer buf);
static void ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
void *callback_data);
-static void BaseBackup(void);
+static void BaseBackup(char *compression_algorithm, char *compression_detail,
+ CompressionLocation compressloc,
+ bc_specification *client_compress);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
bool segment_finished);
@@ -542,7 +544,9 @@ typedef struct
} logstreamer_param;
static int
-LogStreamerMain(logstreamer_param *param)
+LogStreamerMain(logstreamer_param *param,
+ WalCompressionMethod wal_compress_method,
+ int wal_compress_level)
{
StreamCtl stream;
@@ -565,25 +569,14 @@ LogStreamerMain(logstreamer_param *param)
stream.mark_done = true;
stream.partial_suffix = NULL;
stream.replication_slot = replication_slot;
-
if (format == 'p')
stream.walmethod = CreateWalDirectoryMethod(param->xlog,
COMPRESSION_NONE, 0,
stream.do_sync);
- else if (compressloc != COMPRESS_LOCATION_CLIENT)
- stream.walmethod = CreateWalTarMethod(param->xlog,
- COMPRESSION_NONE,
- compresslevel,
- stream.do_sync);
- else if (compressmethod == COMPRESSION_GZIP)
- stream.walmethod = CreateWalTarMethod(param->xlog,
- compressmethod,
- compresslevel,
- stream.do_sync);
else
stream.walmethod = CreateWalTarMethod(param->xlog,
- COMPRESSION_NONE,
- compresslevel,
+ wal_compress_method,
+ wal_compress_level,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
@@ -629,7 +622,9 @@ LogStreamerMain(logstreamer_param *param)
* stream the logfile in parallel with the backups.
*/
static void
-StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
+StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier,
+ WalCompressionMethod wal_compress_method,
+ int wal_compress_level)
{
logstreamer_param *param;
uint32 hi,
@@ -729,7 +724,7 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
int ret;
/* in child process */
- ret = LogStreamerMain(param);
+ ret = LogStreamerMain(param, wal_compress_method, wal_compress_level);
/* temp debugging aid to analyze 019_replslot_limit failures */
if (verbose)
@@ -1004,136 +999,81 @@ parse_max_rate(char *src)
}
/*
- * Utility wrapper to parse the values specified for -Z/--compress.
- * *methodres and *levelres will be optionally filled with values coming
- * from the parsed results.
+ * Basic parsing of a value specified for -Z/--compress.
+ *
+ * We're not concerned here with understanding exactly what behavior the
+ * user wants, but we do need to know whether the user is requesting client
+ * or server side compression or leaving it unspecified, and we need to
+ * separate the name of the compression algorithm from the detail string.
+ *
+ * For instance, if the user writes --compress client-lz4:6, we want to
+ * separate that into (a) client-side compression, (b) algorithm "lz4",
+ * and (c) detail "6". Note, however, that all the client/server prefix is
+ * optional, and so is the detail. The algorithm name is required, unless
+ * the whole string is an integer, in which case we assume "gzip" as the
+ * algorithm and use the integer as the detail.
+ *
+ * We're not concerned with validation at this stage, so if the user writes
+ * --compress client-turkey:sandwich, the requested algorithm is "turkey"
+ * and the detail string is "sandwich". We'll sort out whether that's legal
+ * at a later stage.
*/
static void
-parse_compress_options(char *src, WalCompressionMethod *methodres,
- CompressionLocation *locationres, int *levelres)
+parse_compress_options(char *option, char **algorithm, char **detail,
+ CompressionLocation *locationres)
{
char *sep;
- int firstlen;
- char *firstpart;
+ char *endp;
/*
- * clear 'levelres' so that if there are multiple compression options,
- * the last one fully overrides the earlier ones
- */
- *levelres = 0;
-
- /* check if the option is split in two */
- sep = strchr(src, ':');
-
- /*
- * The first part of the option value could be a method name, or just a
- * level value.
- */
- firstlen = (sep != NULL) ? (sep - src) : strlen(src);
- firstpart = pg_malloc(firstlen + 1);
- memcpy(firstpart, src, firstlen);
- firstpart[firstlen] = '\0';
-
- /*
- * Check if the first part of the string matches with a supported
- * compression method.
+ * Check whether the compression specification consists of a bare integer.
+ *
+ * If so, for backward compatibility, assume gzip.
*/
- if (pg_strcasecmp(firstpart, "gzip") == 0)
+ (void) strtol(option, &endp, 10);
+ if (*endp == '\0')
{
- *methodres = COMPRESSION_GZIP;
*locationres = COMPRESS_LOCATION_UNSPECIFIED;
+ *algorithm = pstrdup("gzip");
+ *detail = pstrdup(option);
+ return;
}
- else if (pg_strcasecmp(firstpart, "client-gzip") == 0)
- {
- *methodres = COMPRESSION_GZIP;
- *locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-gzip") == 0)
+
+ /* Strip off any "client-" or "server-" prefix. */
+ if (strncmp(option, "server-", 7) == 0)
{
- *methodres = COMPRESSION_GZIP;
*locationres = COMPRESS_LOCATION_SERVER;
+ option += 7;
}
- else if (pg_strcasecmp(firstpart, "lz4") == 0)
- {
- *methodres = COMPRESSION_LZ4;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
- }
- else if (pg_strcasecmp(firstpart, "client-lz4") == 0)
+ else if (strncmp(option, "client-", 7) == 0)
{
- *methodres = COMPRESSION_LZ4;
*locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-lz4") == 0)
- {
- *methodres = COMPRESSION_LZ4;
- *locationres = COMPRESS_LOCATION_SERVER;
- }
- else if (pg_strcasecmp(firstpart, "zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
- }
- else if (pg_strcasecmp(firstpart, "client-zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_SERVER;
- }
- else if (pg_strcasecmp(firstpart, "none") == 0)
- {
- *methodres = COMPRESSION_NONE;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
+ option += 7;
}
else
- {
- /*
- * It does not match anything known, so check for the
- * backward-compatible case of only an integer where the implied
- * compression method changes depending on the level value.
- */
- if (!option_parse_int(firstpart, "-Z/--compress", 0,
- INT_MAX, levelres))
- exit(1);
-
- *methodres = (*levelres > 0) ?
- COMPRESSION_GZIP : COMPRESSION_NONE;
*locationres = COMPRESS_LOCATION_UNSPECIFIED;
- free(firstpart);
- return;
- }
-
+ /*
+ * Check whether there is a compression detail following the algorithm
+ * name.
+ */
+ sep = strchr(option, ':');
if (sep == NULL)
{
- /*
- * The caller specified a method without a colon separator, so let any
- * subsequent checks assign a default level.
- */
- free(firstpart);
- return;
+ *algorithm = pstrdup(option);
+ *detail = NULL;
}
-
- /* Check the contents after the colon separator. */
- sep++;
- if (*sep == '\0')
+ else
{
- pg_log_error("no compression level defined for method %s", firstpart);
- exit(1);
- }
+ char *alg;
- /*
- * For any of the methods currently supported, the data after the
- * separator can just be an integer.
- */
- if (!option_parse_int(sep, "-Z/--compress", 0, INT_MAX,
- levelres))
- exit(1);
+ alg = palloc((sep - option) + 1);
+ memcpy(alg, option, sep - option);
+ alg[sep - option] = '\0';
- free(firstpart);
+ *algorithm = alg;
+ *detail = pstrdup(sep + 1);
+ }
}
/*
@@ -1200,7 +1140,8 @@ static bbstreamer *
CreateBackupStreamer(char *archive_name, char *spclocation,
bbstreamer **manifest_inject_streamer_p,
bool is_recovery_guc_supported,
- bool expect_unterminated_tarfile)
+ bool expect_unterminated_tarfile,
+ bc_specification *compress)
{
bbstreamer *streamer = NULL;
bbstreamer *manifest_inject_streamer = NULL;
@@ -1316,32 +1257,28 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
archive_file = NULL;
}
- if (compressmethod == COMPRESSION_NONE ||
- compressloc != COMPRESS_LOCATION_CLIENT)
+ if (compress->algorithm == BACKUP_COMPRESSION_NONE)
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- else if (compressmethod == COMPRESSION_GZIP)
+ else if (compress->algorithm == BACKUP_COMPRESSION_GZIP)
{
strlcat(archive_filename, ".gz", sizeof(archive_filename));
streamer = bbstreamer_gzip_writer_new(archive_filename,
- archive_file,
- compresslevel);
+ archive_file, compress);
}
- else if (compressmethod == COMPRESSION_LZ4)
+ else if (compress->algorithm == BACKUP_COMPRESSION_LZ4)
{
strlcat(archive_filename, ".lz4", sizeof(archive_filename));
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- streamer = bbstreamer_lz4_compressor_new(streamer,
- compresslevel);
+ streamer = bbstreamer_lz4_compressor_new(streamer, compress);
}
- else if (compressmethod == COMPRESSION_ZSTD)
+ else if (compress->algorithm == BACKUP_COMPRESSION_ZSTD)
{
strlcat(archive_filename, ".zst", sizeof(archive_filename));
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- streamer = bbstreamer_zstd_compressor_new(streamer,
- compresslevel);
+ streamer = bbstreamer_zstd_compressor_new(streamer, compress);
}
else
{
@@ -1395,13 +1332,13 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
* If the user has requested a server compressed archive along with archive
* extraction at client then we need to decompress it.
*/
- if (format == 'p' && compressloc == COMPRESS_LOCATION_SERVER)
+ if (format == 'p')
{
- if (compressmethod == COMPRESSION_GZIP)
+ if (is_tar_gz)
streamer = bbstreamer_gzip_decompressor_new(streamer);
- else if (compressmethod == COMPRESSION_LZ4)
+ else if (is_tar_lz4)
streamer = bbstreamer_lz4_decompressor_new(streamer);
- else if (compressmethod == COMPRESSION_ZSTD)
+ else if (is_tar_zstd)
streamer = bbstreamer_zstd_decompressor_new(streamer);
}
@@ -1415,13 +1352,14 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
* manifest if present - as a single COPY stream.
*/
static void
-ReceiveArchiveStream(PGconn *conn)
+ReceiveArchiveStream(PGconn *conn, bc_specification *compress)
{
ArchiveStreamState state;
/* Set up initial state. */
memset(&state, 0, sizeof(state));
state.tablespacenum = -1;
+ state.compress = compress;
/* All the real work happens in ReceiveArchiveStreamChunk. */
ReceiveCopyData(conn, ReceiveArchiveStreamChunk, &state);
@@ -1542,7 +1480,8 @@ ReceiveArchiveStreamChunk(size_t r, char *copybuf, void *callback_data)
CreateBackupStreamer(archive_name,
spclocation,
&state->manifest_inject_streamer,
- true, false);
+ true, false,
+ state->compress);
}
break;
}
@@ -1743,7 +1682,7 @@ ReportCopyDataParseError(size_t r, char *copybuf)
*/
static void
ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
- bool tablespacenum)
+ bool tablespacenum, bc_specification *compress)
{
WriteTarState state;
bbstreamer *manifest_inject_streamer;
@@ -1759,7 +1698,8 @@ ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
state.streamer = CreateBackupStreamer(archive_name, spclocation,
&manifest_inject_streamer,
is_recovery_guc_supported,
- expect_unterminated_tarfile);
+ expect_unterminated_tarfile,
+ compress);
state.tablespacenum = tablespacenum;
ReceiveCopyData(conn, ReceiveTarCopyChunk, &state);
progress_update_filename(NULL);
@@ -1902,7 +1842,8 @@ ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
}
static void
-BaseBackup(void)
+BaseBackup(char *compression_algorithm, char *compression_detail,
+ CompressionLocation compressloc, bc_specification *client_compress)
{
PGresult *res;
char *sysidentifier;
@@ -2055,33 +1996,17 @@ BaseBackup(void)
if (compressloc == COMPRESS_LOCATION_SERVER)
{
- char *compressmethodstr = NULL;
-
if (!use_new_option_syntax)
{
pg_log_error("server does not support server-side compression");
exit(1);
}
- switch (compressmethod)
- {
- case COMPRESSION_GZIP:
- compressmethodstr = "gzip";
- break;
- case COMPRESSION_LZ4:
- compressmethodstr = "lz4";
- break;
- case COMPRESSION_ZSTD:
- compressmethodstr = "zstd";
- break;
- default:
- Assert(false);
- break;
- }
AppendStringCommandOption(&buf, use_new_option_syntax,
- "COMPRESSION", compressmethodstr);
- if (compresslevel >= 1) /* not 0 or Z_DEFAULT_COMPRESSION */
- AppendIntegerCommandOption(&buf, use_new_option_syntax,
- "COMPRESSION_LEVEL", compresslevel);
+ "COMPRESSION", compression_algorithm);
+ if (compression_detail != NULL)
+ AppendStringCommandOption(&buf, use_new_option_syntax,
+ "COMPRESSION_DETAIL",
+ compression_detail);
}
if (verbose)
@@ -2207,15 +2132,33 @@ BaseBackup(void)
*/
if (includewal == STREAM_WAL)
{
+ WalCompressionMethod wal_compress_method;
+ int wal_compress_level;
+
if (verbose)
pg_log_info("starting background WAL receiver");
- StartLogStreamer(xlogstart, starttli, sysidentifier);
+
+ if (client_compress->algorithm == BACKUP_COMPRESSION_GZIP)
+ {
+ wal_compress_method = COMPRESSION_GZIP;
+ wal_compress_level =
+ (client_compress->options & BACKUP_COMPRESSION_OPTION_LEVEL)
+ != 0 ? client_compress->level : 0;
+ }
+ else
+ {
+ wal_compress_method = COMPRESSION_NONE;
+ wal_compress_level = 0;
+ }
+
+ StartLogStreamer(xlogstart, starttli, sysidentifier,
+ wal_compress_method, wal_compress_level);
}
if (serverMajor >= 1500)
{
/* Receive a single tar stream with everything. */
- ReceiveArchiveStream(conn);
+ ReceiveArchiveStream(conn, client_compress);
}
else
{
@@ -2244,7 +2187,8 @@ BaseBackup(void)
spclocation = PQgetvalue(res, i, 1);
}
- ReceiveTarFile(conn, archive_name, spclocation, i);
+ ReceiveTarFile(conn, archive_name, spclocation, i,
+ client_compress);
}
/*
@@ -2511,6 +2455,10 @@ main(int argc, char **argv)
int c;
int option_index;
+ char *compression_algorithm = "none";
+ char *compression_detail = NULL;
+ CompressionLocation compressloc = COMPRESS_LOCATION_UNSPECIFIED;
+ bc_specification client_compress;
pg_logging_init(argv[0]);
progname = get_progname(argv[0]);
@@ -2616,17 +2564,13 @@ main(int argc, char **argv)
do_sync = false;
break;
case 'z':
-#ifdef HAVE_LIBZ
- compresslevel = Z_DEFAULT_COMPRESSION;
-#else
- compresslevel = 1; /* will be rejected below */
-#endif
- compressmethod = COMPRESSION_GZIP;
+ compression_algorithm = "gzip";
+ compression_detail = NULL;
compressloc = COMPRESS_LOCATION_UNSPECIFIED;
break;
case 'Z':
- parse_compress_options(optarg, &compressmethod,
- &compressloc, &compresslevel);
+ parse_compress_options(optarg, &compression_algorithm,
+ &compression_detail, &compressloc);
break;
case 'c':
if (pg_strcasecmp(optarg, "fast") == 0)
@@ -2753,12 +2697,11 @@ main(int argc, char **argv)
}
/*
- * If we're compressing the backup and the user has not said where to
- * perform the compression, do it on the client, unless they specified
- * --target, in which case the server is the only choice.
+ * If the user has not specified where to perform backup compression,
+ * default to the client, unless the user specified --target, in which case
+ * the server is the only choice.
*/
- if (compressmethod != COMPRESSION_NONE &&
- compressloc == COMPRESS_LOCATION_UNSPECIFIED)
+ if (compressloc == COMPRESS_LOCATION_UNSPECIFIED)
{
if (backup_target == NULL)
compressloc = COMPRESS_LOCATION_CLIENT;
@@ -2766,6 +2709,40 @@ main(int argc, char **argv)
compressloc = COMPRESS_LOCATION_SERVER;
}
+ /*
+ * If any compression that we're doing is happening on the client side,
+ * we must try to parse the compression algorithm and detail, but if it's
+ * all on the server side, then we're just going to pass through whatever
+ * was requested and let the server decide what to do.
+ */
+ if (compressloc == COMPRESS_LOCATION_CLIENT)
+ {
+ bc_algorithm alg;
+ char *error_detail;
+
+ if (!parse_bc_algorithm(compression_algorithm, &alg))
+ {
+ pg_log_error("unrecognized compression algorithm \"%s\"",
+ compression_algorithm);
+ exit(1);
+ }
+
+ parse_bc_specification(alg, compression_detail, &client_compress);
+ error_detail = validate_bc_specification(&client_compress);
+ if (error_detail != NULL)
+ {
+ pg_log_error("invalid compression specification: %s",
+ error_detail);
+ exit(1);
+ }
+ }
+ else
+ {
+ Assert(compressloc = COMPRESS_LOCATION_SERVER);
+ client_compress.algorithm = BACKUP_COMPRESSION_NONE;
+ client_compress.options = 0;
+ }
+
/*
* Can't perform client-side compression if the backup is not being
* sent to the client.
@@ -2779,9 +2756,10 @@ main(int argc, char **argv)
}
/*
- * Compression doesn't make sense unless tar format is in use.
+ * Client-side compression doesn't make sense unless tar format is in use.
*/
- if (format == 'p' && compressloc == COMPRESS_LOCATION_CLIENT)
+ if (format == 'p' && compressloc == COMPRESS_LOCATION_CLIENT &&
+ client_compress.algorithm != BACKUP_COMPRESSION_NONE)
{
pg_log_error("only tar mode backups can be compressed");
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
@@ -2882,56 +2860,6 @@ main(int argc, char **argv)
}
}
- /* Sanity checks for compression-related options. */
- switch (compressmethod)
- {
- case COMPRESSION_NONE:
- if (compresslevel != 0)
- {
- pg_log_error("cannot use compression level with method %s",
- "none");
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
- progname);
- exit(1);
- }
- break;
- case COMPRESSION_GZIP:
- if (compresslevel > 9)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 9",
- compresslevel, "gzip");
- exit(1);
- }
- if (compressloc == COMPRESS_LOCATION_CLIENT)
- {
-#ifdef HAVE_LIBZ
- if (compresslevel == 0)
- compresslevel = Z_DEFAULT_COMPRESSION;
-#else
- pg_log_error("this build does not support compression with %s",
- "gzip");
- exit(1);
-#endif
- }
- break;
- case COMPRESSION_LZ4:
- if (compresslevel > 12)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 12",
- compresslevel, "lz4");
- exit(1);
- }
- break;
- case COMPRESSION_ZSTD:
- if (compresslevel > 22)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 22",
- compresslevel, "zstd");
- exit(1);
- }
- break;
- }
-
/*
* Sanity checks for progress reporting options.
*/
@@ -3040,7 +2968,8 @@ main(int argc, char **argv)
free(linkloc);
}
- BaseBackup();
+ BaseBackup(compression_algorithm, compression_detail, compressloc,
+ &client_compress);
success = true;
return 0;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index efefe947d9..ea5ef3152c 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -42,15 +42,15 @@ $node->command_fails(['pg_basebackup'],
# Sanity checks for options
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none:1' ],
- qr/\Qpg_basebackup: error: cannot use compression level with method none/,
+ qr/\Qcompression algorithm "none" does not accept a compression level/,
'failure if method "none" specified with compression level');
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none+' ],
- qr/\Qpg_basebackup: error: invalid value "none+" for option/,
+ qr/\Qunrecognized compression algorithm "none+"/,
'failure on incorrect separator to define compression level');
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none:' ],
- qr/\Qpg_basebackup: error: no compression level defined for method none/,
+ qr/\Qcompression algorithm "none" does not accept a compression level/,
'failure on missing compression level value');
# Some Windows ANSI code pages may reject this filename, in which case we
@@ -89,6 +89,41 @@ print $conf "wal_level = replica\n";
close $conf;
$node->restart;
+# Now that we have a server that supports replication commands, test whether
+# certain scenarios fail on the client side or on the server side.
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'gzip:level=236' ],
+ qr/\Qpg_basebackup: error: invalid compression specification: compression algorithm "gzip" expects a compression level between 1 and 9/,
+ 'client failure on out-of-range compression level');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'server-gzip:level=236' ],
+ qr/\Qpg_basebackup: error: could not initiate base backup: ERROR: invalid compression specification: compression algorithm "gzip" expects a compression level between 1 and 9/,
+ 'server failure on out-of-range compression level');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'client-extrasquishy' ],
+ qr/\Qpg_basebackup: error: unrecognized compression algorithm "extrasquishy"/,
+ 'client failure on invalid compression algorithm');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'server-extrasquishy' ],
+ qr/\Qpg_basebackup: error: could not initiate base backup: ERROR: unrecognized compression algorithm "extrasquishy"/,
+ 'server failure on invalid compression algorithm');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'gzip:thunk' ],
+ qr/\Qpg_basebackup: error: invalid compression specification: unknown compression option "thunk"/,
+ 'failure on missing compression level value');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'gzip:level=high' ],
+ qr/\Qpg_basebackup: error: invalid compression specification: value for compression option "level" must be an integer/,
+ 'failure on non-numeric compression level');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'server-gzip:thunk' ],
+ qr/\Qpg_basebackup: error: could not initiate base backup: ERROR: invalid compression specification: unknown compression option "thunk"/,
+ 'failure on missing compression level value');
+$node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'server-gzip:level=high' ],
+ qr/\Qpg_basebackup: error: could not initiate base backup: ERROR: invalid compression specification: value for compression option "level" must be an integer/,
+ 'failure on non-numeric compression level');
+
# Write some files to test that they are not copied.
foreach my $filename (
qw(backup_label tablespace_map postgresql.auto.conf.tmp
diff --git a/src/common/Makefile b/src/common/Makefile
index 31c0dd366d..f627349835 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -47,6 +47,7 @@ LIBS += $(PTHREAD_LIBS)
OBJS_COMMON = \
archive.o \
+ backup_compression.o \
base64.o \
checksum_helper.o \
config_info.o \
diff --git a/src/common/backup_compression.c b/src/common/backup_compression.c
new file mode 100644
index 0000000000..bf426d6b7e
--- /dev/null
+++ b/src/common/backup_compression.c
@@ -0,0 +1,258 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_compression.c
+ *
+ * Shared code for backup compression methods and specifications.
+ *
+ * A compression specification specifies the parameters that should be used
+ * when performing compression with a specific algorithm. The simplest
+ * possible compression specification is an integer, which sets the
+ * compression level.
+ *
+ * Otherwise, a compression specification is a comma-separated list of items,
+ * each having the form keyword or keyword=value.
+ *
+ * Currently, the only supported keyword is "level".
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/backup_compression.c
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#include "common/backup_compression.h"
+
+static int expect_integer_value(char *keyword, char *value,
+ bc_specification *result);
+
+/*
+ * Look up a compression algorithm by name. Returns true and sets *algorithm
+ * if the name is recognized. Otherwise returns false.
+ */
+bool
+parse_bc_algorithm(char *name, bc_algorithm *algorithm)
+{
+ if (strcmp(name, "none") == 0)
+ *algorithm = BACKUP_COMPRESSION_NONE;
+ else if (strcmp(name, "gzip") == 0)
+ *algorithm = BACKUP_COMPRESSION_GZIP;
+ else if (strcmp(name, "lz4") == 0)
+ *algorithm = BACKUP_COMPRESSION_LZ4;
+ else if (strcmp(name, "zstd") == 0)
+ *algorithm = BACKUP_COMPRESSION_ZSTD;
+ else
+ return false;
+ return true;
+}
+
+/*
+ * Get the human-readable name corresponding to a particular compression
+ * algorithm.
+ */
+const char *
+get_bc_algorithm_name(bc_algorithm algorithm)
+{
+ switch (algorithm)
+ {
+ case BACKUP_COMPRESSION_NONE:
+ return "none";
+ case BACKUP_COMPRESSION_GZIP:
+ return "gzip";
+ case BACKUP_COMPRESSION_LZ4:
+ return "lz4";
+ case BACKUP_COMPRESSION_ZSTD:
+ return "zstd";
+ /* no default, to provoke compiler warnings if values are added */
+ }
+ Assert(false);
+}
+
+/*
+ * Parse a compression specification for a specified algorithm.
+ *
+ * See the file header comments for a brief description of what a compression
+ * specification is expected to look like.
+ *
+ * On return, all fields of the result object will be initialized.
+ * In particular, result->parse_error will be NULL if no errors occurred
+ * during parsing, and will otherwise contain a an appropriate error message.
+ * The caller may free this error message string using pfree, if desired.
+ * Note, however, even if there's no parse error, the string might not make
+ * sense: e.g. for gzip, level=12 is not sensible, but it does parse OK.
+ *
+ * Use validate_bc_specification() to find out whether a compression
+ * specification is semantically sensible.
+ */
+void
+parse_bc_specification(bc_algorithm algorithm, char *specification,
+ bc_specification *result)
+{
+ int bare_level;
+ char *bare_level_endp;
+
+ /* Initial setup of result object. */
+ result->algorithm = algorithm;
+ result->options = 0;
+ result->level = -1;
+ result->parse_error = NULL;
+
+ /* If there is no specification, we're done already. */
+ if (specification == NULL)
+ return;
+
+ /* As a special case, the specification can be a bare integer. */
+ bare_level = strtol(specification, &bare_level_endp, 10);
+ if (*bare_level_endp == '\0')
+ {
+ result->level = bare_level;
+ result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
+ return;
+ }
+
+ /* Look for comma-separated keyword or keyword=value entries. */
+ while (1)
+ {
+ char *kwstart;
+ char *kwend;
+ char *vstart;
+ char *vend;
+ int kwlen;
+ int vlen;
+ char *keyword;
+ char *value;
+
+ /* Figure start, end, and length of next keyword and any value. */
+ kwstart = kwend = specification;
+ while (*kwend != '\0' && *kwend != ',' && *kwend != '=')
+ ++kwend;
+ kwlen = kwend - kwstart;
+ if (*kwend != '=')
+ {
+ vstart = vend = NULL;
+ vlen = 0;
+ }
+ else
+ {
+ vstart = vend = kwend + 1;
+ while (*vend != '\0' && *vend != ',')
+ ++vend;
+ vlen = vend - vstart;
+ }
+
+ /* Reject empty keyword. */
+ if (kwlen == 0)
+ {
+ result->parse_error =
+ pstrdup("found empty string where a compression option was expected");
+ break;
+ }
+
+ /* Extract keyword and value as separate C strings. */
+ keyword = palloc(kwlen + 1);
+ memcpy(keyword, kwstart, kwlen);
+ keyword[kwlen] = '\0';
+ if (vlen == 0)
+ value = NULL;
+ else
+ {
+ value = palloc(vlen + 1);
+ memcpy(value, vstart, vlen);
+ value[vlen] = '\0';
+ }
+
+ /* Handle whatever keyword we found. */
+ if (strcmp(keyword, "level") == 0)
+ {
+ result->level = expect_integer_value(keyword, value, result);
+ result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
+ }
+ else
+ result->parse_error =
+ psprintf("unknown compression option \"%s\"", keyword);
+
+ /* Release memory, just to be tidy. */
+ pfree(keyword);
+ if (value != NULL)
+ pfree(value);
+
+ /* If we got an error or have reached the end of the string, stop. */
+ if (result->parse_error != NULL || *kwend == '\0' || *vend == '\0')
+ break;
+
+ /* Advance to next entry and loop around. */
+ specification = vend == NULL ? kwend + 1 : vend + 1;
+ }
+}
+
+/*
+ * Parse 'value' as an integer and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1.
+ */
+static int
+expect_integer_value(char *keyword, char *value, bc_specification *result)
+{
+ int ivalue;
+ char *ivalue_endp;
+
+ ivalue = strtol(value, &ivalue_endp, 10);
+ if (*ivalue_endp != '\0')
+ {
+ result->parse_error =
+ psprintf("value for compression option \"%s\" must be an integer",
+ keyword);
+ return -1;
+ }
+ return ivalue;
+}
+
+/*
+ * Returns NULL if the compression specification string was syntactically
+ * valid and semantically sensible. Otherwise, returns an error message.
+ *
+ * Does not test whether this build of PostgreSQL supports the requested
+ * compression method.
+ */
+char *
+validate_bc_specification(bc_specification *spec)
+{
+ /* If it didn't even parse OK, it's definitely no good. */
+ if (spec->parse_error != NULL)
+ return spec->parse_error;
+
+ /*
+ * If a compression level was specified, check that the algorithm expects
+ * a compression level and that the level is within the legal range for
+ * the algorithm.
+ */
+ if ((spec->options & BACKUP_COMPRESSION_OPTION_LEVEL) != 0)
+ {
+ int min_level = 1;
+ int max_level;
+
+ if (spec->algorithm == BACKUP_COMPRESSION_GZIP)
+ max_level = 9;
+ else if (spec->algorithm == BACKUP_COMPRESSION_LZ4)
+ max_level = 12;
+ else if (spec->algorithm == BACKUP_COMPRESSION_ZSTD)
+ max_level = 22;
+ else
+ return psprintf("compression algorithm \"%s\" does not accept a compression level",
+ get_bc_algorithm_name(spec->algorithm));
+
+ if (spec->level < min_level || spec->level > max_level)
+ return psprintf("compression algorithm \"%s\" expects a compression level between %d and %d",
+ get_bc_algorithm_name(spec->algorithm),
+ min_level, max_level);
+ }
+
+ return NULL;
+}
diff --git a/src/include/common/backup_compression.h b/src/include/common/backup_compression.h
new file mode 100644
index 0000000000..0565cbc657
--- /dev/null
+++ b/src/include/common/backup_compression.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_compression.h
+ *
+ * Shared definitions for backup compression methods and specifications.
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/backup_compression.h
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef BACKUP_COMPRESSION_H
+#define BACKUP_COMPRESSION_H
+
+typedef enum bc_algorithm
+{
+ BACKUP_COMPRESSION_NONE,
+ BACKUP_COMPRESSION_GZIP,
+ BACKUP_COMPRESSION_LZ4,
+ BACKUP_COMPRESSION_ZSTD
+} bc_algorithm;
+
+#define BACKUP_COMPRESSION_OPTION_LEVEL (1 << 0)
+
+typedef struct bc_specification
+{
+ bc_algorithm algorithm;
+ unsigned options; /* OR of BACKUP_COMPRESSION_OPTION constants */
+ int level;
+ char *parse_error; /* NULL if parsing was OK, else message */
+} bc_specification;
+
+extern bool parse_bc_algorithm(char *name, bc_algorithm *algorithm);
+extern const char *get_bc_algorithm_name(bc_algorithm algorithm);
+
+extern void parse_bc_specification(bc_algorithm algorithm,
+ char *specification,
+ bc_specification *result);
+
+extern char *validate_bc_specification(bc_specification *);
+
+#endif
diff --git a/src/include/replication/basebackup_sink.h b/src/include/replication/basebackup_sink.h
index a7f16758a4..654df28576 100644
--- a/src/include/replication/basebackup_sink.h
+++ b/src/include/replication/basebackup_sink.h
@@ -27,6 +27,7 @@
#define BASEBACKUP_SINK_H
#include "access/xlog_internal.h"
+#include "common/backup_compression.h"
#include "nodes/pg_list.h"
/* Forward declarations. */
@@ -283,9 +284,9 @@ extern void bbsink_forward_cleanup(bbsink *sink);
/* Constructors for various types of sinks. */
extern bbsink *bbsink_copystream_new(bool send_to_client);
-extern bbsink *bbsink_gzip_new(bbsink *next, int compresslevel);
-extern bbsink *bbsink_lz4_new(bbsink *next, int compresslevel);
-extern bbsink *bbsink_zstd_new(bbsink *next, int compresslevel);
+extern bbsink *bbsink_gzip_new(bbsink *next, bc_specification *);
+extern bbsink *bbsink_lz4_new(bbsink *next, bc_specification *);
+extern bbsink *bbsink_zstd_new(bbsink *next, bc_specification *);
extern bbsink *bbsink_progress_new(bbsink *next, bool estimate_backup_size);
extern bbsink *bbsink_server_new(bbsink *next, char *pathname);
extern bbsink *bbsink_throttle_new(bbsink *next, uint32 maxrate);
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 441d6ae6bf..de8676d339 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -124,7 +124,7 @@ sub mkvcbuild
}
our @pgcommonallfiles = qw(
- archive.c base64.c checksum_helper.c
+ archive.c backup_compression.c base64.c checksum_helper.c
config_info.c controldata_utils.c d2s.c encnames.c exec.c
f2s.c file_perm.c file_utils.c hashfn.c ip.c jsonapi.c
keywords.c kwlookup.c link-canary.c md5_common.c
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index eaf3e7a8d4..01748cac07 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3788,3 +3788,5 @@ yyscan_t
z_stream
z_streamp
zic_t
+bc_algorithm
+bc_specification
--
2.24.3 (Apple Git-128)
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-20 19:11 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-21 01:24 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 32+ messages in thread
From: Tom Lane @ 2022-03-20 19:11 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
Robert Haas <robertmhaas@gmail.com> writes:
>> Should this also set/check errno ?
>> And check if value != ivalue_endp ?
>> See strtol(3)
> Even after reading the man page for strtol, it's not clear to me that
> this is needed. That page represents checking *endptr != '\0' as
> sufficient to tell whether an error occurred.
I'm not sure whose man page you looked at, but the POSIX standard [1]
has a pretty clear opinion about this:
Since 0, {LONG_MIN} or {LLONG_MIN}, and {LONG_MAX} or {LLONG_MAX} are
returned on error and are also valid returns on success, an
application wishing to check for error situations should set errno to
0, then call strtol() or strtoll(), then check errno.
Checking *endptr != '\0' is for detecting whether there is trailing
garbage after the number; which may be an error case or not as you
choose, but it's a different matter.
regards, tom lane
[1] https://pubs.opengroup.org/onlinepubs/9699919799/
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:11 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-21 01:24 ` Robert Haas <robertmhaas@gmail.com>
2022-03-21 01:32 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 32+ messages in thread
From: Robert Haas @ 2022-03-21 01:24 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Sun, Mar 20, 2022 at 3:11 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Even after reading the man page for strtol, it's not clear to me that
> > this is needed. That page represents checking *endptr != '\0' as
> > sufficient to tell whether an error occurred.
>
> I'm not sure whose man page you looked at, but the POSIX standard [1]
> has a pretty clear opinion about this:
>
> Since 0, {LONG_MIN} or {LLONG_MIN}, and {LONG_MAX} or {LLONG_MAX} are
> returned on error and are also valid returns on success, an
> application wishing to check for error situations should set errno to
> 0, then call strtol() or strtoll(), then check errno.
>
> Checking *endptr != '\0' is for detecting whether there is trailing
> garbage after the number; which may be an error case or not as you
> choose, but it's a different matter.
I think I'm guilty of verbal inexactitude here but not bad coding.
Checking for *endptr != '\0', as I did, is not sufficient to detect
"whether an error occurred," as I alleged. But, in the part of my
response you didn't quote, I believe I made it clear that I only need
to detect garbage, not out-of-range values. And I think *endptr !=
'\0' will do that.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:11 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-21 01:24 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-21 01:32 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-21 02:03 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 32+ messages in thread
From: Tom Lane @ 2022-03-21 01:32 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
Robert Haas <robertmhaas@gmail.com> writes:
> I think I'm guilty of verbal inexactitude here but not bad coding.
> Checking for *endptr != '\0', as I did, is not sufficient to detect
> "whether an error occurred," as I alleged. But, in the part of my
> response you didn't quote, I believe I made it clear that I only need
> to detect garbage, not out-of-range values. And I think *endptr !=
> '\0' will do that.
Hmm ... do you consider an empty string to be valid input?
regards, tom lane
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:11 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-21 01:24 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 01:32 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-21 02:03 ` Robert Haas <robertmhaas@gmail.com>
0 siblings, 0 replies; 32+ messages in thread
From: Robert Haas @ 2022-03-21 02:03 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Sun, Mar 20, 2022 at 9:32 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > I think I'm guilty of verbal inexactitude here but not bad coding.
> > Checking for *endptr != '\0', as I did, is not sufficient to detect
> > "whether an error occurred," as I alleged. But, in the part of my
> > response you didn't quote, I believe I made it clear that I only need
> > to detect garbage, not out-of-range values. And I think *endptr !=
> > '\0' will do that.
>
> Hmm ... do you consider an empty string to be valid input?
No, and I thought I had checked properly for that condition before
reaching the point in the code where I call strtol(), but it turns out
I have not, which I guess is what Justin has been trying to tell me
for a few emails now.
I'll send an updated patch tomorrow after looking this all over more carefully.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-20 19:40 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 32+ messages in thread
From: Justin Pryzby @ 2022-03-20 19:40 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers@postgresql.org, tushar <tushar.ahuja@enterprisedb.com>
On Sun, Mar 20, 2022 at 03:05:28PM -0400, Robert Haas wrote:
> On Thu, Mar 17, 2022 at 3:41 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> > - errmsg("unrecognized compression algorithm: \"%s\"",
> > + errmsg("unrecognized compression algorithm \"%s\"",
> >
> > Most other places seem to say "compression method". So I'd suggest to change
> > that here, and in doc/src/sgml/ref/pg_basebackup.sgml.
>
> I'm not sure that's really better, and I don't think this patch is
> introducing an altogether novel usage. I think I would probably try to
> standardize on algorithm rather than method if I were standardizing
> the whole source tree, but I think we can leave that discussion for
> another time.
The user-facing docs are already standardized using "compression method", with
2 exceptions, of which one is contrib/ and the other is what I'm suggesting to
make consistent here.
$ git grep 'compression algorithm' doc
doc/src/sgml/pgcrypto.sgml: Which compression algorithm to use. Only available if
doc/src/sgml/ref/pg_basebackup.sgml: compression algorithm is selected, or if server-side compression
> > + result->parse_error =
> > + pstrdup("found empty string where a compression option was expected");
> >
> > Needs to be localized with _() ?
> > Also, document that it's pstrdup'd.
>
> Did the latter. The former would need to be fixed in a bunch of places
> and while I'm happy to accept an expert opinion on exactly what needs
> to be done here, I don't want to try to do it and do it wrong. Better
> to let someone with good knowledge of the subject matter patch it up
> later than do a crummy job now.
I believe it just needs _("foo")
See git grep '= _('
I mentioned another issue off-list:
pg_basebackup.c:2741:10: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
2741 | Assert(compressloc = COMPRESS_LOCATION_SERVER);
| ^~~~~~~~~~~
pg_basebackup.c:2741:3: note: in expansion of macro ‘Assert’
2741 | Assert(compressloc = COMPRESS_LOCATION_SERVER);
This crashes the server using your v2 patch:
src/bin/pg_basebackup/pg_basebackup --wal-method fetch -Ft -D - -h /tmp --no-sync --no-manifest --compress=server-zstd:level, |wc -c
I wonder whether the syntax should really use both ":" and ",".
Maybe ":" isn't needed at all.
This patch also needs to update the other user-facing docs.
typo: contain a an
--
Justin
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
@ 2022-03-21 01:38 ` Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 32+ messages in thread
From: Robert Haas @ 2022-03-21 01:38 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Sun, Mar 20, 2022 at 3:40 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> The user-facing docs are already standardized using "compression method", with
> 2 exceptions, of which one is contrib/ and the other is what I'm suggesting to
> make consistent here.
>
> $ git grep 'compression algorithm' doc
> doc/src/sgml/pgcrypto.sgml: Which compression algorithm to use. Only available if
> doc/src/sgml/ref/pg_basebackup.sgml: compression algorithm is selected, or if server-side compression
Well, if you just count the number of occurrences of each string in
the documentation, sure. But all of the ones that are talking about a
compression method seem to have to do with configurable TOAST
compression, and the fact that the documentation for that feature is
more extensive than for the pre-existing feature that refers to a
compression algorithm does not, at least in my view, turn it into a
project standard from which no deviation is permitted.
> > Did the latter. The former would need to be fixed in a bunch of places
> > and while I'm happy to accept an expert opinion on exactly what needs
> > to be done here, I don't want to try to do it and do it wrong. Better
> > to let someone with good knowledge of the subject matter patch it up
> > later than do a crummy job now.
>
> I believe it just needs _("foo")
> See git grep '= _('
Hmm. Maybe.
> I mentioned another issue off-list:
> pg_basebackup.c:2741:10: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
> 2741 | Assert(compressloc = COMPRESS_LOCATION_SERVER);
> | ^~~~~~~~~~~
> pg_basebackup.c:2741:3: note: in expansion of macro ‘Assert’
> 2741 | Assert(compressloc = COMPRESS_LOCATION_SERVER);
>
> This crashes the server using your v2 patch:
>
> src/bin/pg_basebackup/pg_basebackup --wal-method fetch -Ft -D - -h /tmp --no-sync --no-manifest --compress=server-zstd:level, |wc -c
Well that's unfortunate. Will fix.
> I wonder whether the syntax should really use both ":" and ",".
> Maybe ":" isn't needed at all.
I don't think we should treat the compression method name in the same
way as a compression algorithm option.
> This patch also needs to update the other user-facing docs.
Which ones exactly?
> typo: contain a an
OK, will fix.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-21 13:18 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 32+ messages in thread
From: Justin Pryzby @ 2022-03-21 13:18 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers@postgresql.org, tushar <tushar.ahuja@enterprisedb.com>
On Sun, Mar 20, 2022 at 09:38:44PM -0400, Robert Haas wrote:
> > This patch also needs to update the other user-facing docs.
>
> Which ones exactly?
I mean pg_basebackup -Z
-Z level
-Z [{client|server}-]method[:level]
--compress=level
--compress=[{client|server}-]method[:level]
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
@ 2022-03-21 16:57 ` Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 32+ messages in thread
From: Robert Haas @ 2022-03-21 16:57 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 21, 2022 at 9:18 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> On Sun, Mar 20, 2022 at 09:38:44PM -0400, Robert Haas wrote:
> > > This patch also needs to update the other user-facing docs.
> >
> > Which ones exactly?
>
> I mean pg_basebackup -Z
>
> -Z level
> -Z [{client|server}-]method[:level]
> --compress=level
> --compress=[{client|server}-]method[:level]
Ah, right. Thanks.
Here's v3. I have updated that section of the documentation. I also
went and added a bunch more test cases for validation of compression
detail strings, many inspired by your examples, and fixed all the bugs
that I found in the process. I think the crashes you complained about
are now fixed, but please let me know if I have missed any. I also
added _() calls as you suggested. I searched for the "contain a an"
typo that you mentioned but was not able to find it. Can you give me a
more specific pointer?
I looked a little bit more at the compression method vs. compression
algorithm thing. I agree that there is some inconsistency in
terminology here, but I'm still not sure that we are well-served by
trying to make it totally uniform, especially if we pick the word
"method" as the standard rather than "algorithm". In my opinion,
"method" is less specific than "algorithm". If someone asks me to
choose a compression algorithm, I know that I should give an answer
like "lz4" or "zstd". If they ask me to pick a compression method, I'm
not quite sure whether they want that kind of answer or whether they
want something more detailed, like "use lz4 with compression level 3
and a 1MB block size". After all, that is (at least according to my
understanding of how English works) a perfectly valid answer to the
question "what method should I use to compress this data?" -- but not
to the question "what algorithm should I use to compress this data?".
The latter can ONLY be properly answered by saying something like
"lz4". And I think that's really the root of my hesitation to make the
kinds of changes you want here. If it's just a question of specifying
a compression algorithm and a level, I don't think using the name
"method" for the algorithm is going to be too bad. But as we enrich
the system with multiple compression algorithms each of which may have
multiple and different parameters, I think the whole thing becomes
murkier and the need for precision in language goes up.
Now that is of course an arguable position and you're welcome to
disagree with it, but I think that's part of why I'm hesitating.
Another part of it, at least for me, is that complete uniformity is
not always a positive. I suppose all of us have had the experience at
some point of reading a manual that says something like "to activate
the boil water function, press and release the 'boil water' button"
and rolled our eyes at how useless it was. It's important to me that
we don't fall into that trap. We clearly don't want to go ballistic
and have random inconsistencies in language for no reason, but at the
same time, it's not useful to tell people that METHOD should be
replaced with a compression method and LEVEL with a compression level.
I mean, if you end up saying something like that interspersed with
non-obvious information, that is OK, and I don't want to overstate the
point I'm trying to make. But it seems to me that if there's a little
variation in phrasing and we end up saying that METHOD means the
compression algorithm or that ALGORITHM means the compression method
or whatever, that can actually make things more clear. Here again it's
debatable: how much variation in phraseology is helpful, and at what
point does it just start to seem inconsistent? Well, everyone may have
their own opinion.
I'm not trying to pretend that this patch (or the existing code base)
gets this all right. But I do think that, to the extent that we have a
considered position on what to do here, we can make that change later,
perhaps even after getting some user feedback on what does and does
not make sense to other people. And I also think that what we end up
doing here may well end up being more nuanced than a blanket
search-and-replace. I'm not saying we couldn't make a blanket
search-and-replace. I just don't see it as necessarily creating value,
or being all that closely connected to the goal of this patch, which
is to quickly clean up a forward-compatibility risk before we hit
feature freeze.
Thanks,
--
Robert Haas
EDB: http://www.enterprisedb.com
Attachments:
v3-0001-Replace-BASE_BACKUP-COMPRESSION_LEVEL-option-with.patch (application/octet-stream, 56.7K) download
From 6a27b676550e0b58c49a3d852294ac6e66a6b169 Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Mon, 21 Mar 2022 12:09:11 -0400
Subject: [PATCH v3] Replace BASE_BACKUP COMPRESSION_LEVEL option with
COMPRESSION_DETAIL.
There are more compression parameters that can be specified than just
an integer compression level, so rename the new COMPRESSION_LEVEL
option to COMPRESSION_DETAIL before it gets released. Introduce a
flexible syntax for that option to allow arbitrary options to be
specified without needing to adjust the main replication grammar,
and common code to parse it that is shared between the client and
the server.
This commit doesn't actually add any new compression parameters,
so the only user-visible change is that you can now type something
like pg_basebackup --compress gzip:level=5 instead of writing just
pg_basebackup --compress gzip:5. However, it should make it easy to
add new options. If for example gzip starts offering fries, we can
support pg_basebackup --compress gzip:level=5,fries=true for the
benefit of users who want fries with that.
Along the way, this fixes a few things in pg_basebackup so that the
pg_basebackup can be used with a server-side compression algorithm
that pg_basebackup itself does not understand. For example,
pg_basebackup --compress server-lz4 could still succeed even if
only the server and not the client has LZ4 support, provided that
the other options to pg_basebackup don't require the client to
decompress the archive.
Patch by me. Reviewed by Justin Pryzby.
---
doc/src/sgml/protocol.sgml | 18 +-
doc/src/sgml/ref/pg_basebackup.sgml | 25 +-
src/backend/replication/basebackup.c | 62 +--
src/backend/replication/basebackup_gzip.c | 20 +-
src/backend/replication/basebackup_lz4.c | 19 +-
src/backend/replication/basebackup_zstd.c | 19 +-
src/bin/pg_basebackup/bbstreamer.h | 7 +-
src/bin/pg_basebackup/bbstreamer_gzip.c | 7 +-
src/bin/pg_basebackup/bbstreamer_lz4.c | 4 +-
src/bin/pg_basebackup/bbstreamer_zstd.c | 4 +-
src/bin/pg_basebackup/pg_basebackup.c | 409 ++++++++-----------
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 72 +++-
src/common/Makefile | 1 +
src/common/backup_compression.c | 269 ++++++++++++
src/include/common/backup_compression.h | 44 ++
src/include/replication/basebackup_sink.h | 7 +-
src/tools/msvc/Mkvcbuild.pm | 2 +-
src/tools/pgindent/typedefs.list | 2 +
18 files changed, 662 insertions(+), 329 deletions(-)
create mode 100644 src/common/backup_compression.c
create mode 100644 src/include/common/backup_compression.h
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9178c779ba..00c593f1af 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2731,14 +2731,24 @@ The commands accepted in replication mode are:
</varlistentry>
<varlistentry>
- <term><literal>COMPRESSION_LEVEL</literal> <replaceable>level</replaceable></term>
+ <term><literal>COMPRESSION_DETAIL</literal> <replaceable>detail</replaceable></term>
<listitem>
<para>
Specifies the compression level to be used. This should only be
used in conjunction with the <literal>COMPRESSION</literal> option.
- For <literal>gzip</literal> the value should be an integer between 1
- and 9, for <literal>lz4</literal> between 1 and 12, and for
- <literal>zstd</literal> it should be between 1 and 22.
+ If the value is an integer, it specifies the compression level.
+ Otherwise, it should be a comma-separated list of items, each of
+ the form <literal>keyword</literal> or
+ <literal>keyword=value</literal>. Currently, the only supported
+ keyword is <literal>level</literal>, which sets the compression
+ level.
+ </para>
+
+ <para>
+ For <literal>gzip</literal> the compression level should be an
+ integer between 1 and 9, for <literal>lz4</literal> an integer
+ between 1 and 12, and for <literal>zstd</literal> an integer
+ between 1 and 22.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 4a630b59b7..46d7f15e54 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -399,9 +399,9 @@ PostgreSQL documentation
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
- <term><option>-Z [{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>level</replaceable>]</term>
+ <term><option>-Z [{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>detail</replaceable>]</term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
- <term><option>--compress=[{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>level</replaceable>]</term>
+ <term><option>--compress=[{client|server}-]<replaceable class="parameter">method[:<replaceable>detail</replaceable>]</replaceable></option></term>
<listitem>
<para>
Requests compression of the backup. If <literal>client</literal> or
@@ -419,13 +419,20 @@ PostgreSQL documentation
<para>
The compression method can be set to <literal>gzip</literal>,
<literal>lz4</literal>, <literal>zstd</literal>, or
- <literal>none</literal> for no compression. A compression level can
- optionally be specified, by appending the level number after a colon
- (<literal>:</literal>). If no level is specified, the default
- compression level will be used. If only a level is specified without
- mentioning an algorithm, <literal>gzip</literal> compression will be
- used if the level is greater than 0, and no compression will be used if
- the level is 0.
+ <literal>none</literal> for no compression. A compression detail
+ string can optionally be specified. If the detail string is an
+ integer, it specifies the compression level. Otherwise, it should be
+ a comma-separated list of items, each of the form
+ <literal>keyword</literal> or <literal>keyword=value</literal>.
+ Currently, the only supported keyword is <literal>level</literal>,
+ which sets the compression level.
+ </para>
+ <para>
+ If no compression level is specified, the default compression level
+ will be used. If only a level is specified without mentioning an
+ algorithm, <literal>gzip</literal> compression will be used if the
+ level is greater than 0, and no compression will be used if the level
+ is 0.
</para>
<para>
When the tar format is used with <literal>gzip</literal>,
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index c2aedc14a2..49deead091 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h" /* for pg_start/stop_backup */
+#include "common/backup_compression.h"
#include "common/file_perm.h"
#include "commands/defrem.h"
#include "lib/stringinfo.h"
@@ -54,14 +55,6 @@
*/
#define SINK_BUFFER_LENGTH Max(32768, BLCKSZ)
-typedef enum
-{
- BACKUP_COMPRESSION_NONE,
- BACKUP_COMPRESSION_GZIP,
- BACKUP_COMPRESSION_LZ4,
- BACKUP_COMPRESSION_ZSTD
-} basebackup_compression_type;
-
typedef struct
{
const char *label;
@@ -75,8 +68,8 @@ typedef struct
bool use_copytblspc;
BaseBackupTargetHandle *target_handle;
backup_manifest_option manifest;
- basebackup_compression_type compression;
- int compression_level;
+ bc_algorithm compression;
+ bc_specification compression_specification;
pg_checksum_type manifest_checksum_type;
} basebackup_options;
@@ -713,12 +706,14 @@ parse_basebackup_options(List *options, basebackup_options *opt)
char *target_str = NULL;
char *target_detail_str = NULL;
bool o_compression = false;
- bool o_compression_level = false;
+ bool o_compression_detail = false;
+ char *compression_detail_str = NULL;
MemSet(opt, 0, sizeof(*opt));
opt->manifest = MANIFEST_OPTION_NO;
opt->manifest_checksum_type = CHECKSUM_TYPE_CRC32C;
opt->compression = BACKUP_COMPRESSION_NONE;
+ opt->compression_specification.algorithm = BACKUP_COMPRESSION_NONE;
foreach(lopt, options)
{
@@ -885,29 +880,21 @@ parse_basebackup_options(List *options, basebackup_options *opt)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("duplicate option \"%s\"", defel->defname)));
- if (strcmp(optval, "none") == 0)
- opt->compression = BACKUP_COMPRESSION_NONE;
- else if (strcmp(optval, "gzip") == 0)
- opt->compression = BACKUP_COMPRESSION_GZIP;
- else if (strcmp(optval, "lz4") == 0)
- opt->compression = BACKUP_COMPRESSION_LZ4;
- else if (strcmp(optval, "zstd") == 0)
- opt->compression = BACKUP_COMPRESSION_ZSTD;
- else
+ if (!parse_bc_algorithm(optval, &opt->compression))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("unrecognized compression algorithm: \"%s\"",
+ errmsg("unrecognized compression algorithm \"%s\"",
optval)));
o_compression = true;
}
- else if (strcmp(defel->defname, "compression_level") == 0)
+ else if (strcmp(defel->defname, "compression_detail") == 0)
{
- if (o_compression_level)
+ if (o_compression_detail)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("duplicate option \"%s\"", defel->defname)));
- opt->compression_level = defGetInt32(defel);
- o_compression_level = true;
+ compression_detail_str = defGetString(defel);
+ o_compression_detail = true;
}
else
ereport(ERROR,
@@ -949,10 +936,25 @@ parse_basebackup_options(List *options, basebackup_options *opt)
opt->target_handle =
BaseBackupGetTargetHandle(target_str, target_detail_str);
- if (o_compression_level && !o_compression)
+ if (o_compression_detail && !o_compression)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("compression level requires compression")));
+ errmsg("compression detail requires compression")));
+
+ if (o_compression)
+ {
+ char *error_detail;
+
+ parse_bc_specification(opt->compression, compression_detail_str,
+ &opt->compression_specification);
+ error_detail =
+ validate_bc_specification(&opt->compression_specification);
+ if (error_detail != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("invalid compression specification: %s",
+ error_detail));
+ }
}
@@ -998,11 +1000,11 @@ SendBaseBackup(BaseBackupCmd *cmd)
/* Set up server-side compression, if client requested it */
if (opt.compression == BACKUP_COMPRESSION_GZIP)
- sink = bbsink_gzip_new(sink, opt.compression_level);
+ sink = bbsink_gzip_new(sink, &opt.compression_specification);
else if (opt.compression == BACKUP_COMPRESSION_LZ4)
- sink = bbsink_lz4_new(sink, opt.compression_level);
+ sink = bbsink_lz4_new(sink, &opt.compression_specification);
else if (opt.compression == BACKUP_COMPRESSION_ZSTD)
- sink = bbsink_zstd_new(sink, opt.compression_level);
+ sink = bbsink_zstd_new(sink, &opt.compression_specification);
/* Set up progress reporting. */
sink = bbsink_progress_new(sink, opt.progress);
diff --git a/src/backend/replication/basebackup_gzip.c b/src/backend/replication/basebackup_gzip.c
index b66d3da7a3..703a91ba77 100644
--- a/src/backend/replication/basebackup_gzip.c
+++ b/src/backend/replication/basebackup_gzip.c
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_gzip_ops = {
#endif
/*
- * Create a new basebackup sink that performs gzip compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs gzip compression.
*/
bbsink *
-bbsink_gzip_new(bbsink *next, int compresslevel)
+bbsink_gzip_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef HAVE_LIBZ
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,15 +72,14 @@ bbsink_gzip_new(bbsink *next, int compresslevel)
bbsink_gzip *sink;
Assert(next != NULL);
- Assert(compresslevel >= 0 && compresslevel <= 9);
- if (compresslevel == 0)
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
compresslevel = Z_DEFAULT_COMPRESSION;
- else if (compresslevel < 0 || compresslevel > 9)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("gzip compression level %d is out of range",
- compresslevel)));
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 9);
+ }
sink = palloc0(sizeof(bbsink_gzip));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_gzip_ops;
diff --git a/src/backend/replication/basebackup_lz4.c b/src/backend/replication/basebackup_lz4.c
index d838f723d0..06c161ddc4 100644
--- a/src/backend/replication/basebackup_lz4.c
+++ b/src/backend/replication/basebackup_lz4.c
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_lz4_ops = {
#endif
/*
- * Create a new basebackup sink that performs lz4 compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs lz4 compression.
*/
bbsink *
-bbsink_lz4_new(bbsink *next, int compresslevel)
+bbsink_lz4_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef USE_LZ4
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -72,11 +73,13 @@ bbsink_lz4_new(bbsink *next, int compresslevel)
Assert(next != NULL);
- if (compresslevel < 0 || compresslevel > 12)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("lz4 compression level %d is out of range",
- compresslevel)));
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
+ compresslevel = 0;
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 12);
+ }
sink = palloc0(sizeof(bbsink_lz4));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_lz4_ops;
diff --git a/src/backend/replication/basebackup_zstd.c b/src/backend/replication/basebackup_zstd.c
index c0e2be6e27..96b7985693 100644
--- a/src/backend/replication/basebackup_zstd.c
+++ b/src/backend/replication/basebackup_zstd.c
@@ -55,12 +55,13 @@ const bbsink_ops bbsink_zstd_ops = {
#endif
/*
- * Create a new basebackup sink that performs zstd compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs zstd compression.
*/
bbsink *
-bbsink_zstd_new(bbsink *next, int compresslevel)
+bbsink_zstd_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef USE_ZSTD
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,11 +72,13 @@ bbsink_zstd_new(bbsink *next, int compresslevel)
Assert(next != NULL);
- if (compresslevel < 0 || compresslevel > 22)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("zstd compression level %d is out of range",
- compresslevel)));
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
+ compresslevel = 0;
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 22);
+ }
sink = palloc0(sizeof(bbsink_zstd));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_zstd_ops;
diff --git a/src/bin/pg_basebackup/bbstreamer.h b/src/bin/pg_basebackup/bbstreamer.h
index 02d4c05df6..dfa3f77af4 100644
--- a/src/bin/pg_basebackup/bbstreamer.h
+++ b/src/bin/pg_basebackup/bbstreamer.h
@@ -22,6 +22,7 @@
#ifndef BBSTREAMER_H
#define BBSTREAMER_H
+#include "common/backup_compression.h"
#include "lib/stringinfo.h"
#include "pqexpbuffer.h"
@@ -200,17 +201,17 @@ bbstreamer_buffer_until(bbstreamer *streamer, const char **data, int *len,
*/
extern bbstreamer *bbstreamer_plain_writer_new(char *pathname, FILE *file);
extern bbstreamer *bbstreamer_gzip_writer_new(char *pathname, FILE *file,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_extractor_new(const char *basepath,
const char *(*link_map) (const char *),
void (*report_output_file) (const char *));
extern bbstreamer *bbstreamer_gzip_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_lz4_compressor_new(bbstreamer *next,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_lz4_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_zstd_compressor_new(bbstreamer *next,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_zstd_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_tar_parser_new(bbstreamer *next);
extern bbstreamer *bbstreamer_tar_terminator_new(bbstreamer *next);
diff --git a/src/bin/pg_basebackup/bbstreamer_gzip.c b/src/bin/pg_basebackup/bbstreamer_gzip.c
index 894f857103..1979e95639 100644
--- a/src/bin/pg_basebackup/bbstreamer_gzip.c
+++ b/src/bin/pg_basebackup/bbstreamer_gzip.c
@@ -76,7 +76,8 @@ const bbstreamer_ops bbstreamer_gzip_decompressor_ops = {
* closed so that the data may be written there.
*/
bbstreamer *
-bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
+bbstreamer_gzip_writer_new(char *pathname, FILE *file,
+ bc_specification *compress)
{
#ifdef HAVE_LIBZ
bbstreamer_gzip_writer *streamer;
@@ -115,11 +116,11 @@ bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
}
}
- if (gzsetparams(streamer->gzfile, compresslevel,
+ if (gzsetparams(streamer->gzfile, compress->level,
Z_DEFAULT_STRATEGY) != Z_OK)
{
pg_log_error("could not set compression level %d: %s",
- compresslevel, get_gz_error(streamer->gzfile));
+ compress->level, get_gz_error(streamer->gzfile));
exit(1);
}
diff --git a/src/bin/pg_basebackup/bbstreamer_lz4.c b/src/bin/pg_basebackup/bbstreamer_lz4.c
index 810052e4e3..a6ec317e2b 100644
--- a/src/bin/pg_basebackup/bbstreamer_lz4.c
+++ b/src/bin/pg_basebackup/bbstreamer_lz4.c
@@ -67,7 +67,7 @@ const bbstreamer_ops bbstreamer_lz4_decompressor_ops = {
* blocks.
*/
bbstreamer *
-bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
+bbstreamer_lz4_compressor_new(bbstreamer *next, bc_specification *compress)
{
#ifdef USE_LZ4
bbstreamer_lz4_frame *streamer;
@@ -89,7 +89,7 @@ bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
prefs = &streamer->prefs;
memset(prefs, 0, sizeof(LZ4F_preferences_t));
prefs->frameInfo.blockSizeID = LZ4F_max256KB;
- prefs->compressionLevel = compresslevel;
+ prefs->compressionLevel = compress->level;
/*
* Find out the compression bound, it specifies the minimum destination
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index e86749a8fb..caa5edcaf1 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -63,7 +63,7 @@ const bbstreamer_ops bbstreamer_zstd_decompressor_ops = {
* blocks.
*/
bbstreamer *
-bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
+bbstreamer_zstd_compressor_new(bbstreamer *next, bc_specification *compress)
{
#ifdef USE_ZSTD
bbstreamer_zstd_frame *streamer;
@@ -85,7 +85,7 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
/* Initialize stream compression preferences */
ZSTD_CCtx_setParameter(streamer->cctx, ZSTD_c_compressionLevel,
- compresslevel);
+ compress->level);
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 2943d9ec1a..3e6977df1a 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -29,6 +29,7 @@
#include "access/xlog_internal.h"
#include "bbstreamer.h"
+#include "common/backup_compression.h"
#include "common/file_perm.h"
#include "common/file_utils.h"
#include "common/logging.h"
@@ -57,6 +58,7 @@ typedef struct TablespaceList
typedef struct ArchiveStreamState
{
int tablespacenum;
+ bc_specification *compress;
bbstreamer *streamer;
bbstreamer *manifest_inject_streamer;
PQExpBuffer manifest_buffer;
@@ -132,9 +134,6 @@ static bool checksum_failure = false;
static bool showprogress = false;
static bool estimatesize = true;
static int verbose = 0;
-static int compresslevel = 0;
-static WalCompressionMethod compressmethod = COMPRESSION_NONE;
-static CompressionLocation compressloc = COMPRESS_LOCATION_UNSPECIFIED;
static IncludeWal includewal = STREAM_WAL;
static bool fastcheckpoint = false;
static bool writerecoveryconf = false;
@@ -198,7 +197,8 @@ static void progress_report(int tablespacenum, bool force, bool finished);
static bbstreamer *CreateBackupStreamer(char *archive_name, char *spclocation,
bbstreamer **manifest_inject_streamer_p,
bool is_recovery_guc_supported,
- bool expect_unterminated_tarfile);
+ bool expect_unterminated_tarfile,
+ bc_specification *compress);
static void ReceiveArchiveStreamChunk(size_t r, char *copybuf,
void *callback_data);
static char GetCopyDataByte(size_t r, char *copybuf, size_t *cursor);
@@ -207,7 +207,7 @@ static uint64 GetCopyDataUInt64(size_t r, char *copybuf, size_t *cursor);
static void GetCopyDataEnd(size_t r, char *copybuf, size_t cursor);
static void ReportCopyDataParseError(size_t r, char *copybuf);
static void ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
- bool tablespacenum);
+ bool tablespacenum, bc_specification *compress);
static void ReceiveTarCopyChunk(size_t r, char *copybuf, void *callback_data);
static void ReceiveBackupManifest(PGconn *conn);
static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
@@ -215,7 +215,9 @@ static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
static void ReceiveBackupManifestInMemory(PGconn *conn, PQExpBuffer buf);
static void ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
void *callback_data);
-static void BaseBackup(void);
+static void BaseBackup(char *compression_algorithm, char *compression_detail,
+ CompressionLocation compressloc,
+ bc_specification *client_compress);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
bool segment_finished);
@@ -405,8 +407,8 @@ usage(void)
printf(_(" -X, --wal-method=none|fetch|stream\n"
" include required WAL files with specified method\n"));
printf(_(" -z, --gzip compress tar output\n"));
- printf(_(" -Z, --compress=[{client|server}-]{gzip|lz4|zstd}[:LEVEL]\n"
- " compress tar output with given compression method or level\n"));
+ printf(_(" -Z, --compress=[{client|server}-]METHOD[:DETAIL]\n"
+ " compress on client or server as specified\n"));
printf(_(" -Z, --compress=none do not compress tar output\n"));
printf(_("\nGeneral options:\n"));
printf(_(" -c, --checkpoint=fast|spread\n"
@@ -542,7 +544,9 @@ typedef struct
} logstreamer_param;
static int
-LogStreamerMain(logstreamer_param *param)
+LogStreamerMain(logstreamer_param *param,
+ WalCompressionMethod wal_compress_method,
+ int wal_compress_level)
{
StreamCtl stream;
@@ -565,25 +569,14 @@ LogStreamerMain(logstreamer_param *param)
stream.mark_done = true;
stream.partial_suffix = NULL;
stream.replication_slot = replication_slot;
-
if (format == 'p')
stream.walmethod = CreateWalDirectoryMethod(param->xlog,
COMPRESSION_NONE, 0,
stream.do_sync);
- else if (compressloc != COMPRESS_LOCATION_CLIENT)
- stream.walmethod = CreateWalTarMethod(param->xlog,
- COMPRESSION_NONE,
- compresslevel,
- stream.do_sync);
- else if (compressmethod == COMPRESSION_GZIP)
- stream.walmethod = CreateWalTarMethod(param->xlog,
- compressmethod,
- compresslevel,
- stream.do_sync);
else
stream.walmethod = CreateWalTarMethod(param->xlog,
- COMPRESSION_NONE,
- compresslevel,
+ wal_compress_method,
+ wal_compress_level,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
@@ -629,7 +622,9 @@ LogStreamerMain(logstreamer_param *param)
* stream the logfile in parallel with the backups.
*/
static void
-StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
+StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier,
+ WalCompressionMethod wal_compress_method,
+ int wal_compress_level)
{
logstreamer_param *param;
uint32 hi,
@@ -729,7 +724,7 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
int ret;
/* in child process */
- ret = LogStreamerMain(param);
+ ret = LogStreamerMain(param, wal_compress_method, wal_compress_level);
/* temp debugging aid to analyze 019_replslot_limit failures */
if (verbose)
@@ -1004,136 +999,81 @@ parse_max_rate(char *src)
}
/*
- * Utility wrapper to parse the values specified for -Z/--compress.
- * *methodres and *levelres will be optionally filled with values coming
- * from the parsed results.
+ * Basic parsing of a value specified for -Z/--compress.
+ *
+ * We're not concerned here with understanding exactly what behavior the
+ * user wants, but we do need to know whether the user is requesting client
+ * or server side compression or leaving it unspecified, and we need to
+ * separate the name of the compression algorithm from the detail string.
+ *
+ * For instance, if the user writes --compress client-lz4:6, we want to
+ * separate that into (a) client-side compression, (b) algorithm "lz4",
+ * and (c) detail "6". Note, however, that all the client/server prefix is
+ * optional, and so is the detail. The algorithm name is required, unless
+ * the whole string is an integer, in which case we assume "gzip" as the
+ * algorithm and use the integer as the detail.
+ *
+ * We're not concerned with validation at this stage, so if the user writes
+ * --compress client-turkey:sandwich, the requested algorithm is "turkey"
+ * and the detail string is "sandwich". We'll sort out whether that's legal
+ * at a later stage.
*/
static void
-parse_compress_options(char *src, WalCompressionMethod *methodres,
- CompressionLocation *locationres, int *levelres)
+parse_compress_options(char *option, char **algorithm, char **detail,
+ CompressionLocation *locationres)
{
char *sep;
- int firstlen;
- char *firstpart;
+ char *endp;
/*
- * clear 'levelres' so that if there are multiple compression options,
- * the last one fully overrides the earlier ones
- */
- *levelres = 0;
-
- /* check if the option is split in two */
- sep = strchr(src, ':');
-
- /*
- * The first part of the option value could be a method name, or just a
- * level value.
- */
- firstlen = (sep != NULL) ? (sep - src) : strlen(src);
- firstpart = pg_malloc(firstlen + 1);
- memcpy(firstpart, src, firstlen);
- firstpart[firstlen] = '\0';
-
- /*
- * Check if the first part of the string matches with a supported
- * compression method.
+ * Check whether the compression specification consists of a bare integer.
+ *
+ * If so, for backward compatibility, assume gzip.
*/
- if (pg_strcasecmp(firstpart, "gzip") == 0)
+ (void) strtol(option, &endp, 10);
+ if (*endp == '\0')
{
- *methodres = COMPRESSION_GZIP;
*locationres = COMPRESS_LOCATION_UNSPECIFIED;
+ *algorithm = pstrdup("gzip");
+ *detail = pstrdup(option);
+ return;
}
- else if (pg_strcasecmp(firstpart, "client-gzip") == 0)
- {
- *methodres = COMPRESSION_GZIP;
- *locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-gzip") == 0)
+
+ /* Strip off any "client-" or "server-" prefix. */
+ if (strncmp(option, "server-", 7) == 0)
{
- *methodres = COMPRESSION_GZIP;
*locationres = COMPRESS_LOCATION_SERVER;
+ option += 7;
}
- else if (pg_strcasecmp(firstpart, "lz4") == 0)
- {
- *methodres = COMPRESSION_LZ4;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
- }
- else if (pg_strcasecmp(firstpart, "client-lz4") == 0)
+ else if (strncmp(option, "client-", 7) == 0)
{
- *methodres = COMPRESSION_LZ4;
*locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-lz4") == 0)
- {
- *methodres = COMPRESSION_LZ4;
- *locationres = COMPRESS_LOCATION_SERVER;
- }
- else if (pg_strcasecmp(firstpart, "zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
- }
- else if (pg_strcasecmp(firstpart, "client-zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_SERVER;
- }
- else if (pg_strcasecmp(firstpart, "none") == 0)
- {
- *methodres = COMPRESSION_NONE;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
+ option += 7;
}
else
- {
- /*
- * It does not match anything known, so check for the
- * backward-compatible case of only an integer where the implied
- * compression method changes depending on the level value.
- */
- if (!option_parse_int(firstpart, "-Z/--compress", 0,
- INT_MAX, levelres))
- exit(1);
-
- *methodres = (*levelres > 0) ?
- COMPRESSION_GZIP : COMPRESSION_NONE;
*locationres = COMPRESS_LOCATION_UNSPECIFIED;
- free(firstpart);
- return;
- }
-
+ /*
+ * Check whether there is a compression detail following the algorithm
+ * name.
+ */
+ sep = strchr(option, ':');
if (sep == NULL)
{
- /*
- * The caller specified a method without a colon separator, so let any
- * subsequent checks assign a default level.
- */
- free(firstpart);
- return;
+ *algorithm = pstrdup(option);
+ *detail = NULL;
}
-
- /* Check the contents after the colon separator. */
- sep++;
- if (*sep == '\0')
+ else
{
- pg_log_error("no compression level defined for method %s", firstpart);
- exit(1);
- }
+ char *alg;
- /*
- * For any of the methods currently supported, the data after the
- * separator can just be an integer.
- */
- if (!option_parse_int(sep, "-Z/--compress", 0, INT_MAX,
- levelres))
- exit(1);
+ alg = palloc((sep - option) + 1);
+ memcpy(alg, option, sep - option);
+ alg[sep - option] = '\0';
- free(firstpart);
+ *algorithm = alg;
+ *detail = pstrdup(sep + 1);
+ }
}
/*
@@ -1200,7 +1140,8 @@ static bbstreamer *
CreateBackupStreamer(char *archive_name, char *spclocation,
bbstreamer **manifest_inject_streamer_p,
bool is_recovery_guc_supported,
- bool expect_unterminated_tarfile)
+ bool expect_unterminated_tarfile,
+ bc_specification *compress)
{
bbstreamer *streamer = NULL;
bbstreamer *manifest_inject_streamer = NULL;
@@ -1316,32 +1257,28 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
archive_file = NULL;
}
- if (compressmethod == COMPRESSION_NONE ||
- compressloc != COMPRESS_LOCATION_CLIENT)
+ if (compress->algorithm == BACKUP_COMPRESSION_NONE)
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- else if (compressmethod == COMPRESSION_GZIP)
+ else if (compress->algorithm == BACKUP_COMPRESSION_GZIP)
{
strlcat(archive_filename, ".gz", sizeof(archive_filename));
streamer = bbstreamer_gzip_writer_new(archive_filename,
- archive_file,
- compresslevel);
+ archive_file, compress);
}
- else if (compressmethod == COMPRESSION_LZ4)
+ else if (compress->algorithm == BACKUP_COMPRESSION_LZ4)
{
strlcat(archive_filename, ".lz4", sizeof(archive_filename));
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- streamer = bbstreamer_lz4_compressor_new(streamer,
- compresslevel);
+ streamer = bbstreamer_lz4_compressor_new(streamer, compress);
}
- else if (compressmethod == COMPRESSION_ZSTD)
+ else if (compress->algorithm == BACKUP_COMPRESSION_ZSTD)
{
strlcat(archive_filename, ".zst", sizeof(archive_filename));
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- streamer = bbstreamer_zstd_compressor_new(streamer,
- compresslevel);
+ streamer = bbstreamer_zstd_compressor_new(streamer, compress);
}
else
{
@@ -1395,13 +1332,13 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
* If the user has requested a server compressed archive along with archive
* extraction at client then we need to decompress it.
*/
- if (format == 'p' && compressloc == COMPRESS_LOCATION_SERVER)
+ if (format == 'p')
{
- if (compressmethod == COMPRESSION_GZIP)
+ if (is_tar_gz)
streamer = bbstreamer_gzip_decompressor_new(streamer);
- else if (compressmethod == COMPRESSION_LZ4)
+ else if (is_tar_lz4)
streamer = bbstreamer_lz4_decompressor_new(streamer);
- else if (compressmethod == COMPRESSION_ZSTD)
+ else if (is_tar_zstd)
streamer = bbstreamer_zstd_decompressor_new(streamer);
}
@@ -1415,13 +1352,14 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
* manifest if present - as a single COPY stream.
*/
static void
-ReceiveArchiveStream(PGconn *conn)
+ReceiveArchiveStream(PGconn *conn, bc_specification *compress)
{
ArchiveStreamState state;
/* Set up initial state. */
memset(&state, 0, sizeof(state));
state.tablespacenum = -1;
+ state.compress = compress;
/* All the real work happens in ReceiveArchiveStreamChunk. */
ReceiveCopyData(conn, ReceiveArchiveStreamChunk, &state);
@@ -1542,7 +1480,8 @@ ReceiveArchiveStreamChunk(size_t r, char *copybuf, void *callback_data)
CreateBackupStreamer(archive_name,
spclocation,
&state->manifest_inject_streamer,
- true, false);
+ true, false,
+ state->compress);
}
break;
}
@@ -1743,7 +1682,7 @@ ReportCopyDataParseError(size_t r, char *copybuf)
*/
static void
ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
- bool tablespacenum)
+ bool tablespacenum, bc_specification *compress)
{
WriteTarState state;
bbstreamer *manifest_inject_streamer;
@@ -1759,7 +1698,8 @@ ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
state.streamer = CreateBackupStreamer(archive_name, spclocation,
&manifest_inject_streamer,
is_recovery_guc_supported,
- expect_unterminated_tarfile);
+ expect_unterminated_tarfile,
+ compress);
state.tablespacenum = tablespacenum;
ReceiveCopyData(conn, ReceiveTarCopyChunk, &state);
progress_update_filename(NULL);
@@ -1902,7 +1842,8 @@ ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
}
static void
-BaseBackup(void)
+BaseBackup(char *compression_algorithm, char *compression_detail,
+ CompressionLocation compressloc, bc_specification *client_compress)
{
PGresult *res;
char *sysidentifier;
@@ -2055,33 +1996,17 @@ BaseBackup(void)
if (compressloc == COMPRESS_LOCATION_SERVER)
{
- char *compressmethodstr = NULL;
-
if (!use_new_option_syntax)
{
pg_log_error("server does not support server-side compression");
exit(1);
}
- switch (compressmethod)
- {
- case COMPRESSION_GZIP:
- compressmethodstr = "gzip";
- break;
- case COMPRESSION_LZ4:
- compressmethodstr = "lz4";
- break;
- case COMPRESSION_ZSTD:
- compressmethodstr = "zstd";
- break;
- default:
- Assert(false);
- break;
- }
AppendStringCommandOption(&buf, use_new_option_syntax,
- "COMPRESSION", compressmethodstr);
- if (compresslevel >= 1) /* not 0 or Z_DEFAULT_COMPRESSION */
- AppendIntegerCommandOption(&buf, use_new_option_syntax,
- "COMPRESSION_LEVEL", compresslevel);
+ "COMPRESSION", compression_algorithm);
+ if (compression_detail != NULL)
+ AppendStringCommandOption(&buf, use_new_option_syntax,
+ "COMPRESSION_DETAIL",
+ compression_detail);
}
if (verbose)
@@ -2207,15 +2132,33 @@ BaseBackup(void)
*/
if (includewal == STREAM_WAL)
{
+ WalCompressionMethod wal_compress_method;
+ int wal_compress_level;
+
if (verbose)
pg_log_info("starting background WAL receiver");
- StartLogStreamer(xlogstart, starttli, sysidentifier);
+
+ if (client_compress->algorithm == BACKUP_COMPRESSION_GZIP)
+ {
+ wal_compress_method = COMPRESSION_GZIP;
+ wal_compress_level =
+ (client_compress->options & BACKUP_COMPRESSION_OPTION_LEVEL)
+ != 0 ? client_compress->level : 0;
+ }
+ else
+ {
+ wal_compress_method = COMPRESSION_NONE;
+ wal_compress_level = 0;
+ }
+
+ StartLogStreamer(xlogstart, starttli, sysidentifier,
+ wal_compress_method, wal_compress_level);
}
if (serverMajor >= 1500)
{
/* Receive a single tar stream with everything. */
- ReceiveArchiveStream(conn);
+ ReceiveArchiveStream(conn, client_compress);
}
else
{
@@ -2244,7 +2187,8 @@ BaseBackup(void)
spclocation = PQgetvalue(res, i, 1);
}
- ReceiveTarFile(conn, archive_name, spclocation, i);
+ ReceiveTarFile(conn, archive_name, spclocation, i,
+ client_compress);
}
/*
@@ -2511,6 +2455,10 @@ main(int argc, char **argv)
int c;
int option_index;
+ char *compression_algorithm = "none";
+ char *compression_detail = NULL;
+ CompressionLocation compressloc = COMPRESS_LOCATION_UNSPECIFIED;
+ bc_specification client_compress;
pg_logging_init(argv[0]);
progname = get_progname(argv[0]);
@@ -2616,17 +2564,13 @@ main(int argc, char **argv)
do_sync = false;
break;
case 'z':
-#ifdef HAVE_LIBZ
- compresslevel = Z_DEFAULT_COMPRESSION;
-#else
- compresslevel = 1; /* will be rejected below */
-#endif
- compressmethod = COMPRESSION_GZIP;
+ compression_algorithm = "gzip";
+ compression_detail = NULL;
compressloc = COMPRESS_LOCATION_UNSPECIFIED;
break;
case 'Z':
- parse_compress_options(optarg, &compressmethod,
- &compressloc, &compresslevel);
+ parse_compress_options(optarg, &compression_algorithm,
+ &compression_detail, &compressloc);
break;
case 'c':
if (pg_strcasecmp(optarg, "fast") == 0)
@@ -2753,12 +2697,11 @@ main(int argc, char **argv)
}
/*
- * If we're compressing the backup and the user has not said where to
- * perform the compression, do it on the client, unless they specified
- * --target, in which case the server is the only choice.
+ * If the user has not specified where to perform backup compression,
+ * default to the client, unless the user specified --target, in which case
+ * the server is the only choice.
*/
- if (compressmethod != COMPRESSION_NONE &&
- compressloc == COMPRESS_LOCATION_UNSPECIFIED)
+ if (compressloc == COMPRESS_LOCATION_UNSPECIFIED)
{
if (backup_target == NULL)
compressloc = COMPRESS_LOCATION_CLIENT;
@@ -2766,6 +2709,40 @@ main(int argc, char **argv)
compressloc = COMPRESS_LOCATION_SERVER;
}
+ /*
+ * If any compression that we're doing is happening on the client side,
+ * we must try to parse the compression algorithm and detail, but if it's
+ * all on the server side, then we're just going to pass through whatever
+ * was requested and let the server decide what to do.
+ */
+ if (compressloc == COMPRESS_LOCATION_CLIENT)
+ {
+ bc_algorithm alg;
+ char *error_detail;
+
+ if (!parse_bc_algorithm(compression_algorithm, &alg))
+ {
+ pg_log_error("unrecognized compression algorithm \"%s\"",
+ compression_algorithm);
+ exit(1);
+ }
+
+ parse_bc_specification(alg, compression_detail, &client_compress);
+ error_detail = validate_bc_specification(&client_compress);
+ if (error_detail != NULL)
+ {
+ pg_log_error("invalid compression specification: %s",
+ error_detail);
+ exit(1);
+ }
+ }
+ else
+ {
+ Assert(compressloc == COMPRESS_LOCATION_SERVER);
+ client_compress.algorithm = BACKUP_COMPRESSION_NONE;
+ client_compress.options = 0;
+ }
+
/*
* Can't perform client-side compression if the backup is not being
* sent to the client.
@@ -2779,9 +2756,10 @@ main(int argc, char **argv)
}
/*
- * Compression doesn't make sense unless tar format is in use.
+ * Client-side compression doesn't make sense unless tar format is in use.
*/
- if (format == 'p' && compressloc == COMPRESS_LOCATION_CLIENT)
+ if (format == 'p' && compressloc == COMPRESS_LOCATION_CLIENT &&
+ client_compress.algorithm != BACKUP_COMPRESSION_NONE)
{
pg_log_error("only tar mode backups can be compressed");
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
@@ -2882,56 +2860,6 @@ main(int argc, char **argv)
}
}
- /* Sanity checks for compression-related options. */
- switch (compressmethod)
- {
- case COMPRESSION_NONE:
- if (compresslevel != 0)
- {
- pg_log_error("cannot use compression level with method %s",
- "none");
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
- progname);
- exit(1);
- }
- break;
- case COMPRESSION_GZIP:
- if (compresslevel > 9)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 9",
- compresslevel, "gzip");
- exit(1);
- }
- if (compressloc == COMPRESS_LOCATION_CLIENT)
- {
-#ifdef HAVE_LIBZ
- if (compresslevel == 0)
- compresslevel = Z_DEFAULT_COMPRESSION;
-#else
- pg_log_error("this build does not support compression with %s",
- "gzip");
- exit(1);
-#endif
- }
- break;
- case COMPRESSION_LZ4:
- if (compresslevel > 12)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 12",
- compresslevel, "lz4");
- exit(1);
- }
- break;
- case COMPRESSION_ZSTD:
- if (compresslevel > 22)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 22",
- compresslevel, "zstd");
- exit(1);
- }
- break;
- }
-
/*
* Sanity checks for progress reporting options.
*/
@@ -3040,7 +2968,8 @@ main(int argc, char **argv)
free(linkloc);
}
- BaseBackup();
+ BaseBackup(compression_algorithm, compression_detail, compressloc,
+ &client_compress);
success = true;
return 0;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index efefe947d9..2869a239e7 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -42,16 +42,12 @@ $node->command_fails(['pg_basebackup'],
# Sanity checks for options
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none:1' ],
- qr/\Qpg_basebackup: error: cannot use compression level with method none/,
+ qr/\Qcompression algorithm "none" does not accept a compression level/,
'failure if method "none" specified with compression level');
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none+' ],
- qr/\Qpg_basebackup: error: invalid value "none+" for option/,
+ qr/\Qunrecognized compression algorithm "none+"/,
'failure on incorrect separator to define compression level');
-$node->command_fails_like(
- [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none:' ],
- qr/\Qpg_basebackup: error: no compression level defined for method none/,
- 'failure on missing compression level value');
# Some Windows ANSI code pages may reject this filename, in which case we
# quietly proceed without this bit of test coverage.
@@ -89,6 +85,70 @@ print $conf "wal_level = replica\n";
close $conf;
$node->restart;
+# Now that we have a server that supports replication commands, test whether
+# certain invalid compression commands fail on the client side with client-side
+# compression and on the server side with server-side compression.
+my $client_fails =
+ 'pg_basebackup: error: ';
+my $server_fails =
+ 'pg_basebackup: error: could not initiate base backup: ERROR: ';
+my @compression_failure_tests = (
+ [
+ 'extrasquishy',
+ 'unrecognized compression algorithm "extrasquishy"',
+ 'failure on invalid compression algorithm'
+ ],
+ [
+ 'gzip:',
+ 'invalid compression specification: found empty string where a compression option was expected',
+ 'failure on empty compression options list'
+ ],
+ [
+ 'gzip:thunk',
+ 'invalid compression specification: unknown compression option "thunk"',
+ 'failure on unknown compression option'
+ ],
+ [
+ 'gzip:level',
+ 'invalid compression specification: compression option "level" requires a value',
+ 'failure on missing compression level'
+ ],
+ [
+ 'gzip:level=',
+ 'invalid compression specification: value for compression option "level" must be an integer',
+ 'failure on empty compression level'
+ ],
+ [
+ 'gzip:level=high',
+ 'invalid compression specification: value for compression option "level" must be an integer',
+ 'failure on non-numeric compression level'
+ ],
+ [
+ 'gzip:level=236',
+ 'invalid compression specification: compression algorithm "gzip" expects a compression level between 1 and 9',
+ 'failure on out-of-range compression level'
+ ],
+ [
+ 'gzip:level=9,',
+ 'invalid compression specification: found empty string where a compression option was expected',
+ 'failure on extra, empty compression option'
+ ],
+);
+for my $cft (@compression_failure_tests)
+{
+ my $cfail = quotemeta($client_fails . $cft->[1]);
+ my $sfail = quotemeta($server_fails . $cft->[1]);
+ $node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', $cft->[0] ],
+ qr/$cfail/,
+ 'client '. $cft->[2]);
+ $node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress',
+ 'server-' . $cft->[0] ],
+ qr/$sfail/,
+ 'server ' . $cft->[2]);
+}
+
# Write some files to test that they are not copied.
foreach my $filename (
qw(backup_label tablespace_map postgresql.auto.conf.tmp
diff --git a/src/common/Makefile b/src/common/Makefile
index 31c0dd366d..f627349835 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -47,6 +47,7 @@ LIBS += $(PTHREAD_LIBS)
OBJS_COMMON = \
archive.o \
+ backup_compression.o \
base64.o \
checksum_helper.o \
config_info.o \
diff --git a/src/common/backup_compression.c b/src/common/backup_compression.c
new file mode 100644
index 0000000000..46fa766130
--- /dev/null
+++ b/src/common/backup_compression.c
@@ -0,0 +1,269 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_compression.c
+ *
+ * Shared code for backup compression methods and specifications.
+ *
+ * A compression specification specifies the parameters that should be used
+ * when performing compression with a specific algorithm. The simplest
+ * possible compression specification is an integer, which sets the
+ * compression level.
+ *
+ * Otherwise, a compression specification is a comma-separated list of items,
+ * each having the form keyword or keyword=value.
+ *
+ * Currently, the only supported keyword is "level".
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/backup_compression.c
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#include "common/backup_compression.h"
+
+static int expect_integer_value(char *keyword, char *value,
+ bc_specification *result);
+
+/*
+ * Look up a compression algorithm by name. Returns true and sets *algorithm
+ * if the name is recognized. Otherwise returns false.
+ */
+bool
+parse_bc_algorithm(char *name, bc_algorithm *algorithm)
+{
+ if (strcmp(name, "none") == 0)
+ *algorithm = BACKUP_COMPRESSION_NONE;
+ else if (strcmp(name, "gzip") == 0)
+ *algorithm = BACKUP_COMPRESSION_GZIP;
+ else if (strcmp(name, "lz4") == 0)
+ *algorithm = BACKUP_COMPRESSION_LZ4;
+ else if (strcmp(name, "zstd") == 0)
+ *algorithm = BACKUP_COMPRESSION_ZSTD;
+ else
+ return false;
+ return true;
+}
+
+/*
+ * Get the human-readable name corresponding to a particular compression
+ * algorithm.
+ */
+const char *
+get_bc_algorithm_name(bc_algorithm algorithm)
+{
+ switch (algorithm)
+ {
+ case BACKUP_COMPRESSION_NONE:
+ return "none";
+ case BACKUP_COMPRESSION_GZIP:
+ return "gzip";
+ case BACKUP_COMPRESSION_LZ4:
+ return "lz4";
+ case BACKUP_COMPRESSION_ZSTD:
+ return "zstd";
+ /* no default, to provoke compiler warnings if values are added */
+ }
+ Assert(false);
+}
+
+/*
+ * Parse a compression specification for a specified algorithm.
+ *
+ * See the file header comments for a brief description of what a compression
+ * specification is expected to look like.
+ *
+ * On return, all fields of the result object will be initialized.
+ * In particular, result->parse_error will be NULL if no errors occurred
+ * during parsing, and will otherwise contain a an appropriate error message.
+ * The caller may free this error message string using pfree, if desired.
+ * Note, however, even if there's no parse error, the string might not make
+ * sense: e.g. for gzip, level=12 is not sensible, but it does parse OK.
+ *
+ * Use validate_bc_specification() to find out whether a compression
+ * specification is semantically sensible.
+ */
+void
+parse_bc_specification(bc_algorithm algorithm, char *specification,
+ bc_specification *result)
+{
+ int bare_level;
+ char *bare_level_endp;
+
+ /* Initial setup of result object. */
+ result->algorithm = algorithm;
+ result->options = 0;
+ result->level = -1;
+ result->parse_error = NULL;
+
+ /* If there is no specification, we're done already. */
+ if (specification == NULL)
+ return;
+
+ /* As a special case, the specification can be a bare integer. */
+ bare_level = strtol(specification, &bare_level_endp, 10);
+ if (specification != bare_level_endp && *bare_level_endp == '\0')
+ {
+ result->level = bare_level;
+ result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
+ return;
+ }
+
+ /* Look for comma-separated keyword or keyword=value entries. */
+ while (1)
+ {
+ char *kwstart;
+ char *kwend;
+ char *vstart;
+ char *vend;
+ int kwlen;
+ int vlen;
+ bool has_value;
+ char *keyword;
+ char *value;
+
+ /* Figure start, end, and length of next keyword and any value. */
+ kwstart = kwend = specification;
+ while (*kwend != '\0' && *kwend != ',' && *kwend != '=')
+ ++kwend;
+ kwlen = kwend - kwstart;
+ if (*kwend != '=')
+ {
+ vstart = vend = NULL;
+ vlen = 0;
+ has_value = false;
+ }
+ else
+ {
+ vstart = vend = kwend + 1;
+ while (*vend != '\0' && *vend != ',')
+ ++vend;
+ vlen = vend - vstart;
+ has_value = true;
+ }
+
+ /* Reject empty keyword. */
+ if (kwlen == 0)
+ {
+ result->parse_error =
+ pstrdup(_("found empty string where a compression option was expected"));
+ break;
+ }
+
+ /* Extract keyword and value as separate C strings. */
+ keyword = palloc(kwlen + 1);
+ memcpy(keyword, kwstart, kwlen);
+ keyword[kwlen] = '\0';
+ if (!has_value)
+ value = NULL;
+ else
+ {
+ value = palloc(vlen + 1);
+ memcpy(value, vstart, vlen);
+ value[vlen] = '\0';
+ }
+
+ /* Handle whatever keyword we found. */
+ if (strcmp(keyword, "level") == 0)
+ {
+ result->level = expect_integer_value(keyword, value, result);
+ result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
+ }
+ else
+ result->parse_error =
+ psprintf(_("unknown compression option \"%s\""), keyword);
+
+ /* Release memory, just to be tidy. */
+ pfree(keyword);
+ if (value != NULL)
+ pfree(value);
+
+ /* If we got an error or have reached the end of the string, stop. */
+ if (result->parse_error != NULL || *kwend == '\0' || *vend == '\0')
+ break;
+
+ /* Advance to next entry and loop around. */
+ specification = vend == NULL ? kwend + 1 : vend + 1;
+ }
+}
+
+/*
+ * Parse 'value' as an integer and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1.
+ */
+static int
+expect_integer_value(char *keyword, char *value, bc_specification *result)
+{
+ int ivalue;
+ char *ivalue_endp;
+
+ if (value == NULL)
+ {
+ result->parse_error =
+ psprintf(_("compression option \"%s\" requires a value"),
+ keyword);
+ return -1;
+ }
+
+ ivalue = strtol(value, &ivalue_endp, 10);
+ if (ivalue_endp == value || *ivalue_endp != '\0')
+ {
+ result->parse_error =
+ psprintf(_("value for compression option \"%s\" must be an integer"),
+ keyword);
+ return -1;
+ }
+ return ivalue;
+}
+
+/*
+ * Returns NULL if the compression specification string was syntactically
+ * valid and semantically sensible. Otherwise, returns an error message.
+ *
+ * Does not test whether this build of PostgreSQL supports the requested
+ * compression method.
+ */
+char *
+validate_bc_specification(bc_specification *spec)
+{
+ /* If it didn't even parse OK, it's definitely no good. */
+ if (spec->parse_error != NULL)
+ return spec->parse_error;
+
+ /*
+ * If a compression level was specified, check that the algorithm expects
+ * a compression level and that the level is within the legal range for
+ * the algorithm.
+ */
+ if ((spec->options & BACKUP_COMPRESSION_OPTION_LEVEL) != 0)
+ {
+ int min_level = 1;
+ int max_level;
+
+ if (spec->algorithm == BACKUP_COMPRESSION_GZIP)
+ max_level = 9;
+ else if (spec->algorithm == BACKUP_COMPRESSION_LZ4)
+ max_level = 12;
+ else if (spec->algorithm == BACKUP_COMPRESSION_ZSTD)
+ max_level = 22;
+ else
+ return psprintf(_("compression algorithm \"%s\" does not accept a compression level"),
+ get_bc_algorithm_name(spec->algorithm));
+
+ if (spec->level < min_level || spec->level > max_level)
+ return psprintf(_("compression algorithm \"%s\" expects a compression level between %d and %d"),
+ get_bc_algorithm_name(spec->algorithm),
+ min_level, max_level);
+ }
+
+ return NULL;
+}
diff --git a/src/include/common/backup_compression.h b/src/include/common/backup_compression.h
new file mode 100644
index 0000000000..0565cbc657
--- /dev/null
+++ b/src/include/common/backup_compression.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_compression.h
+ *
+ * Shared definitions for backup compression methods and specifications.
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/backup_compression.h
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef BACKUP_COMPRESSION_H
+#define BACKUP_COMPRESSION_H
+
+typedef enum bc_algorithm
+{
+ BACKUP_COMPRESSION_NONE,
+ BACKUP_COMPRESSION_GZIP,
+ BACKUP_COMPRESSION_LZ4,
+ BACKUP_COMPRESSION_ZSTD
+} bc_algorithm;
+
+#define BACKUP_COMPRESSION_OPTION_LEVEL (1 << 0)
+
+typedef struct bc_specification
+{
+ bc_algorithm algorithm;
+ unsigned options; /* OR of BACKUP_COMPRESSION_OPTION constants */
+ int level;
+ char *parse_error; /* NULL if parsing was OK, else message */
+} bc_specification;
+
+extern bool parse_bc_algorithm(char *name, bc_algorithm *algorithm);
+extern const char *get_bc_algorithm_name(bc_algorithm algorithm);
+
+extern void parse_bc_specification(bc_algorithm algorithm,
+ char *specification,
+ bc_specification *result);
+
+extern char *validate_bc_specification(bc_specification *);
+
+#endif
diff --git a/src/include/replication/basebackup_sink.h b/src/include/replication/basebackup_sink.h
index a7f16758a4..654df28576 100644
--- a/src/include/replication/basebackup_sink.h
+++ b/src/include/replication/basebackup_sink.h
@@ -27,6 +27,7 @@
#define BASEBACKUP_SINK_H
#include "access/xlog_internal.h"
+#include "common/backup_compression.h"
#include "nodes/pg_list.h"
/* Forward declarations. */
@@ -283,9 +284,9 @@ extern void bbsink_forward_cleanup(bbsink *sink);
/* Constructors for various types of sinks. */
extern bbsink *bbsink_copystream_new(bool send_to_client);
-extern bbsink *bbsink_gzip_new(bbsink *next, int compresslevel);
-extern bbsink *bbsink_lz4_new(bbsink *next, int compresslevel);
-extern bbsink *bbsink_zstd_new(bbsink *next, int compresslevel);
+extern bbsink *bbsink_gzip_new(bbsink *next, bc_specification *);
+extern bbsink *bbsink_lz4_new(bbsink *next, bc_specification *);
+extern bbsink *bbsink_zstd_new(bbsink *next, bc_specification *);
extern bbsink *bbsink_progress_new(bbsink *next, bool estimate_backup_size);
extern bbsink *bbsink_server_new(bbsink *next, char *pathname);
extern bbsink *bbsink_throttle_new(bbsink *next, uint32 maxrate);
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 441d6ae6bf..de8676d339 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -124,7 +124,7 @@ sub mkvcbuild
}
our @pgcommonallfiles = qw(
- archive.c base64.c checksum_helper.c
+ archive.c backup_compression.c base64.c checksum_helper.c
config_info.c controldata_utils.c d2s.c encnames.c exec.c
f2s.c file_perm.c file_utils.c hashfn.c ip.c jsonapi.c
keywords.c kwlookup.c link-canary.c md5_common.c
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index eaf3e7a8d4..01748cac07 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3788,3 +3788,5 @@ yyscan_t
z_stream
z_streamp
zic_t
+bc_algorithm
+bc_specification
--
2.24.3 (Apple Git-128)
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-21 18:22 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 32+ messages in thread
From: Justin Pryzby @ 2022-03-21 18:22 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers@postgresql.org, tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 21, 2022 at 12:57:36PM -0400, Robert Haas wrote:
> > typo: contain a an
> I searched for the "contain a an" typo that you mentioned but was not able to
> find it. Can you give me a more specific pointer?
Here:
+ * during parsing, and will otherwise contain a an appropriate error message.
> I looked a little bit more at the compression method vs. compression
> algorithm thing. I agree that there is some inconsistency in
> terminology here, but I'm still not sure that we are well-served by
> trying to make it totally uniform, especially if we pick the word
> "method" as the standard rather than "algorithm". In my opinion,
> "method" is less specific than "algorithm". If someone asks me to
> choose a compression algorithm, I know that I should give an answer
> like "lz4" or "zstd". If they ask me to pick a compression method, I'm
> not quite sure whether they want that kind of answer or whether they
> want something more detailed, like "use lz4 with compression level 3
> and a 1MB block size". After all, that is (at least according to my
> understanding of how English works) a perfectly valid answer to the
> question "what method should I use to compress this data?" -- but not
> to the question "what algorithm should I use to compress this data?".
> The latter can ONLY be properly answered by saying something like
> "lz4". And I think that's really the root of my hesitation to make the
> kinds of changes you want here.
I think "algorithm" could be much more nuanced than "lz4", but I also think
we've spent more than enough time on it now :)
--
Justin
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
@ 2022-03-21 18:25 ` Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
0 siblings, 1 reply; 32+ messages in thread
From: Robert Haas @ 2022-03-21 18:25 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 21, 2022 at 2:22 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> + * during parsing, and will otherwise contain a an appropriate error message.
OK, thanks. v4 attached.
> I think "algorithm" could be much more nuanced than "lz4", but I also think
> we've spent more than enough time on it now :)
Oh dear. But yes.
--
Robert Haas
EDB: http://www.enterprisedb.com
Attachments:
v4-0001-Replace-BASE_BACKUP-COMPRESSION_LEVEL-option-with.patch (application/octet-stream, 56.7K) download
From 2481134c810e8f3526852544868e5c551ccd4ee5 Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Mon, 21 Mar 2022 14:24:15 -0400
Subject: [PATCH v4] Replace BASE_BACKUP COMPRESSION_LEVEL option with
COMPRESSION_DETAIL.
There are more compression parameters that can be specified than just
an integer compression level, so rename the new COMPRESSION_LEVEL
option to COMPRESSION_DETAIL before it gets released. Introduce a
flexible syntax for that option to allow arbitrary options to be
specified without needing to adjust the main replication grammar,
and common code to parse it that is shared between the client and
the server.
This commit doesn't actually add any new compression parameters,
so the only user-visible change is that you can now type something
like pg_basebackup --compress gzip:level=5 instead of writing just
pg_basebackup --compress gzip:5. However, it should make it easy to
add new options. If for example gzip starts offering fries, we can
support pg_basebackup --compress gzip:level=5,fries=true for the
benefit of users who want fries with that.
Along the way, this fixes a few things in pg_basebackup so that the
pg_basebackup can be used with a server-side compression algorithm
that pg_basebackup itself does not understand. For example,
pg_basebackup --compress server-lz4 could still succeed even if
only the server and not the client has LZ4 support, provided that
the other options to pg_basebackup don't require the client to
decompress the archive.
Patch by me. Reviewed by Justin Pryzby.
---
doc/src/sgml/protocol.sgml | 18 +-
doc/src/sgml/ref/pg_basebackup.sgml | 25 +-
src/backend/replication/basebackup.c | 62 +--
src/backend/replication/basebackup_gzip.c | 20 +-
src/backend/replication/basebackup_lz4.c | 19 +-
src/backend/replication/basebackup_zstd.c | 19 +-
src/bin/pg_basebackup/bbstreamer.h | 7 +-
src/bin/pg_basebackup/bbstreamer_gzip.c | 7 +-
src/bin/pg_basebackup/bbstreamer_lz4.c | 4 +-
src/bin/pg_basebackup/bbstreamer_zstd.c | 4 +-
src/bin/pg_basebackup/pg_basebackup.c | 409 ++++++++-----------
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 72 +++-
src/common/Makefile | 1 +
src/common/backup_compression.c | 269 ++++++++++++
src/include/common/backup_compression.h | 44 ++
src/include/replication/basebackup_sink.h | 7 +-
src/tools/msvc/Mkvcbuild.pm | 2 +-
src/tools/pgindent/typedefs.list | 2 +
18 files changed, 662 insertions(+), 329 deletions(-)
create mode 100644 src/common/backup_compression.c
create mode 100644 src/include/common/backup_compression.h
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9178c779ba..00c593f1af 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2731,14 +2731,24 @@ The commands accepted in replication mode are:
</varlistentry>
<varlistentry>
- <term><literal>COMPRESSION_LEVEL</literal> <replaceable>level</replaceable></term>
+ <term><literal>COMPRESSION_DETAIL</literal> <replaceable>detail</replaceable></term>
<listitem>
<para>
Specifies the compression level to be used. This should only be
used in conjunction with the <literal>COMPRESSION</literal> option.
- For <literal>gzip</literal> the value should be an integer between 1
- and 9, for <literal>lz4</literal> between 1 and 12, and for
- <literal>zstd</literal> it should be between 1 and 22.
+ If the value is an integer, it specifies the compression level.
+ Otherwise, it should be a comma-separated list of items, each of
+ the form <literal>keyword</literal> or
+ <literal>keyword=value</literal>. Currently, the only supported
+ keyword is <literal>level</literal>, which sets the compression
+ level.
+ </para>
+
+ <para>
+ For <literal>gzip</literal> the compression level should be an
+ integer between 1 and 9, for <literal>lz4</literal> an integer
+ between 1 and 12, and for <literal>zstd</literal> an integer
+ between 1 and 22.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 4a630b59b7..46d7f15e54 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -399,9 +399,9 @@ PostgreSQL documentation
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
- <term><option>-Z [{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>level</replaceable>]</term>
+ <term><option>-Z [{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>detail</replaceable>]</term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
- <term><option>--compress=[{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>level</replaceable>]</term>
+ <term><option>--compress=[{client|server}-]<replaceable class="parameter">method[:<replaceable>detail</replaceable>]</replaceable></option></term>
<listitem>
<para>
Requests compression of the backup. If <literal>client</literal> or
@@ -419,13 +419,20 @@ PostgreSQL documentation
<para>
The compression method can be set to <literal>gzip</literal>,
<literal>lz4</literal>, <literal>zstd</literal>, or
- <literal>none</literal> for no compression. A compression level can
- optionally be specified, by appending the level number after a colon
- (<literal>:</literal>). If no level is specified, the default
- compression level will be used. If only a level is specified without
- mentioning an algorithm, <literal>gzip</literal> compression will be
- used if the level is greater than 0, and no compression will be used if
- the level is 0.
+ <literal>none</literal> for no compression. A compression detail
+ string can optionally be specified. If the detail string is an
+ integer, it specifies the compression level. Otherwise, it should be
+ a comma-separated list of items, each of the form
+ <literal>keyword</literal> or <literal>keyword=value</literal>.
+ Currently, the only supported keyword is <literal>level</literal>,
+ which sets the compression level.
+ </para>
+ <para>
+ If no compression level is specified, the default compression level
+ will be used. If only a level is specified without mentioning an
+ algorithm, <literal>gzip</literal> compression will be used if the
+ level is greater than 0, and no compression will be used if the level
+ is 0.
</para>
<para>
When the tar format is used with <literal>gzip</literal>,
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index c2aedc14a2..49deead091 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h" /* for pg_start/stop_backup */
+#include "common/backup_compression.h"
#include "common/file_perm.h"
#include "commands/defrem.h"
#include "lib/stringinfo.h"
@@ -54,14 +55,6 @@
*/
#define SINK_BUFFER_LENGTH Max(32768, BLCKSZ)
-typedef enum
-{
- BACKUP_COMPRESSION_NONE,
- BACKUP_COMPRESSION_GZIP,
- BACKUP_COMPRESSION_LZ4,
- BACKUP_COMPRESSION_ZSTD
-} basebackup_compression_type;
-
typedef struct
{
const char *label;
@@ -75,8 +68,8 @@ typedef struct
bool use_copytblspc;
BaseBackupTargetHandle *target_handle;
backup_manifest_option manifest;
- basebackup_compression_type compression;
- int compression_level;
+ bc_algorithm compression;
+ bc_specification compression_specification;
pg_checksum_type manifest_checksum_type;
} basebackup_options;
@@ -713,12 +706,14 @@ parse_basebackup_options(List *options, basebackup_options *opt)
char *target_str = NULL;
char *target_detail_str = NULL;
bool o_compression = false;
- bool o_compression_level = false;
+ bool o_compression_detail = false;
+ char *compression_detail_str = NULL;
MemSet(opt, 0, sizeof(*opt));
opt->manifest = MANIFEST_OPTION_NO;
opt->manifest_checksum_type = CHECKSUM_TYPE_CRC32C;
opt->compression = BACKUP_COMPRESSION_NONE;
+ opt->compression_specification.algorithm = BACKUP_COMPRESSION_NONE;
foreach(lopt, options)
{
@@ -885,29 +880,21 @@ parse_basebackup_options(List *options, basebackup_options *opt)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("duplicate option \"%s\"", defel->defname)));
- if (strcmp(optval, "none") == 0)
- opt->compression = BACKUP_COMPRESSION_NONE;
- else if (strcmp(optval, "gzip") == 0)
- opt->compression = BACKUP_COMPRESSION_GZIP;
- else if (strcmp(optval, "lz4") == 0)
- opt->compression = BACKUP_COMPRESSION_LZ4;
- else if (strcmp(optval, "zstd") == 0)
- opt->compression = BACKUP_COMPRESSION_ZSTD;
- else
+ if (!parse_bc_algorithm(optval, &opt->compression))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("unrecognized compression algorithm: \"%s\"",
+ errmsg("unrecognized compression algorithm \"%s\"",
optval)));
o_compression = true;
}
- else if (strcmp(defel->defname, "compression_level") == 0)
+ else if (strcmp(defel->defname, "compression_detail") == 0)
{
- if (o_compression_level)
+ if (o_compression_detail)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("duplicate option \"%s\"", defel->defname)));
- opt->compression_level = defGetInt32(defel);
- o_compression_level = true;
+ compression_detail_str = defGetString(defel);
+ o_compression_detail = true;
}
else
ereport(ERROR,
@@ -949,10 +936,25 @@ parse_basebackup_options(List *options, basebackup_options *opt)
opt->target_handle =
BaseBackupGetTargetHandle(target_str, target_detail_str);
- if (o_compression_level && !o_compression)
+ if (o_compression_detail && !o_compression)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("compression level requires compression")));
+ errmsg("compression detail requires compression")));
+
+ if (o_compression)
+ {
+ char *error_detail;
+
+ parse_bc_specification(opt->compression, compression_detail_str,
+ &opt->compression_specification);
+ error_detail =
+ validate_bc_specification(&opt->compression_specification);
+ if (error_detail != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("invalid compression specification: %s",
+ error_detail));
+ }
}
@@ -998,11 +1000,11 @@ SendBaseBackup(BaseBackupCmd *cmd)
/* Set up server-side compression, if client requested it */
if (opt.compression == BACKUP_COMPRESSION_GZIP)
- sink = bbsink_gzip_new(sink, opt.compression_level);
+ sink = bbsink_gzip_new(sink, &opt.compression_specification);
else if (opt.compression == BACKUP_COMPRESSION_LZ4)
- sink = bbsink_lz4_new(sink, opt.compression_level);
+ sink = bbsink_lz4_new(sink, &opt.compression_specification);
else if (opt.compression == BACKUP_COMPRESSION_ZSTD)
- sink = bbsink_zstd_new(sink, opt.compression_level);
+ sink = bbsink_zstd_new(sink, &opt.compression_specification);
/* Set up progress reporting. */
sink = bbsink_progress_new(sink, opt.progress);
diff --git a/src/backend/replication/basebackup_gzip.c b/src/backend/replication/basebackup_gzip.c
index b66d3da7a3..703a91ba77 100644
--- a/src/backend/replication/basebackup_gzip.c
+++ b/src/backend/replication/basebackup_gzip.c
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_gzip_ops = {
#endif
/*
- * Create a new basebackup sink that performs gzip compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs gzip compression.
*/
bbsink *
-bbsink_gzip_new(bbsink *next, int compresslevel)
+bbsink_gzip_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef HAVE_LIBZ
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,15 +72,14 @@ bbsink_gzip_new(bbsink *next, int compresslevel)
bbsink_gzip *sink;
Assert(next != NULL);
- Assert(compresslevel >= 0 && compresslevel <= 9);
- if (compresslevel == 0)
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
compresslevel = Z_DEFAULT_COMPRESSION;
- else if (compresslevel < 0 || compresslevel > 9)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("gzip compression level %d is out of range",
- compresslevel)));
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 9);
+ }
sink = palloc0(sizeof(bbsink_gzip));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_gzip_ops;
diff --git a/src/backend/replication/basebackup_lz4.c b/src/backend/replication/basebackup_lz4.c
index d838f723d0..06c161ddc4 100644
--- a/src/backend/replication/basebackup_lz4.c
+++ b/src/backend/replication/basebackup_lz4.c
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_lz4_ops = {
#endif
/*
- * Create a new basebackup sink that performs lz4 compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs lz4 compression.
*/
bbsink *
-bbsink_lz4_new(bbsink *next, int compresslevel)
+bbsink_lz4_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef USE_LZ4
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -72,11 +73,13 @@ bbsink_lz4_new(bbsink *next, int compresslevel)
Assert(next != NULL);
- if (compresslevel < 0 || compresslevel > 12)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("lz4 compression level %d is out of range",
- compresslevel)));
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
+ compresslevel = 0;
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 12);
+ }
sink = palloc0(sizeof(bbsink_lz4));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_lz4_ops;
diff --git a/src/backend/replication/basebackup_zstd.c b/src/backend/replication/basebackup_zstd.c
index c0e2be6e27..96b7985693 100644
--- a/src/backend/replication/basebackup_zstd.c
+++ b/src/backend/replication/basebackup_zstd.c
@@ -55,12 +55,13 @@ const bbsink_ops bbsink_zstd_ops = {
#endif
/*
- * Create a new basebackup sink that performs zstd compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs zstd compression.
*/
bbsink *
-bbsink_zstd_new(bbsink *next, int compresslevel)
+bbsink_zstd_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef USE_ZSTD
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,11 +72,13 @@ bbsink_zstd_new(bbsink *next, int compresslevel)
Assert(next != NULL);
- if (compresslevel < 0 || compresslevel > 22)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("zstd compression level %d is out of range",
- compresslevel)));
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
+ compresslevel = 0;
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 22);
+ }
sink = palloc0(sizeof(bbsink_zstd));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_zstd_ops;
diff --git a/src/bin/pg_basebackup/bbstreamer.h b/src/bin/pg_basebackup/bbstreamer.h
index 02d4c05df6..dfa3f77af4 100644
--- a/src/bin/pg_basebackup/bbstreamer.h
+++ b/src/bin/pg_basebackup/bbstreamer.h
@@ -22,6 +22,7 @@
#ifndef BBSTREAMER_H
#define BBSTREAMER_H
+#include "common/backup_compression.h"
#include "lib/stringinfo.h"
#include "pqexpbuffer.h"
@@ -200,17 +201,17 @@ bbstreamer_buffer_until(bbstreamer *streamer, const char **data, int *len,
*/
extern bbstreamer *bbstreamer_plain_writer_new(char *pathname, FILE *file);
extern bbstreamer *bbstreamer_gzip_writer_new(char *pathname, FILE *file,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_extractor_new(const char *basepath,
const char *(*link_map) (const char *),
void (*report_output_file) (const char *));
extern bbstreamer *bbstreamer_gzip_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_lz4_compressor_new(bbstreamer *next,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_lz4_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_zstd_compressor_new(bbstreamer *next,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_zstd_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_tar_parser_new(bbstreamer *next);
extern bbstreamer *bbstreamer_tar_terminator_new(bbstreamer *next);
diff --git a/src/bin/pg_basebackup/bbstreamer_gzip.c b/src/bin/pg_basebackup/bbstreamer_gzip.c
index 894f857103..1979e95639 100644
--- a/src/bin/pg_basebackup/bbstreamer_gzip.c
+++ b/src/bin/pg_basebackup/bbstreamer_gzip.c
@@ -76,7 +76,8 @@ const bbstreamer_ops bbstreamer_gzip_decompressor_ops = {
* closed so that the data may be written there.
*/
bbstreamer *
-bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
+bbstreamer_gzip_writer_new(char *pathname, FILE *file,
+ bc_specification *compress)
{
#ifdef HAVE_LIBZ
bbstreamer_gzip_writer *streamer;
@@ -115,11 +116,11 @@ bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
}
}
- if (gzsetparams(streamer->gzfile, compresslevel,
+ if (gzsetparams(streamer->gzfile, compress->level,
Z_DEFAULT_STRATEGY) != Z_OK)
{
pg_log_error("could not set compression level %d: %s",
- compresslevel, get_gz_error(streamer->gzfile));
+ compress->level, get_gz_error(streamer->gzfile));
exit(1);
}
diff --git a/src/bin/pg_basebackup/bbstreamer_lz4.c b/src/bin/pg_basebackup/bbstreamer_lz4.c
index 810052e4e3..a6ec317e2b 100644
--- a/src/bin/pg_basebackup/bbstreamer_lz4.c
+++ b/src/bin/pg_basebackup/bbstreamer_lz4.c
@@ -67,7 +67,7 @@ const bbstreamer_ops bbstreamer_lz4_decompressor_ops = {
* blocks.
*/
bbstreamer *
-bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
+bbstreamer_lz4_compressor_new(bbstreamer *next, bc_specification *compress)
{
#ifdef USE_LZ4
bbstreamer_lz4_frame *streamer;
@@ -89,7 +89,7 @@ bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
prefs = &streamer->prefs;
memset(prefs, 0, sizeof(LZ4F_preferences_t));
prefs->frameInfo.blockSizeID = LZ4F_max256KB;
- prefs->compressionLevel = compresslevel;
+ prefs->compressionLevel = compress->level;
/*
* Find out the compression bound, it specifies the minimum destination
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index e86749a8fb..caa5edcaf1 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -63,7 +63,7 @@ const bbstreamer_ops bbstreamer_zstd_decompressor_ops = {
* blocks.
*/
bbstreamer *
-bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
+bbstreamer_zstd_compressor_new(bbstreamer *next, bc_specification *compress)
{
#ifdef USE_ZSTD
bbstreamer_zstd_frame *streamer;
@@ -85,7 +85,7 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
/* Initialize stream compression preferences */
ZSTD_CCtx_setParameter(streamer->cctx, ZSTD_c_compressionLevel,
- compresslevel);
+ compress->level);
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 2943d9ec1a..3e6977df1a 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -29,6 +29,7 @@
#include "access/xlog_internal.h"
#include "bbstreamer.h"
+#include "common/backup_compression.h"
#include "common/file_perm.h"
#include "common/file_utils.h"
#include "common/logging.h"
@@ -57,6 +58,7 @@ typedef struct TablespaceList
typedef struct ArchiveStreamState
{
int tablespacenum;
+ bc_specification *compress;
bbstreamer *streamer;
bbstreamer *manifest_inject_streamer;
PQExpBuffer manifest_buffer;
@@ -132,9 +134,6 @@ static bool checksum_failure = false;
static bool showprogress = false;
static bool estimatesize = true;
static int verbose = 0;
-static int compresslevel = 0;
-static WalCompressionMethod compressmethod = COMPRESSION_NONE;
-static CompressionLocation compressloc = COMPRESS_LOCATION_UNSPECIFIED;
static IncludeWal includewal = STREAM_WAL;
static bool fastcheckpoint = false;
static bool writerecoveryconf = false;
@@ -198,7 +197,8 @@ static void progress_report(int tablespacenum, bool force, bool finished);
static bbstreamer *CreateBackupStreamer(char *archive_name, char *spclocation,
bbstreamer **manifest_inject_streamer_p,
bool is_recovery_guc_supported,
- bool expect_unterminated_tarfile);
+ bool expect_unterminated_tarfile,
+ bc_specification *compress);
static void ReceiveArchiveStreamChunk(size_t r, char *copybuf,
void *callback_data);
static char GetCopyDataByte(size_t r, char *copybuf, size_t *cursor);
@@ -207,7 +207,7 @@ static uint64 GetCopyDataUInt64(size_t r, char *copybuf, size_t *cursor);
static void GetCopyDataEnd(size_t r, char *copybuf, size_t cursor);
static void ReportCopyDataParseError(size_t r, char *copybuf);
static void ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
- bool tablespacenum);
+ bool tablespacenum, bc_specification *compress);
static void ReceiveTarCopyChunk(size_t r, char *copybuf, void *callback_data);
static void ReceiveBackupManifest(PGconn *conn);
static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
@@ -215,7 +215,9 @@ static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
static void ReceiveBackupManifestInMemory(PGconn *conn, PQExpBuffer buf);
static void ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
void *callback_data);
-static void BaseBackup(void);
+static void BaseBackup(char *compression_algorithm, char *compression_detail,
+ CompressionLocation compressloc,
+ bc_specification *client_compress);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
bool segment_finished);
@@ -405,8 +407,8 @@ usage(void)
printf(_(" -X, --wal-method=none|fetch|stream\n"
" include required WAL files with specified method\n"));
printf(_(" -z, --gzip compress tar output\n"));
- printf(_(" -Z, --compress=[{client|server}-]{gzip|lz4|zstd}[:LEVEL]\n"
- " compress tar output with given compression method or level\n"));
+ printf(_(" -Z, --compress=[{client|server}-]METHOD[:DETAIL]\n"
+ " compress on client or server as specified\n"));
printf(_(" -Z, --compress=none do not compress tar output\n"));
printf(_("\nGeneral options:\n"));
printf(_(" -c, --checkpoint=fast|spread\n"
@@ -542,7 +544,9 @@ typedef struct
} logstreamer_param;
static int
-LogStreamerMain(logstreamer_param *param)
+LogStreamerMain(logstreamer_param *param,
+ WalCompressionMethod wal_compress_method,
+ int wal_compress_level)
{
StreamCtl stream;
@@ -565,25 +569,14 @@ LogStreamerMain(logstreamer_param *param)
stream.mark_done = true;
stream.partial_suffix = NULL;
stream.replication_slot = replication_slot;
-
if (format == 'p')
stream.walmethod = CreateWalDirectoryMethod(param->xlog,
COMPRESSION_NONE, 0,
stream.do_sync);
- else if (compressloc != COMPRESS_LOCATION_CLIENT)
- stream.walmethod = CreateWalTarMethod(param->xlog,
- COMPRESSION_NONE,
- compresslevel,
- stream.do_sync);
- else if (compressmethod == COMPRESSION_GZIP)
- stream.walmethod = CreateWalTarMethod(param->xlog,
- compressmethod,
- compresslevel,
- stream.do_sync);
else
stream.walmethod = CreateWalTarMethod(param->xlog,
- COMPRESSION_NONE,
- compresslevel,
+ wal_compress_method,
+ wal_compress_level,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
@@ -629,7 +622,9 @@ LogStreamerMain(logstreamer_param *param)
* stream the logfile in parallel with the backups.
*/
static void
-StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
+StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier,
+ WalCompressionMethod wal_compress_method,
+ int wal_compress_level)
{
logstreamer_param *param;
uint32 hi,
@@ -729,7 +724,7 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
int ret;
/* in child process */
- ret = LogStreamerMain(param);
+ ret = LogStreamerMain(param, wal_compress_method, wal_compress_level);
/* temp debugging aid to analyze 019_replslot_limit failures */
if (verbose)
@@ -1004,136 +999,81 @@ parse_max_rate(char *src)
}
/*
- * Utility wrapper to parse the values specified for -Z/--compress.
- * *methodres and *levelres will be optionally filled with values coming
- * from the parsed results.
+ * Basic parsing of a value specified for -Z/--compress.
+ *
+ * We're not concerned here with understanding exactly what behavior the
+ * user wants, but we do need to know whether the user is requesting client
+ * or server side compression or leaving it unspecified, and we need to
+ * separate the name of the compression algorithm from the detail string.
+ *
+ * For instance, if the user writes --compress client-lz4:6, we want to
+ * separate that into (a) client-side compression, (b) algorithm "lz4",
+ * and (c) detail "6". Note, however, that all the client/server prefix is
+ * optional, and so is the detail. The algorithm name is required, unless
+ * the whole string is an integer, in which case we assume "gzip" as the
+ * algorithm and use the integer as the detail.
+ *
+ * We're not concerned with validation at this stage, so if the user writes
+ * --compress client-turkey:sandwich, the requested algorithm is "turkey"
+ * and the detail string is "sandwich". We'll sort out whether that's legal
+ * at a later stage.
*/
static void
-parse_compress_options(char *src, WalCompressionMethod *methodres,
- CompressionLocation *locationres, int *levelres)
+parse_compress_options(char *option, char **algorithm, char **detail,
+ CompressionLocation *locationres)
{
char *sep;
- int firstlen;
- char *firstpart;
+ char *endp;
/*
- * clear 'levelres' so that if there are multiple compression options,
- * the last one fully overrides the earlier ones
- */
- *levelres = 0;
-
- /* check if the option is split in two */
- sep = strchr(src, ':');
-
- /*
- * The first part of the option value could be a method name, or just a
- * level value.
- */
- firstlen = (sep != NULL) ? (sep - src) : strlen(src);
- firstpart = pg_malloc(firstlen + 1);
- memcpy(firstpart, src, firstlen);
- firstpart[firstlen] = '\0';
-
- /*
- * Check if the first part of the string matches with a supported
- * compression method.
+ * Check whether the compression specification consists of a bare integer.
+ *
+ * If so, for backward compatibility, assume gzip.
*/
- if (pg_strcasecmp(firstpart, "gzip") == 0)
+ (void) strtol(option, &endp, 10);
+ if (*endp == '\0')
{
- *methodres = COMPRESSION_GZIP;
*locationres = COMPRESS_LOCATION_UNSPECIFIED;
+ *algorithm = pstrdup("gzip");
+ *detail = pstrdup(option);
+ return;
}
- else if (pg_strcasecmp(firstpart, "client-gzip") == 0)
- {
- *methodres = COMPRESSION_GZIP;
- *locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-gzip") == 0)
+
+ /* Strip off any "client-" or "server-" prefix. */
+ if (strncmp(option, "server-", 7) == 0)
{
- *methodres = COMPRESSION_GZIP;
*locationres = COMPRESS_LOCATION_SERVER;
+ option += 7;
}
- else if (pg_strcasecmp(firstpart, "lz4") == 0)
- {
- *methodres = COMPRESSION_LZ4;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
- }
- else if (pg_strcasecmp(firstpart, "client-lz4") == 0)
+ else if (strncmp(option, "client-", 7) == 0)
{
- *methodres = COMPRESSION_LZ4;
*locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-lz4") == 0)
- {
- *methodres = COMPRESSION_LZ4;
- *locationres = COMPRESS_LOCATION_SERVER;
- }
- else if (pg_strcasecmp(firstpart, "zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
- }
- else if (pg_strcasecmp(firstpart, "client-zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_SERVER;
- }
- else if (pg_strcasecmp(firstpart, "none") == 0)
- {
- *methodres = COMPRESSION_NONE;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
+ option += 7;
}
else
- {
- /*
- * It does not match anything known, so check for the
- * backward-compatible case of only an integer where the implied
- * compression method changes depending on the level value.
- */
- if (!option_parse_int(firstpart, "-Z/--compress", 0,
- INT_MAX, levelres))
- exit(1);
-
- *methodres = (*levelres > 0) ?
- COMPRESSION_GZIP : COMPRESSION_NONE;
*locationres = COMPRESS_LOCATION_UNSPECIFIED;
- free(firstpart);
- return;
- }
-
+ /*
+ * Check whether there is a compression detail following the algorithm
+ * name.
+ */
+ sep = strchr(option, ':');
if (sep == NULL)
{
- /*
- * The caller specified a method without a colon separator, so let any
- * subsequent checks assign a default level.
- */
- free(firstpart);
- return;
+ *algorithm = pstrdup(option);
+ *detail = NULL;
}
-
- /* Check the contents after the colon separator. */
- sep++;
- if (*sep == '\0')
+ else
{
- pg_log_error("no compression level defined for method %s", firstpart);
- exit(1);
- }
+ char *alg;
- /*
- * For any of the methods currently supported, the data after the
- * separator can just be an integer.
- */
- if (!option_parse_int(sep, "-Z/--compress", 0, INT_MAX,
- levelres))
- exit(1);
+ alg = palloc((sep - option) + 1);
+ memcpy(alg, option, sep - option);
+ alg[sep - option] = '\0';
- free(firstpart);
+ *algorithm = alg;
+ *detail = pstrdup(sep + 1);
+ }
}
/*
@@ -1200,7 +1140,8 @@ static bbstreamer *
CreateBackupStreamer(char *archive_name, char *spclocation,
bbstreamer **manifest_inject_streamer_p,
bool is_recovery_guc_supported,
- bool expect_unterminated_tarfile)
+ bool expect_unterminated_tarfile,
+ bc_specification *compress)
{
bbstreamer *streamer = NULL;
bbstreamer *manifest_inject_streamer = NULL;
@@ -1316,32 +1257,28 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
archive_file = NULL;
}
- if (compressmethod == COMPRESSION_NONE ||
- compressloc != COMPRESS_LOCATION_CLIENT)
+ if (compress->algorithm == BACKUP_COMPRESSION_NONE)
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- else if (compressmethod == COMPRESSION_GZIP)
+ else if (compress->algorithm == BACKUP_COMPRESSION_GZIP)
{
strlcat(archive_filename, ".gz", sizeof(archive_filename));
streamer = bbstreamer_gzip_writer_new(archive_filename,
- archive_file,
- compresslevel);
+ archive_file, compress);
}
- else if (compressmethod == COMPRESSION_LZ4)
+ else if (compress->algorithm == BACKUP_COMPRESSION_LZ4)
{
strlcat(archive_filename, ".lz4", sizeof(archive_filename));
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- streamer = bbstreamer_lz4_compressor_new(streamer,
- compresslevel);
+ streamer = bbstreamer_lz4_compressor_new(streamer, compress);
}
- else if (compressmethod == COMPRESSION_ZSTD)
+ else if (compress->algorithm == BACKUP_COMPRESSION_ZSTD)
{
strlcat(archive_filename, ".zst", sizeof(archive_filename));
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- streamer = bbstreamer_zstd_compressor_new(streamer,
- compresslevel);
+ streamer = bbstreamer_zstd_compressor_new(streamer, compress);
}
else
{
@@ -1395,13 +1332,13 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
* If the user has requested a server compressed archive along with archive
* extraction at client then we need to decompress it.
*/
- if (format == 'p' && compressloc == COMPRESS_LOCATION_SERVER)
+ if (format == 'p')
{
- if (compressmethod == COMPRESSION_GZIP)
+ if (is_tar_gz)
streamer = bbstreamer_gzip_decompressor_new(streamer);
- else if (compressmethod == COMPRESSION_LZ4)
+ else if (is_tar_lz4)
streamer = bbstreamer_lz4_decompressor_new(streamer);
- else if (compressmethod == COMPRESSION_ZSTD)
+ else if (is_tar_zstd)
streamer = bbstreamer_zstd_decompressor_new(streamer);
}
@@ -1415,13 +1352,14 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
* manifest if present - as a single COPY stream.
*/
static void
-ReceiveArchiveStream(PGconn *conn)
+ReceiveArchiveStream(PGconn *conn, bc_specification *compress)
{
ArchiveStreamState state;
/* Set up initial state. */
memset(&state, 0, sizeof(state));
state.tablespacenum = -1;
+ state.compress = compress;
/* All the real work happens in ReceiveArchiveStreamChunk. */
ReceiveCopyData(conn, ReceiveArchiveStreamChunk, &state);
@@ -1542,7 +1480,8 @@ ReceiveArchiveStreamChunk(size_t r, char *copybuf, void *callback_data)
CreateBackupStreamer(archive_name,
spclocation,
&state->manifest_inject_streamer,
- true, false);
+ true, false,
+ state->compress);
}
break;
}
@@ -1743,7 +1682,7 @@ ReportCopyDataParseError(size_t r, char *copybuf)
*/
static void
ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
- bool tablespacenum)
+ bool tablespacenum, bc_specification *compress)
{
WriteTarState state;
bbstreamer *manifest_inject_streamer;
@@ -1759,7 +1698,8 @@ ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
state.streamer = CreateBackupStreamer(archive_name, spclocation,
&manifest_inject_streamer,
is_recovery_guc_supported,
- expect_unterminated_tarfile);
+ expect_unterminated_tarfile,
+ compress);
state.tablespacenum = tablespacenum;
ReceiveCopyData(conn, ReceiveTarCopyChunk, &state);
progress_update_filename(NULL);
@@ -1902,7 +1842,8 @@ ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
}
static void
-BaseBackup(void)
+BaseBackup(char *compression_algorithm, char *compression_detail,
+ CompressionLocation compressloc, bc_specification *client_compress)
{
PGresult *res;
char *sysidentifier;
@@ -2055,33 +1996,17 @@ BaseBackup(void)
if (compressloc == COMPRESS_LOCATION_SERVER)
{
- char *compressmethodstr = NULL;
-
if (!use_new_option_syntax)
{
pg_log_error("server does not support server-side compression");
exit(1);
}
- switch (compressmethod)
- {
- case COMPRESSION_GZIP:
- compressmethodstr = "gzip";
- break;
- case COMPRESSION_LZ4:
- compressmethodstr = "lz4";
- break;
- case COMPRESSION_ZSTD:
- compressmethodstr = "zstd";
- break;
- default:
- Assert(false);
- break;
- }
AppendStringCommandOption(&buf, use_new_option_syntax,
- "COMPRESSION", compressmethodstr);
- if (compresslevel >= 1) /* not 0 or Z_DEFAULT_COMPRESSION */
- AppendIntegerCommandOption(&buf, use_new_option_syntax,
- "COMPRESSION_LEVEL", compresslevel);
+ "COMPRESSION", compression_algorithm);
+ if (compression_detail != NULL)
+ AppendStringCommandOption(&buf, use_new_option_syntax,
+ "COMPRESSION_DETAIL",
+ compression_detail);
}
if (verbose)
@@ -2207,15 +2132,33 @@ BaseBackup(void)
*/
if (includewal == STREAM_WAL)
{
+ WalCompressionMethod wal_compress_method;
+ int wal_compress_level;
+
if (verbose)
pg_log_info("starting background WAL receiver");
- StartLogStreamer(xlogstart, starttli, sysidentifier);
+
+ if (client_compress->algorithm == BACKUP_COMPRESSION_GZIP)
+ {
+ wal_compress_method = COMPRESSION_GZIP;
+ wal_compress_level =
+ (client_compress->options & BACKUP_COMPRESSION_OPTION_LEVEL)
+ != 0 ? client_compress->level : 0;
+ }
+ else
+ {
+ wal_compress_method = COMPRESSION_NONE;
+ wal_compress_level = 0;
+ }
+
+ StartLogStreamer(xlogstart, starttli, sysidentifier,
+ wal_compress_method, wal_compress_level);
}
if (serverMajor >= 1500)
{
/* Receive a single tar stream with everything. */
- ReceiveArchiveStream(conn);
+ ReceiveArchiveStream(conn, client_compress);
}
else
{
@@ -2244,7 +2187,8 @@ BaseBackup(void)
spclocation = PQgetvalue(res, i, 1);
}
- ReceiveTarFile(conn, archive_name, spclocation, i);
+ ReceiveTarFile(conn, archive_name, spclocation, i,
+ client_compress);
}
/*
@@ -2511,6 +2455,10 @@ main(int argc, char **argv)
int c;
int option_index;
+ char *compression_algorithm = "none";
+ char *compression_detail = NULL;
+ CompressionLocation compressloc = COMPRESS_LOCATION_UNSPECIFIED;
+ bc_specification client_compress;
pg_logging_init(argv[0]);
progname = get_progname(argv[0]);
@@ -2616,17 +2564,13 @@ main(int argc, char **argv)
do_sync = false;
break;
case 'z':
-#ifdef HAVE_LIBZ
- compresslevel = Z_DEFAULT_COMPRESSION;
-#else
- compresslevel = 1; /* will be rejected below */
-#endif
- compressmethod = COMPRESSION_GZIP;
+ compression_algorithm = "gzip";
+ compression_detail = NULL;
compressloc = COMPRESS_LOCATION_UNSPECIFIED;
break;
case 'Z':
- parse_compress_options(optarg, &compressmethod,
- &compressloc, &compresslevel);
+ parse_compress_options(optarg, &compression_algorithm,
+ &compression_detail, &compressloc);
break;
case 'c':
if (pg_strcasecmp(optarg, "fast") == 0)
@@ -2753,12 +2697,11 @@ main(int argc, char **argv)
}
/*
- * If we're compressing the backup and the user has not said where to
- * perform the compression, do it on the client, unless they specified
- * --target, in which case the server is the only choice.
+ * If the user has not specified where to perform backup compression,
+ * default to the client, unless the user specified --target, in which case
+ * the server is the only choice.
*/
- if (compressmethod != COMPRESSION_NONE &&
- compressloc == COMPRESS_LOCATION_UNSPECIFIED)
+ if (compressloc == COMPRESS_LOCATION_UNSPECIFIED)
{
if (backup_target == NULL)
compressloc = COMPRESS_LOCATION_CLIENT;
@@ -2766,6 +2709,40 @@ main(int argc, char **argv)
compressloc = COMPRESS_LOCATION_SERVER;
}
+ /*
+ * If any compression that we're doing is happening on the client side,
+ * we must try to parse the compression algorithm and detail, but if it's
+ * all on the server side, then we're just going to pass through whatever
+ * was requested and let the server decide what to do.
+ */
+ if (compressloc == COMPRESS_LOCATION_CLIENT)
+ {
+ bc_algorithm alg;
+ char *error_detail;
+
+ if (!parse_bc_algorithm(compression_algorithm, &alg))
+ {
+ pg_log_error("unrecognized compression algorithm \"%s\"",
+ compression_algorithm);
+ exit(1);
+ }
+
+ parse_bc_specification(alg, compression_detail, &client_compress);
+ error_detail = validate_bc_specification(&client_compress);
+ if (error_detail != NULL)
+ {
+ pg_log_error("invalid compression specification: %s",
+ error_detail);
+ exit(1);
+ }
+ }
+ else
+ {
+ Assert(compressloc == COMPRESS_LOCATION_SERVER);
+ client_compress.algorithm = BACKUP_COMPRESSION_NONE;
+ client_compress.options = 0;
+ }
+
/*
* Can't perform client-side compression if the backup is not being
* sent to the client.
@@ -2779,9 +2756,10 @@ main(int argc, char **argv)
}
/*
- * Compression doesn't make sense unless tar format is in use.
+ * Client-side compression doesn't make sense unless tar format is in use.
*/
- if (format == 'p' && compressloc == COMPRESS_LOCATION_CLIENT)
+ if (format == 'p' && compressloc == COMPRESS_LOCATION_CLIENT &&
+ client_compress.algorithm != BACKUP_COMPRESSION_NONE)
{
pg_log_error("only tar mode backups can be compressed");
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
@@ -2882,56 +2860,6 @@ main(int argc, char **argv)
}
}
- /* Sanity checks for compression-related options. */
- switch (compressmethod)
- {
- case COMPRESSION_NONE:
- if (compresslevel != 0)
- {
- pg_log_error("cannot use compression level with method %s",
- "none");
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
- progname);
- exit(1);
- }
- break;
- case COMPRESSION_GZIP:
- if (compresslevel > 9)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 9",
- compresslevel, "gzip");
- exit(1);
- }
- if (compressloc == COMPRESS_LOCATION_CLIENT)
- {
-#ifdef HAVE_LIBZ
- if (compresslevel == 0)
- compresslevel = Z_DEFAULT_COMPRESSION;
-#else
- pg_log_error("this build does not support compression with %s",
- "gzip");
- exit(1);
-#endif
- }
- break;
- case COMPRESSION_LZ4:
- if (compresslevel > 12)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 12",
- compresslevel, "lz4");
- exit(1);
- }
- break;
- case COMPRESSION_ZSTD:
- if (compresslevel > 22)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 22",
- compresslevel, "zstd");
- exit(1);
- }
- break;
- }
-
/*
* Sanity checks for progress reporting options.
*/
@@ -3040,7 +2968,8 @@ main(int argc, char **argv)
free(linkloc);
}
- BaseBackup();
+ BaseBackup(compression_algorithm, compression_detail, compressloc,
+ &client_compress);
success = true;
return 0;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index efefe947d9..2869a239e7 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -42,16 +42,12 @@ $node->command_fails(['pg_basebackup'],
# Sanity checks for options
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none:1' ],
- qr/\Qpg_basebackup: error: cannot use compression level with method none/,
+ qr/\Qcompression algorithm "none" does not accept a compression level/,
'failure if method "none" specified with compression level');
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none+' ],
- qr/\Qpg_basebackup: error: invalid value "none+" for option/,
+ qr/\Qunrecognized compression algorithm "none+"/,
'failure on incorrect separator to define compression level');
-$node->command_fails_like(
- [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none:' ],
- qr/\Qpg_basebackup: error: no compression level defined for method none/,
- 'failure on missing compression level value');
# Some Windows ANSI code pages may reject this filename, in which case we
# quietly proceed without this bit of test coverage.
@@ -89,6 +85,70 @@ print $conf "wal_level = replica\n";
close $conf;
$node->restart;
+# Now that we have a server that supports replication commands, test whether
+# certain invalid compression commands fail on the client side with client-side
+# compression and on the server side with server-side compression.
+my $client_fails =
+ 'pg_basebackup: error: ';
+my $server_fails =
+ 'pg_basebackup: error: could not initiate base backup: ERROR: ';
+my @compression_failure_tests = (
+ [
+ 'extrasquishy',
+ 'unrecognized compression algorithm "extrasquishy"',
+ 'failure on invalid compression algorithm'
+ ],
+ [
+ 'gzip:',
+ 'invalid compression specification: found empty string where a compression option was expected',
+ 'failure on empty compression options list'
+ ],
+ [
+ 'gzip:thunk',
+ 'invalid compression specification: unknown compression option "thunk"',
+ 'failure on unknown compression option'
+ ],
+ [
+ 'gzip:level',
+ 'invalid compression specification: compression option "level" requires a value',
+ 'failure on missing compression level'
+ ],
+ [
+ 'gzip:level=',
+ 'invalid compression specification: value for compression option "level" must be an integer',
+ 'failure on empty compression level'
+ ],
+ [
+ 'gzip:level=high',
+ 'invalid compression specification: value for compression option "level" must be an integer',
+ 'failure on non-numeric compression level'
+ ],
+ [
+ 'gzip:level=236',
+ 'invalid compression specification: compression algorithm "gzip" expects a compression level between 1 and 9',
+ 'failure on out-of-range compression level'
+ ],
+ [
+ 'gzip:level=9,',
+ 'invalid compression specification: found empty string where a compression option was expected',
+ 'failure on extra, empty compression option'
+ ],
+);
+for my $cft (@compression_failure_tests)
+{
+ my $cfail = quotemeta($client_fails . $cft->[1]);
+ my $sfail = quotemeta($server_fails . $cft->[1]);
+ $node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', $cft->[0] ],
+ qr/$cfail/,
+ 'client '. $cft->[2]);
+ $node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress',
+ 'server-' . $cft->[0] ],
+ qr/$sfail/,
+ 'server ' . $cft->[2]);
+}
+
# Write some files to test that they are not copied.
foreach my $filename (
qw(backup_label tablespace_map postgresql.auto.conf.tmp
diff --git a/src/common/Makefile b/src/common/Makefile
index 31c0dd366d..f627349835 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -47,6 +47,7 @@ LIBS += $(PTHREAD_LIBS)
OBJS_COMMON = \
archive.o \
+ backup_compression.o \
base64.o \
checksum_helper.o \
config_info.o \
diff --git a/src/common/backup_compression.c b/src/common/backup_compression.c
new file mode 100644
index 0000000000..fac5de157d
--- /dev/null
+++ b/src/common/backup_compression.c
@@ -0,0 +1,269 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_compression.c
+ *
+ * Shared code for backup compression methods and specifications.
+ *
+ * A compression specification specifies the parameters that should be used
+ * when performing compression with a specific algorithm. The simplest
+ * possible compression specification is an integer, which sets the
+ * compression level.
+ *
+ * Otherwise, a compression specification is a comma-separated list of items,
+ * each having the form keyword or keyword=value.
+ *
+ * Currently, the only supported keyword is "level".
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/backup_compression.c
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#include "common/backup_compression.h"
+
+static int expect_integer_value(char *keyword, char *value,
+ bc_specification *result);
+
+/*
+ * Look up a compression algorithm by name. Returns true and sets *algorithm
+ * if the name is recognized. Otherwise returns false.
+ */
+bool
+parse_bc_algorithm(char *name, bc_algorithm *algorithm)
+{
+ if (strcmp(name, "none") == 0)
+ *algorithm = BACKUP_COMPRESSION_NONE;
+ else if (strcmp(name, "gzip") == 0)
+ *algorithm = BACKUP_COMPRESSION_GZIP;
+ else if (strcmp(name, "lz4") == 0)
+ *algorithm = BACKUP_COMPRESSION_LZ4;
+ else if (strcmp(name, "zstd") == 0)
+ *algorithm = BACKUP_COMPRESSION_ZSTD;
+ else
+ return false;
+ return true;
+}
+
+/*
+ * Get the human-readable name corresponding to a particular compression
+ * algorithm.
+ */
+const char *
+get_bc_algorithm_name(bc_algorithm algorithm)
+{
+ switch (algorithm)
+ {
+ case BACKUP_COMPRESSION_NONE:
+ return "none";
+ case BACKUP_COMPRESSION_GZIP:
+ return "gzip";
+ case BACKUP_COMPRESSION_LZ4:
+ return "lz4";
+ case BACKUP_COMPRESSION_ZSTD:
+ return "zstd";
+ /* no default, to provoke compiler warnings if values are added */
+ }
+ Assert(false);
+}
+
+/*
+ * Parse a compression specification for a specified algorithm.
+ *
+ * See the file header comments for a brief description of what a compression
+ * specification is expected to look like.
+ *
+ * On return, all fields of the result object will be initialized.
+ * In particular, result->parse_error will be NULL if no errors occurred
+ * during parsing, and will otherwise contain an appropriate error message.
+ * The caller may free this error message string using pfree, if desired.
+ * Note, however, even if there's no parse error, the string might not make
+ * sense: e.g. for gzip, level=12 is not sensible, but it does parse OK.
+ *
+ * Use validate_bc_specification() to find out whether a compression
+ * specification is semantically sensible.
+ */
+void
+parse_bc_specification(bc_algorithm algorithm, char *specification,
+ bc_specification *result)
+{
+ int bare_level;
+ char *bare_level_endp;
+
+ /* Initial setup of result object. */
+ result->algorithm = algorithm;
+ result->options = 0;
+ result->level = -1;
+ result->parse_error = NULL;
+
+ /* If there is no specification, we're done already. */
+ if (specification == NULL)
+ return;
+
+ /* As a special case, the specification can be a bare integer. */
+ bare_level = strtol(specification, &bare_level_endp, 10);
+ if (specification != bare_level_endp && *bare_level_endp == '\0')
+ {
+ result->level = bare_level;
+ result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
+ return;
+ }
+
+ /* Look for comma-separated keyword or keyword=value entries. */
+ while (1)
+ {
+ char *kwstart;
+ char *kwend;
+ char *vstart;
+ char *vend;
+ int kwlen;
+ int vlen;
+ bool has_value;
+ char *keyword;
+ char *value;
+
+ /* Figure start, end, and length of next keyword and any value. */
+ kwstart = kwend = specification;
+ while (*kwend != '\0' && *kwend != ',' && *kwend != '=')
+ ++kwend;
+ kwlen = kwend - kwstart;
+ if (*kwend != '=')
+ {
+ vstart = vend = NULL;
+ vlen = 0;
+ has_value = false;
+ }
+ else
+ {
+ vstart = vend = kwend + 1;
+ while (*vend != '\0' && *vend != ',')
+ ++vend;
+ vlen = vend - vstart;
+ has_value = true;
+ }
+
+ /* Reject empty keyword. */
+ if (kwlen == 0)
+ {
+ result->parse_error =
+ pstrdup(_("found empty string where a compression option was expected"));
+ break;
+ }
+
+ /* Extract keyword and value as separate C strings. */
+ keyword = palloc(kwlen + 1);
+ memcpy(keyword, kwstart, kwlen);
+ keyword[kwlen] = '\0';
+ if (!has_value)
+ value = NULL;
+ else
+ {
+ value = palloc(vlen + 1);
+ memcpy(value, vstart, vlen);
+ value[vlen] = '\0';
+ }
+
+ /* Handle whatever keyword we found. */
+ if (strcmp(keyword, "level") == 0)
+ {
+ result->level = expect_integer_value(keyword, value, result);
+ result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
+ }
+ else
+ result->parse_error =
+ psprintf(_("unknown compression option \"%s\""), keyword);
+
+ /* Release memory, just to be tidy. */
+ pfree(keyword);
+ if (value != NULL)
+ pfree(value);
+
+ /* If we got an error or have reached the end of the string, stop. */
+ if (result->parse_error != NULL || *kwend == '\0' || *vend == '\0')
+ break;
+
+ /* Advance to next entry and loop around. */
+ specification = vend == NULL ? kwend + 1 : vend + 1;
+ }
+}
+
+/*
+ * Parse 'value' as an integer and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1.
+ */
+static int
+expect_integer_value(char *keyword, char *value, bc_specification *result)
+{
+ int ivalue;
+ char *ivalue_endp;
+
+ if (value == NULL)
+ {
+ result->parse_error =
+ psprintf(_("compression option \"%s\" requires a value"),
+ keyword);
+ return -1;
+ }
+
+ ivalue = strtol(value, &ivalue_endp, 10);
+ if (ivalue_endp == value || *ivalue_endp != '\0')
+ {
+ result->parse_error =
+ psprintf(_("value for compression option \"%s\" must be an integer"),
+ keyword);
+ return -1;
+ }
+ return ivalue;
+}
+
+/*
+ * Returns NULL if the compression specification string was syntactically
+ * valid and semantically sensible. Otherwise, returns an error message.
+ *
+ * Does not test whether this build of PostgreSQL supports the requested
+ * compression method.
+ */
+char *
+validate_bc_specification(bc_specification *spec)
+{
+ /* If it didn't even parse OK, it's definitely no good. */
+ if (spec->parse_error != NULL)
+ return spec->parse_error;
+
+ /*
+ * If a compression level was specified, check that the algorithm expects
+ * a compression level and that the level is within the legal range for
+ * the algorithm.
+ */
+ if ((spec->options & BACKUP_COMPRESSION_OPTION_LEVEL) != 0)
+ {
+ int min_level = 1;
+ int max_level;
+
+ if (spec->algorithm == BACKUP_COMPRESSION_GZIP)
+ max_level = 9;
+ else if (spec->algorithm == BACKUP_COMPRESSION_LZ4)
+ max_level = 12;
+ else if (spec->algorithm == BACKUP_COMPRESSION_ZSTD)
+ max_level = 22;
+ else
+ return psprintf(_("compression algorithm \"%s\" does not accept a compression level"),
+ get_bc_algorithm_name(spec->algorithm));
+
+ if (spec->level < min_level || spec->level > max_level)
+ return psprintf(_("compression algorithm \"%s\" expects a compression level between %d and %d"),
+ get_bc_algorithm_name(spec->algorithm),
+ min_level, max_level);
+ }
+
+ return NULL;
+}
diff --git a/src/include/common/backup_compression.h b/src/include/common/backup_compression.h
new file mode 100644
index 0000000000..0565cbc657
--- /dev/null
+++ b/src/include/common/backup_compression.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_compression.h
+ *
+ * Shared definitions for backup compression methods and specifications.
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/backup_compression.h
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef BACKUP_COMPRESSION_H
+#define BACKUP_COMPRESSION_H
+
+typedef enum bc_algorithm
+{
+ BACKUP_COMPRESSION_NONE,
+ BACKUP_COMPRESSION_GZIP,
+ BACKUP_COMPRESSION_LZ4,
+ BACKUP_COMPRESSION_ZSTD
+} bc_algorithm;
+
+#define BACKUP_COMPRESSION_OPTION_LEVEL (1 << 0)
+
+typedef struct bc_specification
+{
+ bc_algorithm algorithm;
+ unsigned options; /* OR of BACKUP_COMPRESSION_OPTION constants */
+ int level;
+ char *parse_error; /* NULL if parsing was OK, else message */
+} bc_specification;
+
+extern bool parse_bc_algorithm(char *name, bc_algorithm *algorithm);
+extern const char *get_bc_algorithm_name(bc_algorithm algorithm);
+
+extern void parse_bc_specification(bc_algorithm algorithm,
+ char *specification,
+ bc_specification *result);
+
+extern char *validate_bc_specification(bc_specification *);
+
+#endif
diff --git a/src/include/replication/basebackup_sink.h b/src/include/replication/basebackup_sink.h
index a7f16758a4..654df28576 100644
--- a/src/include/replication/basebackup_sink.h
+++ b/src/include/replication/basebackup_sink.h
@@ -27,6 +27,7 @@
#define BASEBACKUP_SINK_H
#include "access/xlog_internal.h"
+#include "common/backup_compression.h"
#include "nodes/pg_list.h"
/* Forward declarations. */
@@ -283,9 +284,9 @@ extern void bbsink_forward_cleanup(bbsink *sink);
/* Constructors for various types of sinks. */
extern bbsink *bbsink_copystream_new(bool send_to_client);
-extern bbsink *bbsink_gzip_new(bbsink *next, int compresslevel);
-extern bbsink *bbsink_lz4_new(bbsink *next, int compresslevel);
-extern bbsink *bbsink_zstd_new(bbsink *next, int compresslevel);
+extern bbsink *bbsink_gzip_new(bbsink *next, bc_specification *);
+extern bbsink *bbsink_lz4_new(bbsink *next, bc_specification *);
+extern bbsink *bbsink_zstd_new(bbsink *next, bc_specification *);
extern bbsink *bbsink_progress_new(bbsink *next, bool estimate_backup_size);
extern bbsink *bbsink_server_new(bbsink *next, char *pathname);
extern bbsink *bbsink_throttle_new(bbsink *next, uint32 maxrate);
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 441d6ae6bf..de8676d339 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -124,7 +124,7 @@ sub mkvcbuild
}
our @pgcommonallfiles = qw(
- archive.c base64.c checksum_helper.c
+ archive.c backup_compression.c base64.c checksum_helper.c
config_info.c controldata_utils.c d2s.c encnames.c exec.c
f2s.c file_perm.c file_utils.c hashfn.c ip.c jsonapi.c
keywords.c kwlookup.c link-canary.c md5_common.c
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 93d5190508..1f0d71bc68 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3795,3 +3795,5 @@ yyscan_t
z_stream
z_streamp
zic_t
+bc_algorithm
+bc_specification
--
2.24.3 (Apple Git-128)
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-21 18:41 ` Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 32+ messages in thread
From: Dagfinn Ilmari Mannsåker @ 2022-03-21 18:41 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Mar 21, 2022 at 2:22 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
>> + * during parsing, and will otherwise contain a an appropriate error message.
>
> OK, thanks. v4 attached.
I haven't read the whole patch, but I noticed an omission in the
documentation changes:
> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
> index 9178c779ba..00c593f1af 100644
> --- a/doc/src/sgml/protocol.sgml
> +++ b/doc/src/sgml/protocol.sgml
> @@ -2731,14 +2731,24 @@ The commands accepted in replication mode are:
> <varlistentry>
> - <term><literal>COMPRESSION_LEVEL</literal> <replaceable>level</replaceable></term>
> + <term><literal>COMPRESSION_DETAIL</literal> <replaceable>detail</replaceable></term>
> <listitem>
> <para>
> Specifies the compression level to be used.
This is no longer the accurate. How about something like like "Specifies
details of the chosen compression method"?
- ilmari
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
@ 2022-03-22 15:37 ` Robert Haas <robertmhaas@gmail.com>
2022-03-23 13:19 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 32+ messages in thread
From: Robert Haas @ 2022-03-22 15:37 UTC (permalink / raw)
To: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 21, 2022 at 2:41 PM Dagfinn Ilmari Mannsåker
<ilmari@ilmari.org> wrote:
> This is no longer the accurate. How about something like like "Specifies
> details of the chosen compression method"?
Good catch. v5 attached.
--
Robert Haas
EDB: http://www.enterprisedb.com
Attachments:
v5-0001-Replace-BASE_BACKUP-COMPRESSION_LEVEL-option-with.patch (application/octet-stream, 57.1K) download
From b7754037c9b5e4c3d8354d7eb919fe5583c08a1b Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Tue, 22 Mar 2022 11:31:17 -0400
Subject: [PATCH v5] Replace BASE_BACKUP COMPRESSION_LEVEL option with
COMPRESSION_DETAIL.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
There are more compression parameters that can be specified than just
an integer compression level, so rename the new COMPRESSION_LEVEL
option to COMPRESSION_DETAIL before it gets released. Introduce a
flexible syntax for that option to allow arbitrary options to be
specified without needing to adjust the main replication grammar,
and common code to parse it that is shared between the client and
the server.
This commit doesn't actually add any new compression parameters,
so the only user-visible change is that you can now type something
like pg_basebackup --compress gzip:level=5 instead of writing just
pg_basebackup --compress gzip:5. However, it should make it easy to
add new options. If for example gzip starts offering fries, we can
support pg_basebackup --compress gzip:level=5,fries=true for the
benefit of users who want fries with that.
Along the way, this fixes a few things in pg_basebackup so that the
pg_basebackup can be used with a server-side compression algorithm
that pg_basebackup itself does not understand. For example,
pg_basebackup --compress server-lz4 could still succeed even if
only the server and not the client has LZ4 support, provided that
the other options to pg_basebackup don't require the client to
decompress the archive.
Patch by me. Reviewed by Justin Pryzby and Dagfinn Ilmari Mannsåker.
Discussion: http://postgr.es/m/CA+TgmoYvpetyRAbbg1M8b3-iHsaN4nsgmWPjOENu5-doHuJ7fA@mail.gmail.com
---
doc/src/sgml/protocol.sgml | 22 +-
doc/src/sgml/ref/pg_basebackup.sgml | 25 +-
src/backend/replication/basebackup.c | 62 +--
src/backend/replication/basebackup_gzip.c | 20 +-
src/backend/replication/basebackup_lz4.c | 19 +-
src/backend/replication/basebackup_zstd.c | 19 +-
src/bin/pg_basebackup/bbstreamer.h | 7 +-
src/bin/pg_basebackup/bbstreamer_gzip.c | 7 +-
src/bin/pg_basebackup/bbstreamer_lz4.c | 4 +-
src/bin/pg_basebackup/bbstreamer_zstd.c | 4 +-
src/bin/pg_basebackup/pg_basebackup.c | 409 ++++++++-----------
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 72 +++-
src/common/Makefile | 1 +
src/common/backup_compression.c | 269 ++++++++++++
src/include/common/backup_compression.h | 44 ++
src/include/replication/basebackup_sink.h | 7 +-
src/tools/msvc/Mkvcbuild.pm | 2 +-
src/tools/pgindent/typedefs.list | 2 +
18 files changed, 664 insertions(+), 331 deletions(-)
create mode 100644 src/common/backup_compression.c
create mode 100644 src/include/common/backup_compression.h
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9178c779ba..719b947ef4 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2731,14 +2731,24 @@ The commands accepted in replication mode are:
</varlistentry>
<varlistentry>
- <term><literal>COMPRESSION_LEVEL</literal> <replaceable>level</replaceable></term>
+ <term><literal>COMPRESSION_DETAIL</literal> <replaceable>detail</replaceable></term>
<listitem>
<para>
- Specifies the compression level to be used. This should only be
- used in conjunction with the <literal>COMPRESSION</literal> option.
- For <literal>gzip</literal> the value should be an integer between 1
- and 9, for <literal>lz4</literal> between 1 and 12, and for
- <literal>zstd</literal> it should be between 1 and 22.
+ Specifies details for the chosen compression method. This should only
+ be used in conjunction with the <literal>COMPRESSION</literal>
+ option. If the value is an integer, it specifies the compression
+ level. Otherwise, it should be a comma-separated list of items,
+ each of the form <literal>keyword</literal> or
+ <literal>keyword=value</literal>. Currently, the only supported
+ keyword is <literal>level</literal>, which sets the compression
+ level.
+ </para>
+
+ <para>
+ For <literal>gzip</literal> the compression level should be an
+ integer between 1 and 9, for <literal>lz4</literal> an integer
+ between 1 and 12, and for <literal>zstd</literal> an integer
+ between 1 and 22.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 4a630b59b7..46d7f15e54 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -399,9 +399,9 @@ PostgreSQL documentation
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
- <term><option>-Z [{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>level</replaceable>]</term>
+ <term><option>-Z [{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>detail</replaceable>]</term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
- <term><option>--compress=[{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>level</replaceable>]</term>
+ <term><option>--compress=[{client|server}-]<replaceable class="parameter">method[:<replaceable>detail</replaceable>]</replaceable></option></term>
<listitem>
<para>
Requests compression of the backup. If <literal>client</literal> or
@@ -419,13 +419,20 @@ PostgreSQL documentation
<para>
The compression method can be set to <literal>gzip</literal>,
<literal>lz4</literal>, <literal>zstd</literal>, or
- <literal>none</literal> for no compression. A compression level can
- optionally be specified, by appending the level number after a colon
- (<literal>:</literal>). If no level is specified, the default
- compression level will be used. If only a level is specified without
- mentioning an algorithm, <literal>gzip</literal> compression will be
- used if the level is greater than 0, and no compression will be used if
- the level is 0.
+ <literal>none</literal> for no compression. A compression detail
+ string can optionally be specified. If the detail string is an
+ integer, it specifies the compression level. Otherwise, it should be
+ a comma-separated list of items, each of the form
+ <literal>keyword</literal> or <literal>keyword=value</literal>.
+ Currently, the only supported keyword is <literal>level</literal>,
+ which sets the compression level.
+ </para>
+ <para>
+ If no compression level is specified, the default compression level
+ will be used. If only a level is specified without mentioning an
+ algorithm, <literal>gzip</literal> compression will be used if the
+ level is greater than 0, and no compression will be used if the level
+ is 0.
</para>
<para>
When the tar format is used with <literal>gzip</literal>,
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index c2aedc14a2..49deead091 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h" /* for pg_start/stop_backup */
+#include "common/backup_compression.h"
#include "common/file_perm.h"
#include "commands/defrem.h"
#include "lib/stringinfo.h"
@@ -54,14 +55,6 @@
*/
#define SINK_BUFFER_LENGTH Max(32768, BLCKSZ)
-typedef enum
-{
- BACKUP_COMPRESSION_NONE,
- BACKUP_COMPRESSION_GZIP,
- BACKUP_COMPRESSION_LZ4,
- BACKUP_COMPRESSION_ZSTD
-} basebackup_compression_type;
-
typedef struct
{
const char *label;
@@ -75,8 +68,8 @@ typedef struct
bool use_copytblspc;
BaseBackupTargetHandle *target_handle;
backup_manifest_option manifest;
- basebackup_compression_type compression;
- int compression_level;
+ bc_algorithm compression;
+ bc_specification compression_specification;
pg_checksum_type manifest_checksum_type;
} basebackup_options;
@@ -713,12 +706,14 @@ parse_basebackup_options(List *options, basebackup_options *opt)
char *target_str = NULL;
char *target_detail_str = NULL;
bool o_compression = false;
- bool o_compression_level = false;
+ bool o_compression_detail = false;
+ char *compression_detail_str = NULL;
MemSet(opt, 0, sizeof(*opt));
opt->manifest = MANIFEST_OPTION_NO;
opt->manifest_checksum_type = CHECKSUM_TYPE_CRC32C;
opt->compression = BACKUP_COMPRESSION_NONE;
+ opt->compression_specification.algorithm = BACKUP_COMPRESSION_NONE;
foreach(lopt, options)
{
@@ -885,29 +880,21 @@ parse_basebackup_options(List *options, basebackup_options *opt)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("duplicate option \"%s\"", defel->defname)));
- if (strcmp(optval, "none") == 0)
- opt->compression = BACKUP_COMPRESSION_NONE;
- else if (strcmp(optval, "gzip") == 0)
- opt->compression = BACKUP_COMPRESSION_GZIP;
- else if (strcmp(optval, "lz4") == 0)
- opt->compression = BACKUP_COMPRESSION_LZ4;
- else if (strcmp(optval, "zstd") == 0)
- opt->compression = BACKUP_COMPRESSION_ZSTD;
- else
+ if (!parse_bc_algorithm(optval, &opt->compression))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("unrecognized compression algorithm: \"%s\"",
+ errmsg("unrecognized compression algorithm \"%s\"",
optval)));
o_compression = true;
}
- else if (strcmp(defel->defname, "compression_level") == 0)
+ else if (strcmp(defel->defname, "compression_detail") == 0)
{
- if (o_compression_level)
+ if (o_compression_detail)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("duplicate option \"%s\"", defel->defname)));
- opt->compression_level = defGetInt32(defel);
- o_compression_level = true;
+ compression_detail_str = defGetString(defel);
+ o_compression_detail = true;
}
else
ereport(ERROR,
@@ -949,10 +936,25 @@ parse_basebackup_options(List *options, basebackup_options *opt)
opt->target_handle =
BaseBackupGetTargetHandle(target_str, target_detail_str);
- if (o_compression_level && !o_compression)
+ if (o_compression_detail && !o_compression)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("compression level requires compression")));
+ errmsg("compression detail requires compression")));
+
+ if (o_compression)
+ {
+ char *error_detail;
+
+ parse_bc_specification(opt->compression, compression_detail_str,
+ &opt->compression_specification);
+ error_detail =
+ validate_bc_specification(&opt->compression_specification);
+ if (error_detail != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("invalid compression specification: %s",
+ error_detail));
+ }
}
@@ -998,11 +1000,11 @@ SendBaseBackup(BaseBackupCmd *cmd)
/* Set up server-side compression, if client requested it */
if (opt.compression == BACKUP_COMPRESSION_GZIP)
- sink = bbsink_gzip_new(sink, opt.compression_level);
+ sink = bbsink_gzip_new(sink, &opt.compression_specification);
else if (opt.compression == BACKUP_COMPRESSION_LZ4)
- sink = bbsink_lz4_new(sink, opt.compression_level);
+ sink = bbsink_lz4_new(sink, &opt.compression_specification);
else if (opt.compression == BACKUP_COMPRESSION_ZSTD)
- sink = bbsink_zstd_new(sink, opt.compression_level);
+ sink = bbsink_zstd_new(sink, &opt.compression_specification);
/* Set up progress reporting. */
sink = bbsink_progress_new(sink, opt.progress);
diff --git a/src/backend/replication/basebackup_gzip.c b/src/backend/replication/basebackup_gzip.c
index b66d3da7a3..703a91ba77 100644
--- a/src/backend/replication/basebackup_gzip.c
+++ b/src/backend/replication/basebackup_gzip.c
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_gzip_ops = {
#endif
/*
- * Create a new basebackup sink that performs gzip compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs gzip compression.
*/
bbsink *
-bbsink_gzip_new(bbsink *next, int compresslevel)
+bbsink_gzip_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef HAVE_LIBZ
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,15 +72,14 @@ bbsink_gzip_new(bbsink *next, int compresslevel)
bbsink_gzip *sink;
Assert(next != NULL);
- Assert(compresslevel >= 0 && compresslevel <= 9);
- if (compresslevel == 0)
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
compresslevel = Z_DEFAULT_COMPRESSION;
- else if (compresslevel < 0 || compresslevel > 9)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("gzip compression level %d is out of range",
- compresslevel)));
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 9);
+ }
sink = palloc0(sizeof(bbsink_gzip));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_gzip_ops;
diff --git a/src/backend/replication/basebackup_lz4.c b/src/backend/replication/basebackup_lz4.c
index d838f723d0..06c161ddc4 100644
--- a/src/backend/replication/basebackup_lz4.c
+++ b/src/backend/replication/basebackup_lz4.c
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_lz4_ops = {
#endif
/*
- * Create a new basebackup sink that performs lz4 compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs lz4 compression.
*/
bbsink *
-bbsink_lz4_new(bbsink *next, int compresslevel)
+bbsink_lz4_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef USE_LZ4
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -72,11 +73,13 @@ bbsink_lz4_new(bbsink *next, int compresslevel)
Assert(next != NULL);
- if (compresslevel < 0 || compresslevel > 12)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("lz4 compression level %d is out of range",
- compresslevel)));
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
+ compresslevel = 0;
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 12);
+ }
sink = palloc0(sizeof(bbsink_lz4));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_lz4_ops;
diff --git a/src/backend/replication/basebackup_zstd.c b/src/backend/replication/basebackup_zstd.c
index c0e2be6e27..96b7985693 100644
--- a/src/backend/replication/basebackup_zstd.c
+++ b/src/backend/replication/basebackup_zstd.c
@@ -55,12 +55,13 @@ const bbsink_ops bbsink_zstd_ops = {
#endif
/*
- * Create a new basebackup sink that performs zstd compression using the
- * designated compression level.
+ * Create a new basebackup sink that performs zstd compression.
*/
bbsink *
-bbsink_zstd_new(bbsink *next, int compresslevel)
+bbsink_zstd_new(bbsink *next, bc_specification *compress)
{
+ int compresslevel;
+
#ifndef USE_ZSTD
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,11 +72,13 @@ bbsink_zstd_new(bbsink *next, int compresslevel)
Assert(next != NULL);
- if (compresslevel < 0 || compresslevel > 22)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("zstd compression level %d is out of range",
- compresslevel)));
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
+ compresslevel = 0;
+ else
+ {
+ compresslevel = compress->level;
+ Assert(compresslevel >= 1 && compresslevel <= 22);
+ }
sink = palloc0(sizeof(bbsink_zstd));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_zstd_ops;
diff --git a/src/bin/pg_basebackup/bbstreamer.h b/src/bin/pg_basebackup/bbstreamer.h
index 02d4c05df6..dfa3f77af4 100644
--- a/src/bin/pg_basebackup/bbstreamer.h
+++ b/src/bin/pg_basebackup/bbstreamer.h
@@ -22,6 +22,7 @@
#ifndef BBSTREAMER_H
#define BBSTREAMER_H
+#include "common/backup_compression.h"
#include "lib/stringinfo.h"
#include "pqexpbuffer.h"
@@ -200,17 +201,17 @@ bbstreamer_buffer_until(bbstreamer *streamer, const char **data, int *len,
*/
extern bbstreamer *bbstreamer_plain_writer_new(char *pathname, FILE *file);
extern bbstreamer *bbstreamer_gzip_writer_new(char *pathname, FILE *file,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_extractor_new(const char *basepath,
const char *(*link_map) (const char *),
void (*report_output_file) (const char *));
extern bbstreamer *bbstreamer_gzip_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_lz4_compressor_new(bbstreamer *next,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_lz4_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_zstd_compressor_new(bbstreamer *next,
- int compresslevel);
+ bc_specification *compress);
extern bbstreamer *bbstreamer_zstd_decompressor_new(bbstreamer *next);
extern bbstreamer *bbstreamer_tar_parser_new(bbstreamer *next);
extern bbstreamer *bbstreamer_tar_terminator_new(bbstreamer *next);
diff --git a/src/bin/pg_basebackup/bbstreamer_gzip.c b/src/bin/pg_basebackup/bbstreamer_gzip.c
index 894f857103..1979e95639 100644
--- a/src/bin/pg_basebackup/bbstreamer_gzip.c
+++ b/src/bin/pg_basebackup/bbstreamer_gzip.c
@@ -76,7 +76,8 @@ const bbstreamer_ops bbstreamer_gzip_decompressor_ops = {
* closed so that the data may be written there.
*/
bbstreamer *
-bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
+bbstreamer_gzip_writer_new(char *pathname, FILE *file,
+ bc_specification *compress)
{
#ifdef HAVE_LIBZ
bbstreamer_gzip_writer *streamer;
@@ -115,11 +116,11 @@ bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
}
}
- if (gzsetparams(streamer->gzfile, compresslevel,
+ if (gzsetparams(streamer->gzfile, compress->level,
Z_DEFAULT_STRATEGY) != Z_OK)
{
pg_log_error("could not set compression level %d: %s",
- compresslevel, get_gz_error(streamer->gzfile));
+ compress->level, get_gz_error(streamer->gzfile));
exit(1);
}
diff --git a/src/bin/pg_basebackup/bbstreamer_lz4.c b/src/bin/pg_basebackup/bbstreamer_lz4.c
index 810052e4e3..a6ec317e2b 100644
--- a/src/bin/pg_basebackup/bbstreamer_lz4.c
+++ b/src/bin/pg_basebackup/bbstreamer_lz4.c
@@ -67,7 +67,7 @@ const bbstreamer_ops bbstreamer_lz4_decompressor_ops = {
* blocks.
*/
bbstreamer *
-bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
+bbstreamer_lz4_compressor_new(bbstreamer *next, bc_specification *compress)
{
#ifdef USE_LZ4
bbstreamer_lz4_frame *streamer;
@@ -89,7 +89,7 @@ bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
prefs = &streamer->prefs;
memset(prefs, 0, sizeof(LZ4F_preferences_t));
prefs->frameInfo.blockSizeID = LZ4F_max256KB;
- prefs->compressionLevel = compresslevel;
+ prefs->compressionLevel = compress->level;
/*
* Find out the compression bound, it specifies the minimum destination
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index e86749a8fb..caa5edcaf1 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -63,7 +63,7 @@ const bbstreamer_ops bbstreamer_zstd_decompressor_ops = {
* blocks.
*/
bbstreamer *
-bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
+bbstreamer_zstd_compressor_new(bbstreamer *next, bc_specification *compress)
{
#ifdef USE_ZSTD
bbstreamer_zstd_frame *streamer;
@@ -85,7 +85,7 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
/* Initialize stream compression preferences */
ZSTD_CCtx_setParameter(streamer->cctx, ZSTD_c_compressionLevel,
- compresslevel);
+ compress->level);
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 2943d9ec1a..3e6977df1a 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -29,6 +29,7 @@
#include "access/xlog_internal.h"
#include "bbstreamer.h"
+#include "common/backup_compression.h"
#include "common/file_perm.h"
#include "common/file_utils.h"
#include "common/logging.h"
@@ -57,6 +58,7 @@ typedef struct TablespaceList
typedef struct ArchiveStreamState
{
int tablespacenum;
+ bc_specification *compress;
bbstreamer *streamer;
bbstreamer *manifest_inject_streamer;
PQExpBuffer manifest_buffer;
@@ -132,9 +134,6 @@ static bool checksum_failure = false;
static bool showprogress = false;
static bool estimatesize = true;
static int verbose = 0;
-static int compresslevel = 0;
-static WalCompressionMethod compressmethod = COMPRESSION_NONE;
-static CompressionLocation compressloc = COMPRESS_LOCATION_UNSPECIFIED;
static IncludeWal includewal = STREAM_WAL;
static bool fastcheckpoint = false;
static bool writerecoveryconf = false;
@@ -198,7 +197,8 @@ static void progress_report(int tablespacenum, bool force, bool finished);
static bbstreamer *CreateBackupStreamer(char *archive_name, char *spclocation,
bbstreamer **manifest_inject_streamer_p,
bool is_recovery_guc_supported,
- bool expect_unterminated_tarfile);
+ bool expect_unterminated_tarfile,
+ bc_specification *compress);
static void ReceiveArchiveStreamChunk(size_t r, char *copybuf,
void *callback_data);
static char GetCopyDataByte(size_t r, char *copybuf, size_t *cursor);
@@ -207,7 +207,7 @@ static uint64 GetCopyDataUInt64(size_t r, char *copybuf, size_t *cursor);
static void GetCopyDataEnd(size_t r, char *copybuf, size_t cursor);
static void ReportCopyDataParseError(size_t r, char *copybuf);
static void ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
- bool tablespacenum);
+ bool tablespacenum, bc_specification *compress);
static void ReceiveTarCopyChunk(size_t r, char *copybuf, void *callback_data);
static void ReceiveBackupManifest(PGconn *conn);
static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
@@ -215,7 +215,9 @@ static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
static void ReceiveBackupManifestInMemory(PGconn *conn, PQExpBuffer buf);
static void ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
void *callback_data);
-static void BaseBackup(void);
+static void BaseBackup(char *compression_algorithm, char *compression_detail,
+ CompressionLocation compressloc,
+ bc_specification *client_compress);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
bool segment_finished);
@@ -405,8 +407,8 @@ usage(void)
printf(_(" -X, --wal-method=none|fetch|stream\n"
" include required WAL files with specified method\n"));
printf(_(" -z, --gzip compress tar output\n"));
- printf(_(" -Z, --compress=[{client|server}-]{gzip|lz4|zstd}[:LEVEL]\n"
- " compress tar output with given compression method or level\n"));
+ printf(_(" -Z, --compress=[{client|server}-]METHOD[:DETAIL]\n"
+ " compress on client or server as specified\n"));
printf(_(" -Z, --compress=none do not compress tar output\n"));
printf(_("\nGeneral options:\n"));
printf(_(" -c, --checkpoint=fast|spread\n"
@@ -542,7 +544,9 @@ typedef struct
} logstreamer_param;
static int
-LogStreamerMain(logstreamer_param *param)
+LogStreamerMain(logstreamer_param *param,
+ WalCompressionMethod wal_compress_method,
+ int wal_compress_level)
{
StreamCtl stream;
@@ -565,25 +569,14 @@ LogStreamerMain(logstreamer_param *param)
stream.mark_done = true;
stream.partial_suffix = NULL;
stream.replication_slot = replication_slot;
-
if (format == 'p')
stream.walmethod = CreateWalDirectoryMethod(param->xlog,
COMPRESSION_NONE, 0,
stream.do_sync);
- else if (compressloc != COMPRESS_LOCATION_CLIENT)
- stream.walmethod = CreateWalTarMethod(param->xlog,
- COMPRESSION_NONE,
- compresslevel,
- stream.do_sync);
- else if (compressmethod == COMPRESSION_GZIP)
- stream.walmethod = CreateWalTarMethod(param->xlog,
- compressmethod,
- compresslevel,
- stream.do_sync);
else
stream.walmethod = CreateWalTarMethod(param->xlog,
- COMPRESSION_NONE,
- compresslevel,
+ wal_compress_method,
+ wal_compress_level,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
@@ -629,7 +622,9 @@ LogStreamerMain(logstreamer_param *param)
* stream the logfile in parallel with the backups.
*/
static void
-StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
+StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier,
+ WalCompressionMethod wal_compress_method,
+ int wal_compress_level)
{
logstreamer_param *param;
uint32 hi,
@@ -729,7 +724,7 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
int ret;
/* in child process */
- ret = LogStreamerMain(param);
+ ret = LogStreamerMain(param, wal_compress_method, wal_compress_level);
/* temp debugging aid to analyze 019_replslot_limit failures */
if (verbose)
@@ -1004,136 +999,81 @@ parse_max_rate(char *src)
}
/*
- * Utility wrapper to parse the values specified for -Z/--compress.
- * *methodres and *levelres will be optionally filled with values coming
- * from the parsed results.
+ * Basic parsing of a value specified for -Z/--compress.
+ *
+ * We're not concerned here with understanding exactly what behavior the
+ * user wants, but we do need to know whether the user is requesting client
+ * or server side compression or leaving it unspecified, and we need to
+ * separate the name of the compression algorithm from the detail string.
+ *
+ * For instance, if the user writes --compress client-lz4:6, we want to
+ * separate that into (a) client-side compression, (b) algorithm "lz4",
+ * and (c) detail "6". Note, however, that all the client/server prefix is
+ * optional, and so is the detail. The algorithm name is required, unless
+ * the whole string is an integer, in which case we assume "gzip" as the
+ * algorithm and use the integer as the detail.
+ *
+ * We're not concerned with validation at this stage, so if the user writes
+ * --compress client-turkey:sandwich, the requested algorithm is "turkey"
+ * and the detail string is "sandwich". We'll sort out whether that's legal
+ * at a later stage.
*/
static void
-parse_compress_options(char *src, WalCompressionMethod *methodres,
- CompressionLocation *locationres, int *levelres)
+parse_compress_options(char *option, char **algorithm, char **detail,
+ CompressionLocation *locationres)
{
char *sep;
- int firstlen;
- char *firstpart;
+ char *endp;
/*
- * clear 'levelres' so that if there are multiple compression options,
- * the last one fully overrides the earlier ones
- */
- *levelres = 0;
-
- /* check if the option is split in two */
- sep = strchr(src, ':');
-
- /*
- * The first part of the option value could be a method name, or just a
- * level value.
- */
- firstlen = (sep != NULL) ? (sep - src) : strlen(src);
- firstpart = pg_malloc(firstlen + 1);
- memcpy(firstpart, src, firstlen);
- firstpart[firstlen] = '\0';
-
- /*
- * Check if the first part of the string matches with a supported
- * compression method.
+ * Check whether the compression specification consists of a bare integer.
+ *
+ * If so, for backward compatibility, assume gzip.
*/
- if (pg_strcasecmp(firstpart, "gzip") == 0)
+ (void) strtol(option, &endp, 10);
+ if (*endp == '\0')
{
- *methodres = COMPRESSION_GZIP;
*locationres = COMPRESS_LOCATION_UNSPECIFIED;
+ *algorithm = pstrdup("gzip");
+ *detail = pstrdup(option);
+ return;
}
- else if (pg_strcasecmp(firstpart, "client-gzip") == 0)
- {
- *methodres = COMPRESSION_GZIP;
- *locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-gzip") == 0)
+
+ /* Strip off any "client-" or "server-" prefix. */
+ if (strncmp(option, "server-", 7) == 0)
{
- *methodres = COMPRESSION_GZIP;
*locationres = COMPRESS_LOCATION_SERVER;
+ option += 7;
}
- else if (pg_strcasecmp(firstpart, "lz4") == 0)
- {
- *methodres = COMPRESSION_LZ4;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
- }
- else if (pg_strcasecmp(firstpart, "client-lz4") == 0)
+ else if (strncmp(option, "client-", 7) == 0)
{
- *methodres = COMPRESSION_LZ4;
*locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-lz4") == 0)
- {
- *methodres = COMPRESSION_LZ4;
- *locationres = COMPRESS_LOCATION_SERVER;
- }
- else if (pg_strcasecmp(firstpart, "zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
- }
- else if (pg_strcasecmp(firstpart, "client-zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_CLIENT;
- }
- else if (pg_strcasecmp(firstpart, "server-zstd") == 0)
- {
- *methodres = COMPRESSION_ZSTD;
- *locationres = COMPRESS_LOCATION_SERVER;
- }
- else if (pg_strcasecmp(firstpart, "none") == 0)
- {
- *methodres = COMPRESSION_NONE;
- *locationres = COMPRESS_LOCATION_UNSPECIFIED;
+ option += 7;
}
else
- {
- /*
- * It does not match anything known, so check for the
- * backward-compatible case of only an integer where the implied
- * compression method changes depending on the level value.
- */
- if (!option_parse_int(firstpart, "-Z/--compress", 0,
- INT_MAX, levelres))
- exit(1);
-
- *methodres = (*levelres > 0) ?
- COMPRESSION_GZIP : COMPRESSION_NONE;
*locationres = COMPRESS_LOCATION_UNSPECIFIED;
- free(firstpart);
- return;
- }
-
+ /*
+ * Check whether there is a compression detail following the algorithm
+ * name.
+ */
+ sep = strchr(option, ':');
if (sep == NULL)
{
- /*
- * The caller specified a method without a colon separator, so let any
- * subsequent checks assign a default level.
- */
- free(firstpart);
- return;
+ *algorithm = pstrdup(option);
+ *detail = NULL;
}
-
- /* Check the contents after the colon separator. */
- sep++;
- if (*sep == '\0')
+ else
{
- pg_log_error("no compression level defined for method %s", firstpart);
- exit(1);
- }
+ char *alg;
- /*
- * For any of the methods currently supported, the data after the
- * separator can just be an integer.
- */
- if (!option_parse_int(sep, "-Z/--compress", 0, INT_MAX,
- levelres))
- exit(1);
+ alg = palloc((sep - option) + 1);
+ memcpy(alg, option, sep - option);
+ alg[sep - option] = '\0';
- free(firstpart);
+ *algorithm = alg;
+ *detail = pstrdup(sep + 1);
+ }
}
/*
@@ -1200,7 +1140,8 @@ static bbstreamer *
CreateBackupStreamer(char *archive_name, char *spclocation,
bbstreamer **manifest_inject_streamer_p,
bool is_recovery_guc_supported,
- bool expect_unterminated_tarfile)
+ bool expect_unterminated_tarfile,
+ bc_specification *compress)
{
bbstreamer *streamer = NULL;
bbstreamer *manifest_inject_streamer = NULL;
@@ -1316,32 +1257,28 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
archive_file = NULL;
}
- if (compressmethod == COMPRESSION_NONE ||
- compressloc != COMPRESS_LOCATION_CLIENT)
+ if (compress->algorithm == BACKUP_COMPRESSION_NONE)
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- else if (compressmethod == COMPRESSION_GZIP)
+ else if (compress->algorithm == BACKUP_COMPRESSION_GZIP)
{
strlcat(archive_filename, ".gz", sizeof(archive_filename));
streamer = bbstreamer_gzip_writer_new(archive_filename,
- archive_file,
- compresslevel);
+ archive_file, compress);
}
- else if (compressmethod == COMPRESSION_LZ4)
+ else if (compress->algorithm == BACKUP_COMPRESSION_LZ4)
{
strlcat(archive_filename, ".lz4", sizeof(archive_filename));
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- streamer = bbstreamer_lz4_compressor_new(streamer,
- compresslevel);
+ streamer = bbstreamer_lz4_compressor_new(streamer, compress);
}
- else if (compressmethod == COMPRESSION_ZSTD)
+ else if (compress->algorithm == BACKUP_COMPRESSION_ZSTD)
{
strlcat(archive_filename, ".zst", sizeof(archive_filename));
streamer = bbstreamer_plain_writer_new(archive_filename,
archive_file);
- streamer = bbstreamer_zstd_compressor_new(streamer,
- compresslevel);
+ streamer = bbstreamer_zstd_compressor_new(streamer, compress);
}
else
{
@@ -1395,13 +1332,13 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
* If the user has requested a server compressed archive along with archive
* extraction at client then we need to decompress it.
*/
- if (format == 'p' && compressloc == COMPRESS_LOCATION_SERVER)
+ if (format == 'p')
{
- if (compressmethod == COMPRESSION_GZIP)
+ if (is_tar_gz)
streamer = bbstreamer_gzip_decompressor_new(streamer);
- else if (compressmethod == COMPRESSION_LZ4)
+ else if (is_tar_lz4)
streamer = bbstreamer_lz4_decompressor_new(streamer);
- else if (compressmethod == COMPRESSION_ZSTD)
+ else if (is_tar_zstd)
streamer = bbstreamer_zstd_decompressor_new(streamer);
}
@@ -1415,13 +1352,14 @@ CreateBackupStreamer(char *archive_name, char *spclocation,
* manifest if present - as a single COPY stream.
*/
static void
-ReceiveArchiveStream(PGconn *conn)
+ReceiveArchiveStream(PGconn *conn, bc_specification *compress)
{
ArchiveStreamState state;
/* Set up initial state. */
memset(&state, 0, sizeof(state));
state.tablespacenum = -1;
+ state.compress = compress;
/* All the real work happens in ReceiveArchiveStreamChunk. */
ReceiveCopyData(conn, ReceiveArchiveStreamChunk, &state);
@@ -1542,7 +1480,8 @@ ReceiveArchiveStreamChunk(size_t r, char *copybuf, void *callback_data)
CreateBackupStreamer(archive_name,
spclocation,
&state->manifest_inject_streamer,
- true, false);
+ true, false,
+ state->compress);
}
break;
}
@@ -1743,7 +1682,7 @@ ReportCopyDataParseError(size_t r, char *copybuf)
*/
static void
ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
- bool tablespacenum)
+ bool tablespacenum, bc_specification *compress)
{
WriteTarState state;
bbstreamer *manifest_inject_streamer;
@@ -1759,7 +1698,8 @@ ReceiveTarFile(PGconn *conn, char *archive_name, char *spclocation,
state.streamer = CreateBackupStreamer(archive_name, spclocation,
&manifest_inject_streamer,
is_recovery_guc_supported,
- expect_unterminated_tarfile);
+ expect_unterminated_tarfile,
+ compress);
state.tablespacenum = tablespacenum;
ReceiveCopyData(conn, ReceiveTarCopyChunk, &state);
progress_update_filename(NULL);
@@ -1902,7 +1842,8 @@ ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
}
static void
-BaseBackup(void)
+BaseBackup(char *compression_algorithm, char *compression_detail,
+ CompressionLocation compressloc, bc_specification *client_compress)
{
PGresult *res;
char *sysidentifier;
@@ -2055,33 +1996,17 @@ BaseBackup(void)
if (compressloc == COMPRESS_LOCATION_SERVER)
{
- char *compressmethodstr = NULL;
-
if (!use_new_option_syntax)
{
pg_log_error("server does not support server-side compression");
exit(1);
}
- switch (compressmethod)
- {
- case COMPRESSION_GZIP:
- compressmethodstr = "gzip";
- break;
- case COMPRESSION_LZ4:
- compressmethodstr = "lz4";
- break;
- case COMPRESSION_ZSTD:
- compressmethodstr = "zstd";
- break;
- default:
- Assert(false);
- break;
- }
AppendStringCommandOption(&buf, use_new_option_syntax,
- "COMPRESSION", compressmethodstr);
- if (compresslevel >= 1) /* not 0 or Z_DEFAULT_COMPRESSION */
- AppendIntegerCommandOption(&buf, use_new_option_syntax,
- "COMPRESSION_LEVEL", compresslevel);
+ "COMPRESSION", compression_algorithm);
+ if (compression_detail != NULL)
+ AppendStringCommandOption(&buf, use_new_option_syntax,
+ "COMPRESSION_DETAIL",
+ compression_detail);
}
if (verbose)
@@ -2207,15 +2132,33 @@ BaseBackup(void)
*/
if (includewal == STREAM_WAL)
{
+ WalCompressionMethod wal_compress_method;
+ int wal_compress_level;
+
if (verbose)
pg_log_info("starting background WAL receiver");
- StartLogStreamer(xlogstart, starttli, sysidentifier);
+
+ if (client_compress->algorithm == BACKUP_COMPRESSION_GZIP)
+ {
+ wal_compress_method = COMPRESSION_GZIP;
+ wal_compress_level =
+ (client_compress->options & BACKUP_COMPRESSION_OPTION_LEVEL)
+ != 0 ? client_compress->level : 0;
+ }
+ else
+ {
+ wal_compress_method = COMPRESSION_NONE;
+ wal_compress_level = 0;
+ }
+
+ StartLogStreamer(xlogstart, starttli, sysidentifier,
+ wal_compress_method, wal_compress_level);
}
if (serverMajor >= 1500)
{
/* Receive a single tar stream with everything. */
- ReceiveArchiveStream(conn);
+ ReceiveArchiveStream(conn, client_compress);
}
else
{
@@ -2244,7 +2187,8 @@ BaseBackup(void)
spclocation = PQgetvalue(res, i, 1);
}
- ReceiveTarFile(conn, archive_name, spclocation, i);
+ ReceiveTarFile(conn, archive_name, spclocation, i,
+ client_compress);
}
/*
@@ -2511,6 +2455,10 @@ main(int argc, char **argv)
int c;
int option_index;
+ char *compression_algorithm = "none";
+ char *compression_detail = NULL;
+ CompressionLocation compressloc = COMPRESS_LOCATION_UNSPECIFIED;
+ bc_specification client_compress;
pg_logging_init(argv[0]);
progname = get_progname(argv[0]);
@@ -2616,17 +2564,13 @@ main(int argc, char **argv)
do_sync = false;
break;
case 'z':
-#ifdef HAVE_LIBZ
- compresslevel = Z_DEFAULT_COMPRESSION;
-#else
- compresslevel = 1; /* will be rejected below */
-#endif
- compressmethod = COMPRESSION_GZIP;
+ compression_algorithm = "gzip";
+ compression_detail = NULL;
compressloc = COMPRESS_LOCATION_UNSPECIFIED;
break;
case 'Z':
- parse_compress_options(optarg, &compressmethod,
- &compressloc, &compresslevel);
+ parse_compress_options(optarg, &compression_algorithm,
+ &compression_detail, &compressloc);
break;
case 'c':
if (pg_strcasecmp(optarg, "fast") == 0)
@@ -2753,12 +2697,11 @@ main(int argc, char **argv)
}
/*
- * If we're compressing the backup and the user has not said where to
- * perform the compression, do it on the client, unless they specified
- * --target, in which case the server is the only choice.
+ * If the user has not specified where to perform backup compression,
+ * default to the client, unless the user specified --target, in which case
+ * the server is the only choice.
*/
- if (compressmethod != COMPRESSION_NONE &&
- compressloc == COMPRESS_LOCATION_UNSPECIFIED)
+ if (compressloc == COMPRESS_LOCATION_UNSPECIFIED)
{
if (backup_target == NULL)
compressloc = COMPRESS_LOCATION_CLIENT;
@@ -2766,6 +2709,40 @@ main(int argc, char **argv)
compressloc = COMPRESS_LOCATION_SERVER;
}
+ /*
+ * If any compression that we're doing is happening on the client side,
+ * we must try to parse the compression algorithm and detail, but if it's
+ * all on the server side, then we're just going to pass through whatever
+ * was requested and let the server decide what to do.
+ */
+ if (compressloc == COMPRESS_LOCATION_CLIENT)
+ {
+ bc_algorithm alg;
+ char *error_detail;
+
+ if (!parse_bc_algorithm(compression_algorithm, &alg))
+ {
+ pg_log_error("unrecognized compression algorithm \"%s\"",
+ compression_algorithm);
+ exit(1);
+ }
+
+ parse_bc_specification(alg, compression_detail, &client_compress);
+ error_detail = validate_bc_specification(&client_compress);
+ if (error_detail != NULL)
+ {
+ pg_log_error("invalid compression specification: %s",
+ error_detail);
+ exit(1);
+ }
+ }
+ else
+ {
+ Assert(compressloc == COMPRESS_LOCATION_SERVER);
+ client_compress.algorithm = BACKUP_COMPRESSION_NONE;
+ client_compress.options = 0;
+ }
+
/*
* Can't perform client-side compression if the backup is not being
* sent to the client.
@@ -2779,9 +2756,10 @@ main(int argc, char **argv)
}
/*
- * Compression doesn't make sense unless tar format is in use.
+ * Client-side compression doesn't make sense unless tar format is in use.
*/
- if (format == 'p' && compressloc == COMPRESS_LOCATION_CLIENT)
+ if (format == 'p' && compressloc == COMPRESS_LOCATION_CLIENT &&
+ client_compress.algorithm != BACKUP_COMPRESSION_NONE)
{
pg_log_error("only tar mode backups can be compressed");
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
@@ -2882,56 +2860,6 @@ main(int argc, char **argv)
}
}
- /* Sanity checks for compression-related options. */
- switch (compressmethod)
- {
- case COMPRESSION_NONE:
- if (compresslevel != 0)
- {
- pg_log_error("cannot use compression level with method %s",
- "none");
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
- progname);
- exit(1);
- }
- break;
- case COMPRESSION_GZIP:
- if (compresslevel > 9)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 9",
- compresslevel, "gzip");
- exit(1);
- }
- if (compressloc == COMPRESS_LOCATION_CLIENT)
- {
-#ifdef HAVE_LIBZ
- if (compresslevel == 0)
- compresslevel = Z_DEFAULT_COMPRESSION;
-#else
- pg_log_error("this build does not support compression with %s",
- "gzip");
- exit(1);
-#endif
- }
- break;
- case COMPRESSION_LZ4:
- if (compresslevel > 12)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 12",
- compresslevel, "lz4");
- exit(1);
- }
- break;
- case COMPRESSION_ZSTD:
- if (compresslevel > 22)
- {
- pg_log_error("compression level %d of method %s higher than maximum of 22",
- compresslevel, "zstd");
- exit(1);
- }
- break;
- }
-
/*
* Sanity checks for progress reporting options.
*/
@@ -3040,7 +2968,8 @@ main(int argc, char **argv)
free(linkloc);
}
- BaseBackup();
+ BaseBackup(compression_algorithm, compression_detail, compressloc,
+ &client_compress);
success = true;
return 0;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index efefe947d9..2869a239e7 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -42,16 +42,12 @@ $node->command_fails(['pg_basebackup'],
# Sanity checks for options
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none:1' ],
- qr/\Qpg_basebackup: error: cannot use compression level with method none/,
+ qr/\Qcompression algorithm "none" does not accept a compression level/,
'failure if method "none" specified with compression level');
$node->command_fails_like(
[ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none+' ],
- qr/\Qpg_basebackup: error: invalid value "none+" for option/,
+ qr/\Qunrecognized compression algorithm "none+"/,
'failure on incorrect separator to define compression level');
-$node->command_fails_like(
- [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', 'none:' ],
- qr/\Qpg_basebackup: error: no compression level defined for method none/,
- 'failure on missing compression level value');
# Some Windows ANSI code pages may reject this filename, in which case we
# quietly proceed without this bit of test coverage.
@@ -89,6 +85,70 @@ print $conf "wal_level = replica\n";
close $conf;
$node->restart;
+# Now that we have a server that supports replication commands, test whether
+# certain invalid compression commands fail on the client side with client-side
+# compression and on the server side with server-side compression.
+my $client_fails =
+ 'pg_basebackup: error: ';
+my $server_fails =
+ 'pg_basebackup: error: could not initiate base backup: ERROR: ';
+my @compression_failure_tests = (
+ [
+ 'extrasquishy',
+ 'unrecognized compression algorithm "extrasquishy"',
+ 'failure on invalid compression algorithm'
+ ],
+ [
+ 'gzip:',
+ 'invalid compression specification: found empty string where a compression option was expected',
+ 'failure on empty compression options list'
+ ],
+ [
+ 'gzip:thunk',
+ 'invalid compression specification: unknown compression option "thunk"',
+ 'failure on unknown compression option'
+ ],
+ [
+ 'gzip:level',
+ 'invalid compression specification: compression option "level" requires a value',
+ 'failure on missing compression level'
+ ],
+ [
+ 'gzip:level=',
+ 'invalid compression specification: value for compression option "level" must be an integer',
+ 'failure on empty compression level'
+ ],
+ [
+ 'gzip:level=high',
+ 'invalid compression specification: value for compression option "level" must be an integer',
+ 'failure on non-numeric compression level'
+ ],
+ [
+ 'gzip:level=236',
+ 'invalid compression specification: compression algorithm "gzip" expects a compression level between 1 and 9',
+ 'failure on out-of-range compression level'
+ ],
+ [
+ 'gzip:level=9,',
+ 'invalid compression specification: found empty string where a compression option was expected',
+ 'failure on extra, empty compression option'
+ ],
+);
+for my $cft (@compression_failure_tests)
+{
+ my $cfail = quotemeta($client_fails . $cft->[1]);
+ my $sfail = quotemeta($server_fails . $cft->[1]);
+ $node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress', $cft->[0] ],
+ qr/$cfail/,
+ 'client '. $cft->[2]);
+ $node->command_fails_like(
+ [ 'pg_basebackup', '-D', "$tempdir/backup", '--compress',
+ 'server-' . $cft->[0] ],
+ qr/$sfail/,
+ 'server ' . $cft->[2]);
+}
+
# Write some files to test that they are not copied.
foreach my $filename (
qw(backup_label tablespace_map postgresql.auto.conf.tmp
diff --git a/src/common/Makefile b/src/common/Makefile
index 31c0dd366d..f627349835 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -47,6 +47,7 @@ LIBS += $(PTHREAD_LIBS)
OBJS_COMMON = \
archive.o \
+ backup_compression.o \
base64.o \
checksum_helper.o \
config_info.o \
diff --git a/src/common/backup_compression.c b/src/common/backup_compression.c
new file mode 100644
index 0000000000..fac5de157d
--- /dev/null
+++ b/src/common/backup_compression.c
@@ -0,0 +1,269 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_compression.c
+ *
+ * Shared code for backup compression methods and specifications.
+ *
+ * A compression specification specifies the parameters that should be used
+ * when performing compression with a specific algorithm. The simplest
+ * possible compression specification is an integer, which sets the
+ * compression level.
+ *
+ * Otherwise, a compression specification is a comma-separated list of items,
+ * each having the form keyword or keyword=value.
+ *
+ * Currently, the only supported keyword is "level".
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/backup_compression.c
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#include "common/backup_compression.h"
+
+static int expect_integer_value(char *keyword, char *value,
+ bc_specification *result);
+
+/*
+ * Look up a compression algorithm by name. Returns true and sets *algorithm
+ * if the name is recognized. Otherwise returns false.
+ */
+bool
+parse_bc_algorithm(char *name, bc_algorithm *algorithm)
+{
+ if (strcmp(name, "none") == 0)
+ *algorithm = BACKUP_COMPRESSION_NONE;
+ else if (strcmp(name, "gzip") == 0)
+ *algorithm = BACKUP_COMPRESSION_GZIP;
+ else if (strcmp(name, "lz4") == 0)
+ *algorithm = BACKUP_COMPRESSION_LZ4;
+ else if (strcmp(name, "zstd") == 0)
+ *algorithm = BACKUP_COMPRESSION_ZSTD;
+ else
+ return false;
+ return true;
+}
+
+/*
+ * Get the human-readable name corresponding to a particular compression
+ * algorithm.
+ */
+const char *
+get_bc_algorithm_name(bc_algorithm algorithm)
+{
+ switch (algorithm)
+ {
+ case BACKUP_COMPRESSION_NONE:
+ return "none";
+ case BACKUP_COMPRESSION_GZIP:
+ return "gzip";
+ case BACKUP_COMPRESSION_LZ4:
+ return "lz4";
+ case BACKUP_COMPRESSION_ZSTD:
+ return "zstd";
+ /* no default, to provoke compiler warnings if values are added */
+ }
+ Assert(false);
+}
+
+/*
+ * Parse a compression specification for a specified algorithm.
+ *
+ * See the file header comments for a brief description of what a compression
+ * specification is expected to look like.
+ *
+ * On return, all fields of the result object will be initialized.
+ * In particular, result->parse_error will be NULL if no errors occurred
+ * during parsing, and will otherwise contain an appropriate error message.
+ * The caller may free this error message string using pfree, if desired.
+ * Note, however, even if there's no parse error, the string might not make
+ * sense: e.g. for gzip, level=12 is not sensible, but it does parse OK.
+ *
+ * Use validate_bc_specification() to find out whether a compression
+ * specification is semantically sensible.
+ */
+void
+parse_bc_specification(bc_algorithm algorithm, char *specification,
+ bc_specification *result)
+{
+ int bare_level;
+ char *bare_level_endp;
+
+ /* Initial setup of result object. */
+ result->algorithm = algorithm;
+ result->options = 0;
+ result->level = -1;
+ result->parse_error = NULL;
+
+ /* If there is no specification, we're done already. */
+ if (specification == NULL)
+ return;
+
+ /* As a special case, the specification can be a bare integer. */
+ bare_level = strtol(specification, &bare_level_endp, 10);
+ if (specification != bare_level_endp && *bare_level_endp == '\0')
+ {
+ result->level = bare_level;
+ result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
+ return;
+ }
+
+ /* Look for comma-separated keyword or keyword=value entries. */
+ while (1)
+ {
+ char *kwstart;
+ char *kwend;
+ char *vstart;
+ char *vend;
+ int kwlen;
+ int vlen;
+ bool has_value;
+ char *keyword;
+ char *value;
+
+ /* Figure start, end, and length of next keyword and any value. */
+ kwstart = kwend = specification;
+ while (*kwend != '\0' && *kwend != ',' && *kwend != '=')
+ ++kwend;
+ kwlen = kwend - kwstart;
+ if (*kwend != '=')
+ {
+ vstart = vend = NULL;
+ vlen = 0;
+ has_value = false;
+ }
+ else
+ {
+ vstart = vend = kwend + 1;
+ while (*vend != '\0' && *vend != ',')
+ ++vend;
+ vlen = vend - vstart;
+ has_value = true;
+ }
+
+ /* Reject empty keyword. */
+ if (kwlen == 0)
+ {
+ result->parse_error =
+ pstrdup(_("found empty string where a compression option was expected"));
+ break;
+ }
+
+ /* Extract keyword and value as separate C strings. */
+ keyword = palloc(kwlen + 1);
+ memcpy(keyword, kwstart, kwlen);
+ keyword[kwlen] = '\0';
+ if (!has_value)
+ value = NULL;
+ else
+ {
+ value = palloc(vlen + 1);
+ memcpy(value, vstart, vlen);
+ value[vlen] = '\0';
+ }
+
+ /* Handle whatever keyword we found. */
+ if (strcmp(keyword, "level") == 0)
+ {
+ result->level = expect_integer_value(keyword, value, result);
+ result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
+ }
+ else
+ result->parse_error =
+ psprintf(_("unknown compression option \"%s\""), keyword);
+
+ /* Release memory, just to be tidy. */
+ pfree(keyword);
+ if (value != NULL)
+ pfree(value);
+
+ /* If we got an error or have reached the end of the string, stop. */
+ if (result->parse_error != NULL || *kwend == '\0' || *vend == '\0')
+ break;
+
+ /* Advance to next entry and loop around. */
+ specification = vend == NULL ? kwend + 1 : vend + 1;
+ }
+}
+
+/*
+ * Parse 'value' as an integer and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1.
+ */
+static int
+expect_integer_value(char *keyword, char *value, bc_specification *result)
+{
+ int ivalue;
+ char *ivalue_endp;
+
+ if (value == NULL)
+ {
+ result->parse_error =
+ psprintf(_("compression option \"%s\" requires a value"),
+ keyword);
+ return -1;
+ }
+
+ ivalue = strtol(value, &ivalue_endp, 10);
+ if (ivalue_endp == value || *ivalue_endp != '\0')
+ {
+ result->parse_error =
+ psprintf(_("value for compression option \"%s\" must be an integer"),
+ keyword);
+ return -1;
+ }
+ return ivalue;
+}
+
+/*
+ * Returns NULL if the compression specification string was syntactically
+ * valid and semantically sensible. Otherwise, returns an error message.
+ *
+ * Does not test whether this build of PostgreSQL supports the requested
+ * compression method.
+ */
+char *
+validate_bc_specification(bc_specification *spec)
+{
+ /* If it didn't even parse OK, it's definitely no good. */
+ if (spec->parse_error != NULL)
+ return spec->parse_error;
+
+ /*
+ * If a compression level was specified, check that the algorithm expects
+ * a compression level and that the level is within the legal range for
+ * the algorithm.
+ */
+ if ((spec->options & BACKUP_COMPRESSION_OPTION_LEVEL) != 0)
+ {
+ int min_level = 1;
+ int max_level;
+
+ if (spec->algorithm == BACKUP_COMPRESSION_GZIP)
+ max_level = 9;
+ else if (spec->algorithm == BACKUP_COMPRESSION_LZ4)
+ max_level = 12;
+ else if (spec->algorithm == BACKUP_COMPRESSION_ZSTD)
+ max_level = 22;
+ else
+ return psprintf(_("compression algorithm \"%s\" does not accept a compression level"),
+ get_bc_algorithm_name(spec->algorithm));
+
+ if (spec->level < min_level || spec->level > max_level)
+ return psprintf(_("compression algorithm \"%s\" expects a compression level between %d and %d"),
+ get_bc_algorithm_name(spec->algorithm),
+ min_level, max_level);
+ }
+
+ return NULL;
+}
diff --git a/src/include/common/backup_compression.h b/src/include/common/backup_compression.h
new file mode 100644
index 0000000000..0565cbc657
--- /dev/null
+++ b/src/include/common/backup_compression.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_compression.h
+ *
+ * Shared definitions for backup compression methods and specifications.
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/backup_compression.h
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef BACKUP_COMPRESSION_H
+#define BACKUP_COMPRESSION_H
+
+typedef enum bc_algorithm
+{
+ BACKUP_COMPRESSION_NONE,
+ BACKUP_COMPRESSION_GZIP,
+ BACKUP_COMPRESSION_LZ4,
+ BACKUP_COMPRESSION_ZSTD
+} bc_algorithm;
+
+#define BACKUP_COMPRESSION_OPTION_LEVEL (1 << 0)
+
+typedef struct bc_specification
+{
+ bc_algorithm algorithm;
+ unsigned options; /* OR of BACKUP_COMPRESSION_OPTION constants */
+ int level;
+ char *parse_error; /* NULL if parsing was OK, else message */
+} bc_specification;
+
+extern bool parse_bc_algorithm(char *name, bc_algorithm *algorithm);
+extern const char *get_bc_algorithm_name(bc_algorithm algorithm);
+
+extern void parse_bc_specification(bc_algorithm algorithm,
+ char *specification,
+ bc_specification *result);
+
+extern char *validate_bc_specification(bc_specification *);
+
+#endif
diff --git a/src/include/replication/basebackup_sink.h b/src/include/replication/basebackup_sink.h
index a7f16758a4..654df28576 100644
--- a/src/include/replication/basebackup_sink.h
+++ b/src/include/replication/basebackup_sink.h
@@ -27,6 +27,7 @@
#define BASEBACKUP_SINK_H
#include "access/xlog_internal.h"
+#include "common/backup_compression.h"
#include "nodes/pg_list.h"
/* Forward declarations. */
@@ -283,9 +284,9 @@ extern void bbsink_forward_cleanup(bbsink *sink);
/* Constructors for various types of sinks. */
extern bbsink *bbsink_copystream_new(bool send_to_client);
-extern bbsink *bbsink_gzip_new(bbsink *next, int compresslevel);
-extern bbsink *bbsink_lz4_new(bbsink *next, int compresslevel);
-extern bbsink *bbsink_zstd_new(bbsink *next, int compresslevel);
+extern bbsink *bbsink_gzip_new(bbsink *next, bc_specification *);
+extern bbsink *bbsink_lz4_new(bbsink *next, bc_specification *);
+extern bbsink *bbsink_zstd_new(bbsink *next, bc_specification *);
extern bbsink *bbsink_progress_new(bbsink *next, bool estimate_backup_size);
extern bbsink *bbsink_server_new(bbsink *next, char *pathname);
extern bbsink *bbsink_throttle_new(bbsink *next, uint32 maxrate);
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 441d6ae6bf..de8676d339 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -124,7 +124,7 @@ sub mkvcbuild
}
our @pgcommonallfiles = qw(
- archive.c base64.c checksum_helper.c
+ archive.c backup_compression.c base64.c checksum_helper.c
config_info.c controldata_utils.c d2s.c encnames.c exec.c
f2s.c file_perm.c file_utils.c hashfn.c ip.c jsonapi.c
keywords.c kwlookup.c link-canary.c md5_common.c
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 93d5190508..1f0d71bc68 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3795,3 +3795,5 @@ yyscan_t
z_stream
z_streamp
zic_t
+bc_algorithm
+bc_specification
--
2.24.3 (Apple Git-128)
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-23 13:19 ` Robert Haas <robertmhaas@gmail.com>
1 sibling, 0 replies; 32+ messages in thread
From: Robert Haas @ 2022-03-23 13:19 UTC (permalink / raw)
To: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Tue, Mar 22, 2022 at 11:37 AM Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Mar 21, 2022 at 2:41 PM Dagfinn Ilmari Mannsåker
> <ilmari@ilmari.org> wrote:
> > This is no longer the accurate. How about something like like "Specifies
> > details of the chosen compression method"?
>
> Good catch. v5 attached.
And committed.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-27 17:47 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 19:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 32+ messages in thread
From: Tom Lane @ 2022-03-27 17:47 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; Justin Pryzby <pryzby@telsasoft.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
Robert Haas <robertmhaas@gmail.com> writes:
> [ v5-0001-Replace-BASE_BACKUP-COMPRESSION_LEVEL-option-with.patch ]
Coverity has a nitpick about this:
/srv/coverity/git/pgsql-git/postgresql/src/common/backup_compression.c: 194 in parse_bc_specification()
193 /* Advance to next entry and loop around. */
>>> CID 1503251: Null pointer dereferences (REVERSE_INULL)
>>> Null-checking "vend" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.
194 specification = vend == NULL ? kwend + 1 : vend + 1;
195 }
196 }
Not sure if you should remove this null-check or add some other ones,
but I think you ought to do one or the other.
regards, tom lane
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-28 19:50 ` Robert Haas <robertmhaas@gmail.com>
2022-03-28 20:38 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 20:53 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 2 replies; 32+ messages in thread
From: Robert Haas @ 2022-03-28 19:50 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; Justin Pryzby <pryzby@telsasoft.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Sun, Mar 27, 2022 at 1:47 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Coverity has a nitpick about this:
>
> /srv/coverity/git/pgsql-git/postgresql/src/common/backup_compression.c: 194 in parse_bc_specification()
> 193 /* Advance to next entry and loop around. */
> >>> CID 1503251: Null pointer dereferences (REVERSE_INULL)
> >>> Null-checking "vend" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.
> 194 specification = vend == NULL ? kwend + 1 : vend + 1;
> 195 }
> 196 }
>
> Not sure if you should remove this null-check or add some other ones,
> but I think you ought to do one or the other.
Yes, I think this is buggy. I think there's only a theoretical bug
right now, because the only keyword we have is "level" and that
requires a value. But if I add an example keyword that does not
require an associated value (as demonstrated in the attached patch)
and do something like pg_basebackup -cfast -D whatever --compress
lz4:example, then the present code will dereference "vend" even though
it's NULL, which is not good. The attached patch also shows how I
think that should be fixed.
As I hope is apparent, the first hunk of this patch is not for commit,
and the second hunk is for commit.
--
Robert Haas
EDB: http://www.enterprisedb.com
Attachments:
coverity-backup-compression-fix.patch (application/octet-stream, 998B) download
diff --git a/src/common/backup_compression.c b/src/common/backup_compression.c
index 0650f975c4..8eb670848b 100644
--- a/src/common/backup_compression.c
+++ b/src/common/backup_compression.c
@@ -177,6 +177,10 @@ parse_bc_specification(bc_algorithm algorithm, char *specification,
result->level = expect_integer_value(keyword, value, result);
result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
}
+ else if (strcmp(keyword, "example") == 0)
+ {
+ /* this is just an example */
+ }
else
result->parse_error =
psprintf(_("unknown compression option \"%s\""), keyword);
@@ -187,7 +191,8 @@ parse_bc_specification(bc_algorithm algorithm, char *specification,
pfree(value);
/* If we got an error or have reached the end of the string, stop. */
- if (result->parse_error != NULL || *kwend == '\0' || *vend == '\0')
+ if (result->parse_error != NULL ||
+ (vend == NULL ? *kwend == '\0' : *vend == '\0'))
break;
/* Advance to next entry and loop around. */
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 19:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-28 20:38 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 32+ messages in thread
From: Tom Lane @ 2022-03-28 20:38 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; Justin Pryzby <pryzby@telsasoft.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
Robert Haas <robertmhaas@gmail.com> writes:
> On Sun, Mar 27, 2022 at 1:47 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Not sure if you should remove this null-check or add some other ones,
>> but I think you ought to do one or the other.
> As I hope is apparent, the first hunk of this patch is not for commit,
> and the second hunk is for commit.
Looks plausible to me.
regards, tom lane
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 19:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-28 20:53 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-28 21:39 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 32+ messages in thread
From: Justin Pryzby @ 2022-03-28 20:53 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers@postgresql.org, tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 28, 2022 at 03:50:50PM -0400, Robert Haas wrote:
> On Sun, Mar 27, 2022 at 1:47 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Coverity has a nitpick about this:
> >
> > /srv/coverity/git/pgsql-git/postgresql/src/common/backup_compression.c: 194 in parse_bc_specification()
> > 193 /* Advance to next entry and loop around. */
> > >>> CID 1503251: Null pointer dereferences (REVERSE_INULL)
> > >>> Null-checking "vend" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.
> > 194 specification = vend == NULL ? kwend + 1 : vend + 1;
> > 195 }
> > 196 }
> >
> > Not sure if you should remove this null-check or add some other ones,
> > but I think you ought to do one or the other.
>
> Yes, I think this is buggy. I think there's only a theoretical bug
> right now, because the only keyword we have is "level" and that
> requires a value. But if I add an example keyword that does not
> require an associated value (as demonstrated in the attached patch)
> and do something like pg_basebackup -cfast -D whatever --compress
> lz4:example, then the present code will dereference "vend" even though
> it's NULL, which is not good. The attached patch also shows how I
> think that should be fixed.
>
> As I hope is apparent, the first hunk of this patch is not for commit,
> and the second hunk is for commit.
Confirmed that it's a real issue with my patch for zstd long match mode. But
you need to specify another option after the value-less flag option for it to
crash.
I suggest to write it differently, as in 0002.
This also fixes some rebase-induced errors with my previous patches, and adds
expect_boolean().
Attachments: download all (4 files, .tar.gz)
0001-Allow-parallel-zstd-compression-when-taking-a-base-b.patch (text/x-diff, 14.6K) download
From 9bedbfc6bfa471473a8b3479ffd1888d5da285ab Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Mon, 28 Mar 2022 13:25:44 -0400
Subject: [PATCH 1/4] Allow parallel zstd compression when taking a base
backup.
libzstd allows transparent parallel compression just by setting
an option when creating the compression context, so permit that
for both client and server-side backup compression. To use this,
use something like pg_basebackup --compress WHERE-zstd:workers=N
where WHERE is "client" or "server" and N is an integer.
When compression is performed on the server side, this will spawn
threads inside the PostgreSQL backend. While there is almost no
PostgreSQL server code which is thread-safe, the threads here are used
internally by libzstd and touch only data structures controlled by
libzstd.
Patch by me, based in part on earlier work by Dipesh Pandit
and Jeevan Ladhe. Reviewed by Justin Pryzby.
---
doc/src/sgml/protocol.sgml | 12 +++--
doc/src/sgml/ref/pg_basebackup.sgml | 4 +-
src/backend/replication/basebackup_zstd.c | 45 ++++++++++++-------
src/bin/pg_basebackup/bbstreamer_zstd.c | 40 ++++++++++++-----
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 5 +++
src/bin/pg_verifybackup/t/009_extract.pl | 29 +++++++++++-
src/bin/pg_verifybackup/t/010_client_untar.pl | 33 ++++++++++++--
src/common/backup_compression.c | 16 +++++++
src/include/common/backup_compression.h | 2 +
src/test/perl/PostgreSQL/Test/Cluster.pm | 3 +-
10 files changed, 148 insertions(+), 41 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 2fa3cedfe9e..98f0bc3cc34 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2739,17 +2739,23 @@ The commands accepted in replication mode are:
option. If the value is an integer, it specifies the compression
level. Otherwise, it should be a comma-separated list of items,
each of the form <literal>keyword</literal> or
- <literal>keyword=value</literal>. Currently, the only supported
- keyword is <literal>level</literal>, which sets the compression
- level.
+ <literal>keyword=value</literal>. Currently, the supported keywords
+ are <literal>level</literal> and <literal>workers</literal>.
</para>
<para>
+ The <literal>level</literal> keyword sets the compression level.
For <literal>gzip</literal> the compression level should be an
integer between 1 and 9, for <literal>lz4</literal> an integer
between 1 and 12, and for <literal>zstd</literal> an integer
between 1 and 22.
</para>
+
+ <para>
+ The <literal>workers</literal> keyword sets the number of threads
+ that should be used for parallel compression. Parallel compression
+ is supported only for <literal>zstd</literal>.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index d9233beb8e1..82f5f606250 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -424,8 +424,8 @@ PostgreSQL documentation
integer, it specifies the compression level. Otherwise, it should be
a comma-separated list of items, each of the form
<literal>keyword</literal> or <literal>keyword=value</literal>.
- Currently, the only supported keyword is <literal>level</literal>,
- which sets the compression level.
+ Currently, the supported keywords are <literal>level</literal>
+ and <literal>workers</literal>.
</para>
<para>
If no compression level is specified, the default compression level
diff --git a/src/backend/replication/basebackup_zstd.c b/src/backend/replication/basebackup_zstd.c
index 5496eaa72b7..f6876f48118 100644
--- a/src/backend/replication/basebackup_zstd.c
+++ b/src/backend/replication/basebackup_zstd.c
@@ -25,8 +25,8 @@ typedef struct bbsink_zstd
/* Common information for all types of sink. */
bbsink base;
- /* Compression level */
- int compresslevel;
+ /* Compression options */
+ bc_specification *compress;
ZSTD_CCtx *cctx;
ZSTD_outBuffer zstd_outBuf;
@@ -67,22 +67,13 @@ bbsink_zstd_new(bbsink *next, bc_specification *compress)
return NULL; /* keep compiler quiet */
#else
bbsink_zstd *sink;
- int compresslevel;
Assert(next != NULL);
- if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
- compresslevel = 0;
- else
- {
- compresslevel = compress->level;
- Assert(compresslevel >= 1 && compresslevel <= 22);
- }
-
sink = palloc0(sizeof(bbsink_zstd));
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_zstd_ops;
sink->base.bbs_next = next;
- sink->compresslevel = compresslevel;
+ sink->compress = compress;
return &sink->base;
#endif
@@ -99,16 +90,36 @@ bbsink_zstd_begin_backup(bbsink *sink)
bbsink_zstd *mysink = (bbsink_zstd *) sink;
size_t output_buffer_bound;
size_t ret;
+ bc_specification *compress = mysink->compress;
mysink->cctx = ZSTD_createCCtx();
if (!mysink->cctx)
elog(ERROR, "could not create zstd compression context");
- ret = ZSTD_CCtx_setParameter(mysink->cctx, ZSTD_c_compressionLevel,
- mysink->compresslevel);
- if (ZSTD_isError(ret))
- elog(ERROR, "could not set zstd compression level to %d: %s",
- mysink->compresslevel, ZSTD_getErrorName(ret));
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(mysink->cctx, ZSTD_c_compressionLevel,
+ compress->level);
+ if (ZSTD_isError(ret))
+ elog(ERROR, "could not set zstd compression level to %d: %s",
+ compress->level, ZSTD_getErrorName(ret));
+ }
+
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_WORKERS) != 0)
+ {
+ /*
+ * On older versions of libzstd, this option does not exist, and trying
+ * to set it will fail. Similarly for newer versions if they are
+ * compiled without threading support.
+ */
+ ret = ZSTD_CCtx_setParameter(mysink->cctx, ZSTD_c_nbWorkers,
+ compress->workers);
+ if (ZSTD_isError(ret))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not set compression worker count to %d: %s",
+ compress->workers, ZSTD_getErrorName(ret)));
+ }
/*
* We need our own buffer, because we're going to pass different data to
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index 7946b6350b6..f94c5c041d3 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -67,7 +67,6 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, bc_specification *compress)
{
#ifdef USE_ZSTD
bbstreamer_zstd_frame *streamer;
- int compresslevel;
size_t ret;
Assert(next != NULL);
@@ -88,18 +87,35 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, bc_specification *compress)
exit(1);
}
- /* Initialize stream compression preferences */
- if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
- compresslevel = 0;
- else
- compresslevel = compress->level;
- ret = ZSTD_CCtx_setParameter(streamer->cctx, ZSTD_c_compressionLevel,
- compresslevel);
- if (ZSTD_isError(ret))
+ /* Set compression level, if specified */
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) != 0)
{
- pg_log_error("could not set zstd compression level to %d: %s",
- compresslevel, ZSTD_getErrorName(ret));
- exit(1);
+ ret = ZSTD_CCtx_setParameter(streamer->cctx, ZSTD_c_compressionLevel,
+ compress->level);
+ if (ZSTD_isError(ret))
+ {
+ pg_log_error("could not set zstd compression level to %d: %s",
+ compress->level, ZSTD_getErrorName(ret));
+ exit(1);
+ }
+ }
+
+ /* Set # of workers, if specified */
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_WORKERS) != 0)
+ {
+ /*
+ * On older versions of libzstd, this option does not exist, and
+ * trying to set it will fail. Similarly for newer versions if they
+ * are compiled without threading support.
+ */
+ ret = ZSTD_CCtx_setParameter(streamer->cctx, ZSTD_c_nbWorkers,
+ compress->workers);
+ if (ZSTD_isError(ret))
+ {
+ pg_log_error("could not set compression worker count to %d: %s",
+ compress->workers, ZSTD_getErrorName(ret));
+ exit(1);
+ }
}
/* Initialize the ZSTD output buffer. */
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 47f3d00ac45..5ba84c22509 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -130,6 +130,11 @@ my @compression_failure_tests = (
'invalid compression specification: found empty string where a compression option was expected',
'failure on extra, empty compression option'
],
+ [
+ 'gzip:workers=3',
+ 'invalid compression specification: compression algorithm "gzip" does not accept a worker count',
+ 'failure on worker count for gzip'
+ ],
);
for my $cft (@compression_failure_tests)
{
diff --git a/src/bin/pg_verifybackup/t/009_extract.pl b/src/bin/pg_verifybackup/t/009_extract.pl
index 41a5b370cc5..d6f11b95535 100644
--- a/src/bin/pg_verifybackup/t/009_extract.pl
+++ b/src/bin/pg_verifybackup/t/009_extract.pl
@@ -34,6 +34,12 @@ my @test_configuration = (
'compression_method' => 'zstd',
'backup_flags' => ['--compress', 'server-zstd:5'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
+ {
+ 'compression_method' => 'parallel zstd',
+ 'backup_flags' => ['--compress', 'server-zstd:workers=3'],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1"),
+ 'possibly_unsupported' => qr/could not set compression worker count to 3: Unsupported parameter/
}
);
@@ -55,8 +61,27 @@ for my $tc (@test_configuration)
my @verify = ('pg_verifybackup', '-e', $backup_path);
# A backup with a valid compression method should work.
- $primary->command_ok(\@backup,
- "backup done, compression method \"$method\"");
+ my $backup_stdout = '';
+ my $backup_stderr = '';
+ my $backup_result = $primary->run_log(\@backup, '>', \$backup_stdout,
+ '2>', \$backup_stderr);
+ if ($backup_stdout ne '')
+ {
+ print "# standard output was:\n$backup_stdout";
+ }
+ if ($backup_stderr ne '')
+ {
+ print "# standard error was:\n$backup_stderr";
+ }
+ if (! $backup_result && $tc->{'possibly_unsupported'} &&
+ $backup_stderr =~ /$tc->{'possibly_unsupported'}/)
+ {
+ skip "compression with $method not supported by this build", 2;
+ }
+ else
+ {
+ ok($backup_result, "backup done, compression $method");
+ }
# Make sure that it verifies OK.
$primary->command_ok(\@verify,
diff --git a/src/bin/pg_verifybackup/t/010_client_untar.pl b/src/bin/pg_verifybackup/t/010_client_untar.pl
index 488a6d1edee..c1cd12cb065 100644
--- a/src/bin/pg_verifybackup/t/010_client_untar.pl
+++ b/src/bin/pg_verifybackup/t/010_client_untar.pl
@@ -49,6 +49,15 @@ my @test_configuration = (
'decompress_program' => $ENV{'ZSTD'},
'decompress_flags' => [ '-d' ],
'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
+ {
+ 'compression_method' => 'parallel zstd',
+ 'backup_flags' => ['--compress', 'client-zstd:workers=3'],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => [ '-d' ],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1"),
+ 'possibly_unsupported' => qr/could not set compression worker count to 3: Unsupported parameter/
}
);
@@ -69,9 +78,27 @@ for my $tc (@test_configuration)
'pg_basebackup', '-D', $backup_path,
'-Xfetch', '--no-sync', '-cfast', '-Ft');
push @backup, @{$tc->{'backup_flags'}};
- $primary->command_ok(\@backup,
- "client side backup, compression $method");
-
+ my $backup_stdout = '';
+ my $backup_stderr = '';
+ my $backup_result = $primary->run_log(\@backup, '>', \$backup_stdout,
+ '2>', \$backup_stderr);
+ if ($backup_stdout ne '')
+ {
+ print "# standard output was:\n$backup_stdout";
+ }
+ if ($backup_stderr ne '')
+ {
+ print "# standard error was:\n$backup_stderr";
+ }
+ if (! $backup_result && $tc->{'possibly_unsupported'} &&
+ $backup_stderr =~ /$tc->{'possibly_unsupported'}/)
+ {
+ skip "compression with $method not supported by this build", 3;
+ }
+ else
+ {
+ ok($backup_result, "client side backup, compression $method");
+ }
# Verify that the we got the files we expected.
my $backup_files = join(',',
diff --git a/src/common/backup_compression.c b/src/common/backup_compression.c
index 0650f975c44..969e08cca20 100644
--- a/src/common/backup_compression.c
+++ b/src/common/backup_compression.c
@@ -177,6 +177,11 @@ parse_bc_specification(bc_algorithm algorithm, char *specification,
result->level = expect_integer_value(keyword, value, result);
result->options |= BACKUP_COMPRESSION_OPTION_LEVEL;
}
+ else if (strcmp(keyword, "workers") == 0)
+ {
+ result->workers = expect_integer_value(keyword, value, result);
+ result->options |= BACKUP_COMPRESSION_OPTION_WORKERS;
+ }
else
result->parse_error =
psprintf(_("unknown compression option \"%s\""), keyword);
@@ -266,5 +271,16 @@ validate_bc_specification(bc_specification *spec)
min_level, max_level);
}
+ /*
+ * Of the compression algorithms that we currently support, only zstd
+ * allows parallel workers.
+ */
+ if ((spec->options & BACKUP_COMPRESSION_OPTION_WORKERS) != 0 &&
+ (spec->algorithm != BACKUP_COMPRESSION_ZSTD))
+ {
+ return psprintf(_("compression algorithm \"%s\" does not accept a worker count"),
+ get_bc_algorithm_name(spec->algorithm));
+ }
+
return NULL;
}
diff --git a/src/include/common/backup_compression.h b/src/include/common/backup_compression.h
index 0565cbc657d..6a0ecaa99c9 100644
--- a/src/include/common/backup_compression.h
+++ b/src/include/common/backup_compression.h
@@ -23,12 +23,14 @@ typedef enum bc_algorithm
} bc_algorithm;
#define BACKUP_COMPRESSION_OPTION_LEVEL (1 << 0)
+#define BACKUP_COMPRESSION_OPTION_WORKERS (1 << 1)
typedef struct bc_specification
{
bc_algorithm algorithm;
unsigned options; /* OR of BACKUP_COMPRESSION_OPTION constants */
int level;
+ int workers;
char *parse_error; /* NULL if parsing was OK, else message */
} bc_specification;
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index bee6aacf47c..b6e33516110 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2502,8 +2502,7 @@ sub run_log
local %ENV = $self->_get_env();
- PostgreSQL::Test::Utils::run_log(@_);
- return;
+ return PostgreSQL::Test::Utils::run_log(@_);
}
=pod
--
2.17.1
0002-Avoid-crash-on-backup-connection-strings-with-flags-.patch (text/x-diff, 975B) download
From a0c100c4473863335dc54ffc6167669cdc858096 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 28 Mar 2022 15:16:50 -0500
Subject: [PATCH 2/4] Avoid crash on backup connection strings with flags with
no value
---
src/common/backup_compression.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/src/common/backup_compression.c b/src/common/backup_compression.c
index 969e08cca20..477dc7eb49b 100644
--- a/src/common/backup_compression.c
+++ b/src/common/backup_compression.c
@@ -192,7 +192,11 @@ parse_bc_specification(bc_algorithm algorithm, char *specification,
pfree(value);
/* If we got an error or have reached the end of the string, stop. */
- if (result->parse_error != NULL || *kwend == '\0' || *vend == '\0')
+ if (result->parse_error != NULL)
+ break;
+ if (*kwend == '\0')
+ break;
+ if (vend != NULL && *vend == '\0')
break;
/* Advance to next entry and loop around. */
--
2.17.1
0003-basebackup-support-Z-zstd-long.patch (text/x-diff, 7.2K) download
From f0e5ee4d78dce6bc4d111b8b574c6b75f546ee4a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 27 Mar 2022 11:55:01 -0500
Subject: [PATCH 3/4] basebackup: support -Z zstd:long
---
doc/src/sgml/protocol.sgml | 10 +++++-
doc/src/sgml/ref/pg_basebackup.sgml | 4 +--
src/backend/replication/basebackup_zstd.c | 12 +++++++
src/bin/pg_basebackup/bbstreamer_zstd.c | 13 +++++++
src/common/backup_compression.c | 44 +++++++++++++++++++++++
src/include/common/backup_compression.h | 2 ++
6 files changed, 82 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 98f0bc3cc34..80f1a1f9a04 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2740,7 +2740,8 @@ The commands accepted in replication mode are:
level. Otherwise, it should be a comma-separated list of items,
each of the form <literal>keyword</literal> or
<literal>keyword=value</literal>. Currently, the supported keywords
- are <literal>level</literal> and <literal>workers</literal>.
+ are <literal>level</literal>, <literal>long</literal>, and
+ <literal>workers</literal>.
</para>
<para>
@@ -2751,6 +2752,13 @@ The commands accepted in replication mode are:
between 1 and 22.
</para>
+ <para>
+ The <literal>long</literal> keyword enables long-distance matching
+ mode, for improved compression ratio, at the expense of higher memory
+ use. Long-distance mode is supported only for
+ <literal>zstd</literal>.
+ </para>
+
<para>
The <literal>workers</literal> keyword sets the number of threads
that should be used for parallel compression. Parallel compression
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 82f5f606250..014c454bfab 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -424,8 +424,8 @@ PostgreSQL documentation
integer, it specifies the compression level. Otherwise, it should be
a comma-separated list of items, each of the form
<literal>keyword</literal> or <literal>keyword=value</literal>.
- Currently, the supported keywords are <literal>level</literal>
- and <literal>workers</literal>.
+ Currently, the supported keywords are <literal>level</literal>,
+ <literal>long</literal>, and <literal>workers</literal>.
</para>
<para>
If no compression level is specified, the default compression level
diff --git a/src/backend/replication/basebackup_zstd.c b/src/backend/replication/basebackup_zstd.c
index f6876f48118..dc23898f7fd 100644
--- a/src/backend/replication/basebackup_zstd.c
+++ b/src/backend/replication/basebackup_zstd.c
@@ -121,6 +121,18 @@ bbsink_zstd_begin_backup(bbsink *sink)
compress->workers, ZSTD_getErrorName(ret)));
}
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_ZSTD_LONG) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(mysink->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->zstd_long);
+ if (ZSTD_isError(ret))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret)));
+ }
+
/*
* We need our own buffer, because we're going to pass different data to
* the next sink than what gets passed to us.
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index f94c5c041d3..051b97458ba 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -118,6 +118,19 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, bc_specification *compress)
}
}
+ if ((compress->options & BACKUP_COMPRESSION_OPTION_ZSTD_LONG) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(streamer->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->zstd_long);
+ if (ZSTD_isError(ret))
+ {
+ pg_log_error("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret));
+ exit(1);
+ }
+ }
+
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
streamer->zstd_outBuf.size = streamer->base.bbs_buffer.maxlen;
diff --git a/src/common/backup_compression.c b/src/common/backup_compression.c
index 477dc7eb49b..9fc865ff299 100644
--- a/src/common/backup_compression.c
+++ b/src/common/backup_compression.c
@@ -31,6 +31,8 @@
static int expect_integer_value(char *keyword, char *value,
bc_specification *result);
+static bool expect_boolean_value(char *keyword, char *value,
+ bc_specification *result);
/*
* Look up a compression algorithm by name. Returns true and sets *algorithm
@@ -182,6 +184,11 @@ parse_bc_specification(bc_algorithm algorithm, char *specification,
result->workers = expect_integer_value(keyword, value, result);
result->options |= BACKUP_COMPRESSION_OPTION_WORKERS;
}
+ else if (strcmp(keyword, "long") == 0)
+ {
+ result->zstd_long = expect_boolean_value(keyword, value, result);
+ result->options |= BACKUP_COMPRESSION_OPTION_ZSTD_LONG;
+ }
else
result->parse_error =
psprintf(_("unknown compression option \"%s\""), keyword);
@@ -235,6 +242,43 @@ expect_integer_value(char *keyword, char *value, bc_specification *result)
return ivalue;
}
+/*
+ * Parse 'value' as an boolean and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1. The caller must check result->parse_error to determine if
+ * the call was successful.
+ *
+ * Valid values are: yes, no, on, off, 1, 0.
+ *
+ * Inspired by ParseVariableBool().
+ */
+static bool
+expect_boolean_value(char *keyword, char *value, bc_specification *result)
+{
+ if (value == NULL)
+ return true;
+
+ if (pg_strcasecmp(value, "yes") == 0)
+ return true;
+ if (pg_strcasecmp(value, "on") == 0)
+ return true;
+ if (pg_strcasecmp(value, "1") == 0)
+ return true;
+
+ if (pg_strcasecmp(value, "no") == 0)
+ return false;
+ if (pg_strcasecmp(value, "off") == 0)
+ return false;
+ if (pg_strcasecmp(value, "0") == 0)
+ return false;
+
+ result->parse_error =
+ psprintf(_("value for compression option \"%s\" must be a boolean"),
+ keyword);
+ return false;
+}
+
/*
* Returns NULL if the compression specification string was syntactically
* valid and semantically sensible. Otherwise, returns an error message.
diff --git a/src/include/common/backup_compression.h b/src/include/common/backup_compression.h
index 6a0ecaa99c9..a378631a8da 100644
--- a/src/include/common/backup_compression.h
+++ b/src/include/common/backup_compression.h
@@ -24,6 +24,7 @@ typedef enum bc_algorithm
#define BACKUP_COMPRESSION_OPTION_LEVEL (1 << 0)
#define BACKUP_COMPRESSION_OPTION_WORKERS (1 << 1)
+#define BACKUP_COMPRESSION_OPTION_ZSTD_LONG (1 << 2)
typedef struct bc_specification
{
@@ -31,6 +32,7 @@ typedef struct bc_specification
unsigned options; /* OR of BACKUP_COMPRESSION_OPTION constants */
int level;
int workers;
+ int zstd_long;
char *parse_error; /* NULL if parsing was OK, else message */
} bc_specification;
--
2.17.1
0004-pg_basebackup-support-Zstd-negative-compression-leve.patch (text/x-diff, 1.9K) download
From bc6846ed93af475b079c4ab9bfa2a33c49a8a185 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Thu, 10 Mar 2022 20:16:19 -0600
Subject: [PATCH 4/4] pg_basebackup: support Zstd negative compression levels
"higher than maximum" is bogus
TODO: each compression methods should enforce its own levels
---
src/bin/pg_basebackup/bbstreamer_zstd.c | 1 +
src/common/backup_compression.c | 6 +++++-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index 051b97458ba..491d6106cf5 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -114,6 +114,7 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, bc_specification *compress)
{
pg_log_error("could not set compression worker count to %d: %s",
compress->workers, ZSTD_getErrorName(ret));
+
exit(1);
}
}
diff --git a/src/common/backup_compression.c b/src/common/backup_compression.c
index 9fc865ff299..dbaf008af8e 100644
--- a/src/common/backup_compression.c
+++ b/src/common/backup_compression.c
@@ -308,13 +308,17 @@ validate_bc_specification(bc_specification *spec)
else if (spec->algorithm == BACKUP_COMPRESSION_LZ4)
max_level = 12;
else if (spec->algorithm == BACKUP_COMPRESSION_ZSTD)
+ {
max_level = 22;
+ /* The minimum level depends on the version.. */
+ min_level = -7;
+ }
else
return psprintf(_("compression algorithm \"%s\" does not accept a compression level"),
get_bc_algorithm_name(spec->algorithm));
if (spec->level < min_level || spec->level > max_level)
- return psprintf(_("compression algorithm \"%s\" expects a compression level between %d and %d"),
+ return psprintf(_("compression algorithm \"%s\" expects a nonzero compression level between %d and %d"),
get_bc_algorithm_name(spec->algorithm),
min_level, max_level);
}
--
2.17.1
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 19:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-28 20:53 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
@ 2022-03-28 21:39 ` Robert Haas <robertmhaas@gmail.com>
2022-03-29 00:07 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 32+ messages in thread
From: Robert Haas @ 2022-03-28 21:39 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 28, 2022 at 4:53 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> I suggest to write it differently, as in 0002.
That doesn't seem better to me. What's the argument for it?
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 19:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-28 20:53 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-28 21:39 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-29 00:07 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-29 00:11 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 32+ messages in thread
From: Justin Pryzby @ 2022-03-29 00:07 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers@postgresql.org, tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 28, 2022 at 05:39:31PM -0400, Robert Haas wrote:
> On Mon, Mar 28, 2022 at 4:53 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> > I suggest to write it differently, as in 0002.
>
> That doesn't seem better to me. What's the argument for it?
I find this much easier to understand:
/* If we got an error or have reached the end of the string, stop. */
- if (result->parse_error != NULL || *kwend == '\0' || *vend == '\0')
+ if (result->parse_error != NULL)
+ break;
+ if (*kwend == '\0')
+ break;
+ if (vend != NULL && *vend == '\0')
break;
than
/* If we got an error or have reached the end of the string, stop. */
- if (result->parse_error != NULL || *kwend == '\0' || *vend == '\0')
+ if (result->parse_error != NULL ||
+ (vend == NULL ? *kwend == '\0' : *vend == '\0'))
Also, why wouldn't *kwend be checked in any case ?
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 19:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-28 20:53 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-28 21:39 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-29 00:07 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
@ 2022-03-29 00:11 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-29 12:51 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 32+ messages in thread
From: Tom Lane @ 2022-03-29 00:11 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers@postgresql.org, tushar <tushar.ahuja@enterprisedb.com>
Justin Pryzby <pryzby@telsasoft.com> writes:
> Also, why wouldn't *kwend be checked in any case ?
I suspect Robert wrote it that way intentionally --- but if so,
I agree it could do with more than zero commentary.
regards, tom lane
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 19:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-28 20:53 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-28 21:39 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-29 00:07 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-29 00:11 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-29 12:51 ` Robert Haas <robertmhaas@gmail.com>
2022-03-30 20:00 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 32+ messages in thread
From: Robert Haas @ 2022-03-29 12:51 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Mon, Mar 28, 2022 at 8:11 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I suspect Robert wrote it that way intentionally --- but if so,
> I agree it could do with more than zero commentary.
Well, the point is, we stop advancing kwend when we get to the end of
the keyword, and *vend when we get to the end of the value. If there's
a value, the end of the keyword can't have been the end of the string,
but the end of the value might have been. If there's no value, the end
of the keyword could be the end of the string.
Maybe if I just put that last sentence into the comment it's clear enough?
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 19:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-28 20:53 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-28 21:39 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-29 00:07 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-29 00:11 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-29 12:51 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-30 20:00 ` Robert Haas <robertmhaas@gmail.com>
2022-03-30 20:14 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 32+ messages in thread
From: Robert Haas @ 2022-03-30 20:00 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
On Tue, Mar 29, 2022 at 8:51 AM Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Mar 28, 2022 at 8:11 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > I suspect Robert wrote it that way intentionally --- but if so,
> > I agree it could do with more than zero commentary.
>
> Well, the point is, we stop advancing kwend when we get to the end of
> the keyword, and *vend when we get to the end of the value. If there's
> a value, the end of the keyword can't have been the end of the string,
> but the end of the value might have been. If there's no value, the end
> of the keyword could be the end of the string.
>
> Maybe if I just put that last sentence into the comment it's clear enough?
Done that way, since I thought it was better to fix the bug than wait
for more feedback on the wording. We can still adjust the wording, or
the coding, if it's not clear enough.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 19:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-28 20:53 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-28 21:39 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-29 00:07 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-29 00:11 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-29 12:51 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-30 20:00 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
@ 2022-03-30 20:14 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-30 20:27 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 32+ messages in thread
From: Tom Lane @ 2022-03-30 20:14 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers; tushar <tushar.ahuja@enterprisedb.com>
Robert Haas <robertmhaas@gmail.com> writes:
>> Maybe if I just put that last sentence into the comment it's clear enough?
> Done that way, since I thought it was better to fix the bug than wait
> for more feedback on the wording. We can still adjust the wording, or
> the coding, if it's not clear enough.
FWIW, I thought that explanation was fine, but I was deferring to
Justin who was the one who thought things were unclear.
regards, tom lane
^ permalink raw reply [nested|flat] 32+ messages in thread
* Re: refactoring basebackup.c (zstd workers)
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-20 19:05 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Re: refactoring basebackup.c (zstd workers) Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 19:50 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-28 20:53 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-28 21:39 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-29 00:07 ` Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-29 00:11 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
2022-03-29 12:51 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-30 20:00 ` Re: refactoring basebackup.c (zstd workers) Robert Haas <robertmhaas@gmail.com>
2022-03-30 20:14 ` Re: refactoring basebackup.c (zstd workers) Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-30 20:27 ` Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 0 replies; 32+ messages in thread
From: Justin Pryzby @ 2022-03-30 20:27 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andres Freund <andres@anarazel.de>; Jeevan Ladhe <jeevanladhe.os@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Shinoda, Noriyoshi (PN Japan FSIP) <noriyoshi.shinoda@hpe.com>; Abhijit Menon-Sen <ams@toroid.org>; Dmitry Dolgov <9erthalion6@gmail.com>; Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>; Mark Dilger <mark.dilger@enterprisedb.com>; pgsql-hackers@postgresql.org, tushar <tushar.ahuja@enterprisedb.com>
On Wed, Mar 30, 2022 at 04:14:47PM -0400, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> >> Maybe if I just put that last sentence into the comment it's clear enough?
>
> > Done that way, since I thought it was better to fix the bug than wait
> > for more feedback on the wording. We can still adjust the wording, or
> > the coding, if it's not clear enough.
>
> FWIW, I thought that explanation was fine, but I was deferring to
> Justin who was the one who thought things were unclear.
I still think it's unnecessarily confusing to nest "if" and "?:" conditionals
in one statement, instead of 2 or 3 separate "if"s, or "||"s.
But it's also not worth fussing over any more.
^ permalink raw reply [nested|flat] 32+ messages in thread
end of thread, other threads:[~2022-03-30 20:27 UTC | newest]
Thread overview: 32+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-03-14 16:35 Re: refactoring basebackup.c (zstd workers) Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:02 ` Robert Haas <robertmhaas@gmail.com>
2022-03-14 17:11 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-14 17:21 ` Robert Haas <robertmhaas@gmail.com>
2022-03-17 15:50 ` Robert Haas <robertmhaas@gmail.com>
2022-03-17 19:41 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-17 20:29 ` Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:05 ` Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:11 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-21 01:24 ` Robert Haas <robertmhaas@gmail.com>
2022-03-21 01:32 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-21 02:03 ` Robert Haas <robertmhaas@gmail.com>
2022-03-20 19:40 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 01:38 ` Robert Haas <robertmhaas@gmail.com>
2022-03-21 13:18 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 16:57 ` Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:22 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-21 18:25 ` Robert Haas <robertmhaas@gmail.com>
2022-03-21 18:41 ` Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2022-03-22 15:37 ` Robert Haas <robertmhaas@gmail.com>
2022-03-23 13:19 ` Robert Haas <robertmhaas@gmail.com>
2022-03-27 17:47 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 19:50 ` Robert Haas <robertmhaas@gmail.com>
2022-03-28 20:38 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-28 20:53 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-28 21:39 ` Robert Haas <robertmhaas@gmail.com>
2022-03-29 00:07 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-29 00:11 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-29 12:51 ` Robert Haas <robertmhaas@gmail.com>
2022-03-30 20:00 ` Robert Haas <robertmhaas@gmail.com>
2022-03-30 20:14 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-30 20:27 ` Justin Pryzby <pryzby@telsasoft.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox