Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1p0Yxr-0000vh-43 for pgsql-hackers@arkaria.postgresql.org; Thu, 01 Dec 2022 02:06:16 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1p0Yxp-0006AX-Jn for pgsql-hackers@arkaria.postgresql.org; Thu, 01 Dec 2022 02:06:13 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1p0Yxn-00063J-U8 for pgsql-hackers@lists.postgresql.org; Thu, 01 Dec 2022 02:06:13 +0000 Received: from wout4-smtp.messagingengine.com ([64.147.123.20]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1p0Yxi-0006R8-07 for pgsql-hackers@lists.postgresql.org; Thu, 01 Dec 2022 02:06:11 +0000 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id B3D703200952; Wed, 30 Nov 2022 21:06:01 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Wed, 30 Nov 2022 21:06:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=paquier.xyz; h= cc:cc:content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1669860361; x=1669946761; bh=2oPsEihV4J 5PcokOEFVTGMSsus8cXRkP5oc+vbEc/+Q=; b=f2b3gLyeWcYyO2wGFWMSIQkc0s DNkEEfseJ+1/J9iCXzfu+pZE9RIOckz8ErICO+EU3Cs1q5f06hhlq7WnyPH8uRYr UsmzEEgOHakanZpy6e6KFiiThEuYpPwV2ZNfJYg2yVKH0WpabyLFSH5488eHe/Tt rhhKuNF3yREhWabfCIIF89SZUBESukcPD/IRkb5V8F+NHmRq3GOqGxKjALwHyFws lH2ToOtr9eMHxumzbMsFfGqzkoXmNFNL3nf9FIw7E4bTdlItOCeYNL8aNQTqzjsC 5BOQgPE9jJzLJSQDglQDgyyVBSYiNZJt1+uxn0LrsmZhklS1HPYslwBQODTw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1669860361; x=1669946761; bh=2oPsEihV4J5PcokOEFVTGMSsus8c XRkP5oc+vbEc/+Q=; b=nJKnnBxKMKPJqNirhx/fHcc6Cv1z2/3X4+eR44Vp92q4 Re2tavLjY/lDoDt5bB+fWXNBUFiG9JmXHKsFsVCWYw5i4EkZ7fweJdTlKf8puFX/ JfofPF3pQ3y+rx33SMEgXVFkz/SVwgcu4kCBYD2uXVqReVjJ02mzfk2Wv1R1xEDD daAHe9OD3lt+9fbECn1BHWTVJTBVhvLUrsStRz2qilh5KWlnKnxunR82FHbVaxU8 gS4VHyUtF7tOUJRu7bwujVlU8oCkiDXvHvfnNjsWKj1n15VAqlvQ59oAulkjjJ/C DtoHPYv63iUChZKWIJWRtY00p/iEvZNB7R/JRoaU5Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrtdeggdeggecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenfg hrlhcuvffnffculddvfedmnecujfgurhepfffhvfevuffkfhggtggujgesghdtreertddt vdenucfhrhhomhepofhitghhrggvlhcurfgrqhhuihgvrhcuoehmihgthhgrvghlsehprg hquhhivghrrdighiiiqeenucggtffrrghtthgvrhhnpeegffejvefgveduvdejtddvtdei jeehudeuledvudeftdfgfeejvdekveekiedvvdenucffohhmrghinhepphhoshhtghhrvg hsqhhlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhf rhhomhepmhhitghhrggvlhesphgrqhhuihgvrhdrgiihii X-ME-Proxy: Feedback-ID: i0fe9450f:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 30 Nov 2022 21:05:58 -0500 (EST) Date: Thu, 1 Dec 2022 11:05:54 +0900 From: Michael Paquier To: gkokolatos@pm.me Cc: Justin Pryzby , pgsql-hackers@lists.postgresql.org, Rachel Heaton Subject: Re: Add LZ4 compression in pg_dump Message-ID: References: <-pU80vtiyXMfSSpn4SWOR97t71MVmDOyHaJtvGsdEfroFsFUDdc-UaqD-8LMPMw2sy-7B2_IVnlrATfVvXwtv8n3kexLePZ7iWgiOnE4jRI=@pm.me> <20221120172611.GQ11463@telsasoft.com> <1j_ONBQqU3lQW7OqwbAFKbgcqjQosikcp7VPINqegLrHMCMGtFcXAuyjmW__VeviKyCbKTtucxFyPFUJ7sfqeOf4O90JEuUIe99Z5v7sdOg=@pm.me> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="HOBd0O/w67ZHsizP" Content-Disposition: inline In-Reply-To: <1j_ONBQqU3lQW7OqwbAFKbgcqjQosikcp7VPINqegLrHMCMGtFcXAuyjmW__VeviKyCbKTtucxFyPFUJ7sfqeOf4O90JEuUIe99Z5v7sdOg=@pm.me> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --HOBd0O/w67ZHsizP Content-Type: multipart/mixed; boundary="Ab/7/En0vSesZYLI" Content-Disposition: inline --Ab/7/En0vSesZYLI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wed, Nov 30, 2022 at 05:11:44PM +0000, gkokolatos@pm.me wrote: > Fair enough. The atteched v11 does that. 0001 introduces compression > specification and is using it throughout. 0002 paves the way to the > new interface by homogenizing the use of cfp. 0003 introduces the new > API and stores the compression algorithm in the custom format header > instead of the compression level integer. Finally 0004 adds support for > LZ4. I have been looking at 0001, and.. Hmm. I am really wondering whether it would not be better to just nuke this warning into orbit. This stuff enforces non-compression even if -Z has been used to a non-default value. This has been moved to its current location by cae2bb1 as of this thread: https://www.postgresql.org/message-id/20160526.185551.242041780.horiguchi.kyotaro%40lab.ntt.co.jp However, this is only active if -Z is used when not building with zlib. At the end, it comes down to whether we want to prioritize the portability of pg_dump commands specifying a -Z/--compress across environments knowing that these may or may not be built with zlib, vs the amount of simplification/uniformity we would get across the binaries in the tree once we switch everything to use the compression specifications. Now that pg_basebackup and pg_receivewal are managed by compression specifications, and that we'd want more compression options for pg_dump, I would tend to do the latter and from now on complain if attempting to do a pg_dump -Z under --without-zlib with a compression level > 0. zlib is also widely available, and we don't document the fact that non-compression is enforced in this case, either. (Two TAP tests with the custom format had to be tweaked.) As per the patch, it is true that we do not need to bump the format of the dump archives, as we can still store only the compression level and guess the method from it. I have added some notes about that in ReadHead and WriteHead to not forget. Most of the changes are really-straight forward, and it has resisted my tests, so I think that this is in a rather-commitable shape as-is. -- Michael --Ab/7/En0vSesZYLI Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v12-0001-Teach-pg_dump-about-compress_spec-and-use-it-thr.patch" Content-Transfer-Encoding: quoted-printable =46rom a4fa522d0259e8969cde32798a917321cced0415 Mon Sep 17 00:00:00 2001 =46rom: Michael Paquier Date: Thu, 1 Dec 2022 11:03:41 +0900 Subject: [PATCH v12] Teach pg_dump about compress_spec and use it throughou= t. Align pg_dump with the rest of the binaries which use common compression. I= t is teaching pg_dump.c about the common compression definitions and interfaces.= Then it propagates those throughout the code. --- src/bin/pg_dump/compress_io.c | 107 ++++++++------------ src/bin/pg_dump/compress_io.h | 20 ++-- src/bin/pg_dump/pg_backup.h | 7 +- src/bin/pg_dump/pg_backup_archiver.c | 76 +++++++++----- src/bin/pg_dump/pg_backup_archiver.h | 10 +- src/bin/pg_dump/pg_backup_custom.c | 6 +- src/bin/pg_dump/pg_backup_directory.c | 13 ++- src/bin/pg_dump/pg_backup_tar.c | 11 +- src/bin/pg_dump/pg_dump.c | 67 +++++++----- src/bin/pg_dump/t/001_basic.pl | 34 +++++-- src/bin/pg_dump/t/002_pg_dump.pl | 3 +- src/test/modules/test_pg_dump/t/001_base.pl | 16 +++ doc/src/sgml/ref/pg_dump.sgml | 34 +++++-- src/tools/pgindent/typedefs.list | 1 - 14 files changed, 242 insertions(+), 163 deletions(-) diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 62f940ff7a..8f0d6d6210 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -64,7 +64,7 @@ /* typedef appears in compress_io.h */ struct CompressorState { - CompressionAlgorithm comprAlg; + pg_compress_specification compression_spec; WriteFunc writeF; =20 #ifdef HAVE_LIBZ @@ -74,9 +74,6 @@ struct CompressorState #endif }; =20 -static void ParseCompressionOption(int compression, CompressionAlgorithm *= alg, - int *level); - /* Routines that support zlib compressed data I/O */ #ifdef HAVE_LIBZ static void InitCompressorZlib(CompressorState *cs, int level); @@ -93,57 +90,30 @@ static void ReadDataFromArchiveNone(ArchiveHandle *AH, = ReadFunc readF); static void WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs, const char *data, size_t dLen); =20 -/* - * Interprets a numeric 'compression' value. The algorithm implied by the - * value (zlib or none at the moment), is returned in *alg, and the - * zlib compression level in *level. - */ -static void -ParseCompressionOption(int compression, CompressionAlgorithm *alg, int *le= vel) -{ - if (compression =3D=3D Z_DEFAULT_COMPRESSION || - (compression > 0 && compression <=3D 9)) - *alg =3D COMPR_ALG_LIBZ; - else if (compression =3D=3D 0) - *alg =3D COMPR_ALG_NONE; - else - { - pg_fatal("invalid compression code: %d", compression); - *alg =3D COMPR_ALG_NONE; /* keep compiler quiet */ - } - - /* The level is just the passed-in value. */ - if (level) - *level =3D compression; -} - /* Public interface routines */ =20 /* Allocate a new compressor */ CompressorState * -AllocateCompressor(int compression, WriteFunc writeF) +AllocateCompressor(const pg_compress_specification compression_spec, + WriteFunc writeF) { CompressorState *cs; - CompressionAlgorithm alg; - int level; - - ParseCompressionOption(compression, &alg, &level); =20 #ifndef HAVE_LIBZ - if (alg =3D=3D COMPR_ALG_LIBZ) + if (compression_spec.algorithm =3D=3D PG_COMPRESSION_GZIP) pg_fatal("not built with zlib support"); #endif =20 cs =3D (CompressorState *) pg_malloc0(sizeof(CompressorState)); cs->writeF =3D writeF; - cs->comprAlg =3D alg; + cs->compression_spec =3D compression_spec; =20 /* * Perform compression algorithm specific initialization. */ #ifdef HAVE_LIBZ - if (alg =3D=3D COMPR_ALG_LIBZ) - InitCompressorZlib(cs, level); + if (cs->compression_spec.algorithm =3D=3D PG_COMPRESSION_GZIP) + InitCompressorZlib(cs, cs->compression_spec.level); #endif =20 return cs; @@ -154,15 +124,12 @@ AllocateCompressor(int compression, WriteFunc writeF) * out with ahwrite(). */ void -ReadDataFromArchive(ArchiveHandle *AH, int compression, ReadFunc readF) +ReadDataFromArchive(ArchiveHandle *AH, pg_compress_specification compressi= on_spec, + ReadFunc readF) { - CompressionAlgorithm alg; - - ParseCompressionOption(compression, &alg, NULL); - - if (alg =3D=3D COMPR_ALG_NONE) + if (compression_spec.algorithm =3D=3D PG_COMPRESSION_NONE) ReadDataFromArchiveNone(AH, readF); - if (alg =3D=3D COMPR_ALG_LIBZ) + if (compression_spec.algorithm =3D=3D PG_COMPRESSION_GZIP) { #ifdef HAVE_LIBZ ReadDataFromArchiveZlib(AH, readF); @@ -179,18 +146,23 @@ void WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs, const void *data, size_t dLen) { - switch (cs->comprAlg) + switch (cs->compression_spec.algorithm) { - case COMPR_ALG_LIBZ: + case PG_COMPRESSION_GZIP: #ifdef HAVE_LIBZ WriteDataToArchiveZlib(AH, cs, data, dLen); #else pg_fatal("not built with zlib support"); #endif break; - case COMPR_ALG_NONE: + case PG_COMPRESSION_NONE: WriteDataToArchiveNone(AH, cs, data, dLen); break; + case PG_COMPRESSION_LZ4: + /* fallthrough */ + case PG_COMPRESSION_ZSTD: + pg_fatal("invalid compression method"); + break; } } =20 @@ -201,7 +173,7 @@ void EndCompressor(ArchiveHandle *AH, CompressorState *cs) { #ifdef HAVE_LIBZ - if (cs->comprAlg =3D=3D COMPR_ALG_LIBZ) + if (cs->compression_spec.algorithm =3D=3D PG_COMPRESSION_GZIP) EndCompressorZlib(AH, cs); #endif free(cs); @@ -453,20 +425,27 @@ cfopen_read(const char *path, const char *mode) { cfp *fp; =20 + pg_compress_specification compression_spec =3D {0}; + #ifdef HAVE_LIBZ if (hasSuffix(path, ".gz")) - fp =3D cfopen(path, mode, 1); + { + compression_spec.algorithm =3D PG_COMPRESSION_GZIP; + fp =3D cfopen(path, mode, compression_spec); + } else #endif { - fp =3D cfopen(path, mode, 0); + compression_spec.algorithm =3D PG_COMPRESSION_NONE; + fp =3D cfopen(path, mode, compression_spec); #ifdef HAVE_LIBZ if (fp =3D=3D NULL) { char *fname; =20 fname =3D psprintf("%s.gz", path); - fp =3D cfopen(fname, mode, 1); + compression_spec.algorithm =3D PG_COMPRESSION_GZIP; + fp =3D cfopen(fname, mode, compression_spec); free_keep_errno(fname); } #endif @@ -479,26 +458,27 @@ cfopen_read(const char *path, const char *mode) * be a filemode as accepted by fopen() and gzopen() that indicates writing * ("w", "wb", "a", or "ab"). * - * If 'compression' is non-zero, a gzip compressed stream is opened, and - * 'compression' indicates the compression level used. The ".gz" suffix - * is automatically added to 'path' in that case. + * If 'compression_spec.algorithm' is GZIP, a gzip compressed stream is op= ened, + * and 'compression_spec.level' used. The ".gz" suffix is automatically ad= ded to + * 'path' in that case. * * On failure, return NULL with an error code in errno. */ cfp * -cfopen_write(const char *path, const char *mode, int compression) +cfopen_write(const char *path, const char *mode, + const pg_compress_specification compression_spec) { cfp *fp; =20 - if (compression =3D=3D 0) - fp =3D cfopen(path, mode, 0); + if (compression_spec.algorithm =3D=3D PG_COMPRESSION_NONE) + fp =3D cfopen(path, mode, compression_spec); else { #ifdef HAVE_LIBZ char *fname; =20 fname =3D psprintf("%s.gz", path); - fp =3D cfopen(fname, mode, compression); + fp =3D cfopen(fname, mode, compression_spec); free_keep_errno(fname); #else pg_fatal("not built with zlib support"); @@ -509,26 +489,27 @@ cfopen_write(const char *path, const char *mode, int = compression) } =20 /* - * Opens file 'path' in 'mode'. If 'compression' is non-zero, the file + * Opens file 'path' in 'mode'. If compression is GZIP, the file * is opened with libz gzopen(), otherwise with plain fopen(). * * On failure, return NULL with an error code in errno. */ cfp * -cfopen(const char *path, const char *mode, int compression) +cfopen(const char *path, const char *mode, + const pg_compress_specification compression_spec) { cfp *fp =3D pg_malloc(sizeof(cfp)); =20 - if (compression !=3D 0) + if (compression_spec.algorithm =3D=3D PG_COMPRESSION_GZIP) { #ifdef HAVE_LIBZ - if (compression !=3D Z_DEFAULT_COMPRESSION) + if (compression_spec.level !=3D Z_DEFAULT_COMPRESSION) { /* user has specified a compression level, so tell zlib to use it */ char mode_compression[32]; =20 snprintf(mode_compression, sizeof(mode_compression), "%s%d", - mode, compression); + mode, compression_spec.level); fp->compressedfp =3D gzopen(path, mode_compression); } else diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index f635787692..6fad6c2cd5 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -21,12 +21,6 @@ #define ZLIB_OUT_SIZE 4096 #define ZLIB_IN_SIZE 4096 =20 -typedef enum -{ - COMPR_ALG_NONE, - COMPR_ALG_LIBZ -} CompressionAlgorithm; - /* Prototype for callback function to WriteDataToArchive() */ typedef void (*WriteFunc) (ArchiveHandle *AH, const char *buf, size_t len); =20 @@ -46,8 +40,10 @@ typedef size_t (*ReadFunc) (ArchiveHandle *AH, char **bu= f, size_t *buflen); /* struct definition appears in compress_io.c */ typedef struct CompressorState CompressorState; =20 -extern CompressorState *AllocateCompressor(int compression, WriteFunc writ= eF); -extern void ReadDataFromArchive(ArchiveHandle *AH, int compression, +extern CompressorState *AllocateCompressor(const pg_compress_specification= compression_spec, + WriteFunc writeF); +extern void ReadDataFromArchive(ArchiveHandle *AH, + const pg_compress_specification compression_spec, ReadFunc readF); extern void WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs, const void *data, size_t dLen); @@ -56,9 +52,13 @@ extern void EndCompressor(ArchiveHandle *AH, CompressorS= tate *cs); =20 typedef struct cfp cfp; =20 -extern cfp *cfopen(const char *path, const char *mode, int compression); +extern cfp *cfopen(const char *path, const char *mode, + const pg_compress_specification compression_spec); +extern cfp *cfdopen(int fd, const char *mode, + pg_compress_specification compression_spec); extern cfp *cfopen_read(const char *path, const char *mode); -extern cfp *cfopen_write(const char *path, const char *mode, int compressi= on); +extern cfp *cfopen_write(const char *path, const char *mode, + const pg_compress_specification compression_spec); extern int cfread(void *ptr, int size, cfp *fp); extern int cfwrite(const void *ptr, int size, cfp *fp); extern int cfgetc(cfp *fp); diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index e8b7898297..bc6b6594af 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -23,6 +23,7 @@ #ifndef PG_BACKUP_H #define PG_BACKUP_H =20 +#include "common/compression.h" #include "fe_utils/simple_list.h" #include "libpq-fe.h" =20 @@ -143,7 +144,8 @@ typedef struct _restoreOptions =20 int noDataForFailedTables; int exit_on_error; - int compression; + pg_compress_specification compression_spec; /* Specification for + * compression */ int suppressDumpWarnings; /* Suppress output of WARNING entries * to stderr */ bool single_txn; @@ -303,7 +305,8 @@ extern Archive *OpenArchive(const char *FileSpec, const= ArchiveFormat fmt); =20 /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fm= t, - const int compression, bool dosync, ArchiveMode mode, + const pg_compress_specification compression_spec, + bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker); =20 /* The --list option */ diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_back= up_archiver.c index f39c0fa36f..22238539fd 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -70,7 +70,8 @@ typedef struct _parallelReadyList =20 =20 static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat f= mt, - const int compression, bool dosync, ArchiveMode mode, + const pg_compress_specification compression_spec, + bool dosync, ArchiveMode mode, SetupWorkerPtrType setupWorkerPtr); static void _getObjectDescription(PQExpBuffer buf, const TocEntry *te); static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, bool isData); @@ -98,7 +99,8 @@ static int _discoverArchiveFormat(ArchiveHandle *AH); static int RestoringToDB(ArchiveHandle *AH); static void dump_lo_buf(ArchiveHandle *AH); static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim); -static void SetOutput(ArchiveHandle *AH, const char *filename, int compres= sion); +static void SetOutput(ArchiveHandle *AH, const char *filename, + const pg_compress_specification compression_spec); static OutputContext SaveOutput(ArchiveHandle *AH); static void RestoreOutput(ArchiveHandle *AH, OutputContext savedContext); =20 @@ -239,12 +241,13 @@ setupRestoreWorker(Archive *AHX) /* Public */ Archive * CreateArchive(const char *FileSpec, const ArchiveFormat fmt, - const int compression, bool dosync, ArchiveMode mode, + const pg_compress_specification compression_spec, + bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker) =20 { - ArchiveHandle *AH =3D _allocAH(FileSpec, fmt, compression, dosync, - mode, setupDumpWorker); + ArchiveHandle *AH =3D _allocAH(FileSpec, fmt, compression_spec, + dosync, mode, setupDumpWorker); =20 return (Archive *) AH; } @@ -254,7 +257,12 @@ CreateArchive(const char *FileSpec, const ArchiveForma= t fmt, Archive * OpenArchive(const char *FileSpec, const ArchiveFormat fmt) { - ArchiveHandle *AH =3D _allocAH(FileSpec, fmt, 0, true, archModeRead, setu= pRestoreWorker); + ArchiveHandle *AH; + pg_compress_specification compression_spec =3D {0}; + + compression_spec.algorithm =3D PG_COMPRESSION_NONE; + AH =3D _allocAH(FileSpec, fmt, compression_spec, true, + archModeRead, setupRestoreWorker); =20 return (Archive *) AH; } @@ -384,7 +392,8 @@ RestoreArchive(Archive *AHX) * Make sure we won't need (de)compression we haven't got */ #ifndef HAVE_LIBZ - if (AH->compression !=3D 0 && AH->PrintTocDataPtr !=3D NULL) + if (AH->compression_spec.algorithm =3D=3D PG_COMPRESSION_GZIP && + AH->PrintTocDataPtr !=3D NULL) { for (te =3D AH->toc->next; te !=3D AH->toc; te =3D te->next) { @@ -459,8 +468,8 @@ RestoreArchive(Archive *AHX) * Setup the output file if necessary. */ sav =3D SaveOutput(AH); - if (ropt->filename || ropt->compression) - SetOutput(AH, ropt->filename, ropt->compression); + if (ropt->filename || ropt->compression_spec.algorithm !=3D PG_COMPRESSIO= N_NONE) + SetOutput(AH, ropt->filename, ropt->compression_spec); =20 ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n"); =20 @@ -739,7 +748,7 @@ RestoreArchive(Archive *AHX) */ AH->stage =3D STAGE_FINALIZING; =20 - if (ropt->filename || ropt->compression) + if (ropt->filename || ropt->compression_spec.algorithm !=3D PG_COMPRESSIO= N_NONE) RestoreOutput(AH, sav); =20 if (ropt->useDB) @@ -969,6 +978,8 @@ NewRestoreOptions(void) opts->format =3D archUnknown; opts->cparams.promptPassword =3D TRI_DEFAULT; opts->dumpSections =3D DUMP_UNSECTIONED; + opts->compression_spec.algorithm =3D PG_COMPRESSION_NONE; + opts->compression_spec.level =3D 0; =20 return opts; } @@ -1115,23 +1126,28 @@ PrintTOCSummary(Archive *AHX) ArchiveHandle *AH =3D (ArchiveHandle *) AHX; RestoreOptions *ropt =3D AH->public.ropt; TocEntry *te; + pg_compress_specification out_compression_spec =3D {0}; teSection curSection; OutputContext sav; const char *fmtName; char stamp_str[64]; =20 + /* TOC is always uncompressed */ + out_compression_spec.algorithm =3D PG_COMPRESSION_NONE; + sav =3D SaveOutput(AH); if (ropt->filename) - SetOutput(AH, ropt->filename, 0 /* no compression */ ); + SetOutput(AH, ropt->filename, out_compression_spec); =20 if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT, localtime(&AH->createDate)) =3D=3D 0) strcpy(stamp_str, "[unknown]"); =20 ahprintf(AH, ";\n; Archive created at %s\n", stamp_str); - ahprintf(AH, "; dbname: %s\n; TOC Entries: %d\n; Compression:= %d\n", + ahprintf(AH, "; dbname: %s\n; TOC Entries: %d\n; Compression:= %s\n", sanitize_line(AH->archdbname, false), - AH->tocCount, AH->compression); + AH->tocCount, + get_compress_algorithm_name(AH->compression_spec.algorithm)); =20 switch (AH->format) { @@ -1485,7 +1501,8 @@ archprintf(Archive *AH, const char *fmt,...) *******************************/ =20 static void -SetOutput(ArchiveHandle *AH, const char *filename, int compression) +SetOutput(ArchiveHandle *AH, const char *filename, + const pg_compress_specification compression_spec) { int fn; =20 @@ -1508,12 +1525,12 @@ SetOutput(ArchiveHandle *AH, const char *filename, = int compression) =20 /* If compression explicitly requested, use gzopen */ #ifdef HAVE_LIBZ - if (compression !=3D 0) + if (compression_spec.algorithm =3D=3D PG_COMPRESSION_GZIP) { char fmode[14]; =20 /* Don't use PG_BINARY_x since this is zlib */ - sprintf(fmode, "wb%d", compression); + sprintf(fmode, "wb%d", compression_spec.level); if (fn >=3D 0) AH->OF =3D gzdopen(dup(fn), fmode); else @@ -2198,7 +2215,8 @@ _discoverArchiveFormat(ArchiveHandle *AH) */ static ArchiveHandle * _allocAH(const char *FileSpec, const ArchiveFormat fmt, - const int compression, bool dosync, ArchiveMode mode, + const pg_compress_specification compression_spec, + bool dosync, ArchiveMode mode, SetupWorkerPtrType setupWorkerPtr) { ArchiveHandle *AH; @@ -2249,7 +2267,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fm= t, AH->toc->prev =3D AH->toc; =20 AH->mode =3D mode; - AH->compression =3D compression; + AH->compression_spec =3D compression_spec; AH->dosync =3D dosync; =20 memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse)); @@ -2264,7 +2282,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fm= t, * Force stdin/stdout into binary mode if that is what we are using. */ #ifdef WIN32 - if ((fmt !=3D archNull || compression !=3D 0) && + if ((fmt !=3D archNull || compression_spec.algorithm !=3D PG_COMPRESSION_= NONE) && (AH->fSpec =3D=3D NULL || strcmp(AH->fSpec, "") =3D=3D 0)) { if (mode =3D=3D archModeWrite) @@ -3669,7 +3687,12 @@ WriteHead(ArchiveHandle *AH) AH->WriteBytePtr(AH, AH->intSize); AH->WriteBytePtr(AH, AH->offSize); AH->WriteBytePtr(AH, AH->format); - WriteInt(AH, AH->compression); + /* + * For now the compression type is implied by the level. This will need + * to change once support for more compression algorithms is added, + * requiring a format bump. + */ + WriteInt(AH, AH->compression_spec.level); crtm =3D *localtime(&AH->createDate); WriteInt(AH, crtm.tm_sec); WriteInt(AH, crtm.tm_min); @@ -3740,18 +3763,23 @@ ReadHead(ArchiveHandle *AH) pg_fatal("expected format (%d) differs from format found in file (%d)", AH->format, fmt); =20 + /* Guess the compression method based on the level */ + AH->compression_spec.algorithm =3D PG_COMPRESSION_NONE; if (AH->version >=3D K_VERS_1_2) { if (AH->version < K_VERS_1_4) - AH->compression =3D AH->ReadBytePtr(AH); + AH->compression_spec.level =3D AH->ReadBytePtr(AH); else - AH->compression =3D ReadInt(AH); + AH->compression_spec.level =3D ReadInt(AH); + + if (AH->compression_spec.level !=3D 0) + AH->compression_spec.algorithm =3D PG_COMPRESSION_GZIP; } else - AH->compression =3D Z_DEFAULT_COMPRESSION; + AH->compression_spec.algorithm =3D PG_COMPRESSION_GZIP; =20 #ifndef HAVE_LIBZ - if (AH->compression !=3D 0) + if (AH->compression_spec.algorithm =3D=3D PG_COMPRESSION_GZIP) pg_log_warning("archive is compressed, but this installation does not su= pport compression -- no data will be available"); #endif =20 diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_back= up_archiver.h index 42687c4ec8..a9560c6045 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -331,14 +331,8 @@ struct _archiveHandle DumpId *tableDataId; /* TABLE DATA ids, indexed by table dumpId */ =20 struct _tocEntry *currToc; /* Used when dumping data */ - int compression; /*--------- - * Compression requested on open(). - * Possible values for compression: - * -1 Z_DEFAULT_COMPRESSION - * 0 COMPRESSION_NONE - * 1-9 levels for gzip compression - *--------- - */ + pg_compress_specification compression_spec; /* Requested specification for + * compression */ bool dosync; /* data requested to be synced on sight */ ArchiveMode mode; /* File mode - r or w */ void *formatData; /* Header data specific to file format */ diff --git a/src/bin/pg_dump/pg_backup_custom.c b/src/bin/pg_dump/pg_backup= _custom.c index a0a55a1edd..f413d01fcb 100644 --- a/src/bin/pg_dump/pg_backup_custom.c +++ b/src/bin/pg_dump/pg_backup_custom.c @@ -298,7 +298,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) _WriteByte(AH, BLK_DATA); /* Block type */ WriteInt(AH, te->dumpId); /* For sanity check */ =20 - ctx->cs =3D AllocateCompressor(AH->compression, _CustomWriteFunc); + ctx->cs =3D AllocateCompressor(AH->compression_spec, _CustomWriteFunc); } =20 /* @@ -377,7 +377,7 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid) =20 WriteInt(AH, oid); =20 - ctx->cs =3D AllocateCompressor(AH->compression, _CustomWriteFunc); + ctx->cs =3D AllocateCompressor(AH->compression_spec, _CustomWriteFunc); } =20 /* @@ -566,7 +566,7 @@ _PrintTocData(ArchiveHandle *AH, TocEntry *te) static void _PrintData(ArchiveHandle *AH) { - ReadDataFromArchive(AH, AH->compression, _CustomReadFunc); + ReadDataFromArchive(AH, AH->compression_spec, _CustomReadFunc); } =20 static void diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_bac= kup_directory.c index 798182b6f7..53ef8db728 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -327,7 +327,8 @@ _StartData(ArchiveHandle *AH, TocEntry *te) =20 setFilePath(AH, fname, tctx->filename); =20 - ctx->dataFH =3D cfopen_write(fname, PG_BINARY_W, AH->compression); + ctx->dataFH =3D cfopen_write(fname, PG_BINARY_W, + AH->compression_spec); if (ctx->dataFH =3D=3D NULL) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -573,6 +574,7 @@ _CloseArchive(ArchiveHandle *AH) if (AH->mode =3D=3D archModeWrite) { cfp *tocFH; + pg_compress_specification compression_spec =3D {0}; char fname[MAXPGPATH]; =20 setFilePath(AH, fname, "toc.dat"); @@ -581,7 +583,8 @@ _CloseArchive(ArchiveHandle *AH) ctx->pstate =3D ParallelBackupStart(AH); =20 /* The TOC is always created uncompressed */ - tocFH =3D cfopen_write(fname, PG_BINARY_W, 0); + compression_spec.algorithm =3D PG_COMPRESSION_NONE; + tocFH =3D cfopen_write(fname, PG_BINARY_W, compression_spec); if (tocFH =3D=3D NULL) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH =3D tocFH; @@ -639,12 +642,14 @@ static void _StartBlobs(ArchiveHandle *AH, TocEntry *te) { lclContext *ctx =3D (lclContext *) AH->formatData; + pg_compress_specification compression_spec =3D {0}; char fname[MAXPGPATH]; =20 setFilePath(AH, fname, "blobs.toc"); =20 /* The blob TOC file is never compressed */ - ctx->blobsTocFH =3D cfopen_write(fname, "ab", 0); + compression_spec.algorithm =3D PG_COMPRESSION_NONE; + ctx->blobsTocFH =3D cfopen_write(fname, "ab", compression_spec); if (ctx->blobsTocFH =3D=3D NULL) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -662,7 +667,7 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid) =20 snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); =20 - ctx->dataFH =3D cfopen_write(fname, PG_BINARY_W, AH->compression); + ctx->dataFH =3D cfopen_write(fname, PG_BINARY_W, AH->compression_spec); =20 if (ctx->dataFH =3D=3D NULL) pg_fatal("could not open output file \"%s\": %m", fname); diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_ta= r.c index 402b93c610..99f3f5bcae 100644 --- a/src/bin/pg_dump/pg_backup_tar.c +++ b/src/bin/pg_dump/pg_backup_tar.c @@ -194,7 +194,7 @@ InitArchiveFmt_Tar(ArchiveHandle *AH) * possible since gzdopen uses buffered IO which totally screws file * positioning. */ - if (AH->compression !=3D 0) + if (AH->compression_spec.algorithm !=3D PG_COMPRESSION_NONE) pg_fatal("compression is not supported by tar archive format"); } else @@ -328,7 +328,7 @@ tarOpen(ArchiveHandle *AH, const char *filename, char m= ode) } } =20 - if (AH->compression =3D=3D 0) + if (AH->compression_spec.algorithm =3D=3D PG_COMPRESSION_NONE) tm->nFH =3D ctx->tarFH; else pg_fatal("compression is not supported by tar archive format"); @@ -383,7 +383,7 @@ tarOpen(ArchiveHandle *AH, const char *filename, char m= ode) =20 umask(old_umask); =20 - if (AH->compression =3D=3D 0) + if (AH->compression_spec.algorithm =3D=3D PG_COMPRESSION_NONE) tm->nFH =3D tm->tmpFH; else pg_fatal("compression is not supported by tar archive format"); @@ -401,7 +401,7 @@ tarOpen(ArchiveHandle *AH, const char *filename, char m= ode) static void tarClose(ArchiveHandle *AH, TAR_MEMBER *th) { - if (AH->compression !=3D 0) + if (AH->compression_spec.algorithm !=3D PG_COMPRESSION_NONE) pg_fatal("compression is not supported by tar archive format"); =20 if (th->mode =3D=3D 'w') @@ -800,7 +800,6 @@ _CloseArchive(ArchiveHandle *AH) memcpy(ropt, AH->public.ropt, sizeof(RestoreOptions)); ropt->filename =3D NULL; ropt->dropSchema =3D 1; - ropt->compression =3D 0; ropt->superuser =3D NULL; ropt->suppressDumpWarnings =3D true; =20 @@ -888,7 +887,7 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid) if (oid =3D=3D 0) pg_fatal("invalid OID for large object (%u)", oid); =20 - if (AH->compression !=3D 0) + if (AH->compression_spec.algorithm !=3D PG_COMPRESSION_NONE) pg_fatal("compression is not supported by tar archive format"); =20 sprintf(fname, "blob_%u.dat", oid); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index da427f4d4a..510555ef21 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -105,6 +105,8 @@ static Oid g_last_builtin_oid; /* value of the last bui= ltin oid */ /* The specified names/patterns should to match at least one entity */ static int strict_names =3D 0; =20 +static pg_compress_algorithm compression_algorithm =3D PG_COMPRESSION_NONE; + /* * Object inclusion/exclusion lists * @@ -340,10 +342,13 @@ main(int argc, char **argv) const char *dumpsnapshot =3D NULL; char *use_role =3D NULL; int numWorkers =3D 1; - int compressLevel =3D -1; int plainText =3D 0; ArchiveFormat archiveFormat =3D archUnknown; ArchiveMode archiveMode; + pg_compress_specification compression_spec =3D {0}; + char *compression_detail =3D NULL; + char *compression_algorithm_str =3D "none"; + char *error_detail =3D NULL; =20 static DumpOptions dopt; =20 @@ -561,10 +566,9 @@ main(int argc, char **argv) dopt.aclsSkip =3D true; break; =20 - case 'Z': /* Compression Level */ - if (!option_parse_int(optarg, "-Z/--compress", 0, 9, - &compressLevel)) - exit_nicely(1); + case 'Z': /* Compression */ + parse_compress_options(optarg, &compression_algorithm_str, + &compression_detail); break; =20 case 0: @@ -687,22 +691,33 @@ main(int argc, char **argv) if (archiveFormat =3D=3D archNull) plainText =3D 1; =20 - /* Custom and directory formats are compressed by default, others not */ - if (compressLevel =3D=3D -1) - { -#ifdef HAVE_LIBZ - if (archiveFormat =3D=3D archCustom || archiveFormat =3D=3D archDirector= y) - compressLevel =3D Z_DEFAULT_COMPRESSION; - else -#endif - compressLevel =3D 0; - } + /* + * Compression options + */ + if (!parse_compress_algorithm(compression_algorithm_str, + &compression_algorithm)) + pg_fatal("unrecognized compression algorithm: \"%s\"", + compression_algorithm_str); =20 -#ifndef HAVE_LIBZ - if (compressLevel !=3D 0) - pg_log_warning("requested compression not available in this installation= -- archive will be uncompressed"); - compressLevel =3D 0; -#endif + parse_compress_specification(compression_algorithm, compression_detail, + &compression_spec); + error_detail =3D validate_compress_specification(&compression_spec); + if (error_detail !=3D NULL) + pg_fatal("invalid compression specification: %s", + error_detail); + + switch (compression_algorithm) + { + case PG_COMPRESSION_NONE: + case PG_COMPRESSION_GZIP: + break; + case PG_COMPRESSION_ZSTD: + pg_fatal("compression with %s is not yet supported", "ZSTD"); + break; + case PG_COMPRESSION_LZ4: + pg_fatal("compression with %s is not yet supported", "LZ4"); + break; + } =20 /* * If emitting an archive format, we always want to emit a DATABASE item, @@ -716,8 +731,8 @@ main(int argc, char **argv) pg_fatal("parallel backup only supported by the directory format"); =20 /* Open the output file */ - fout =3D CreateArchive(filename, archiveFormat, compressLevel, dosync, - archiveMode, setupDumpWorker); + fout =3D CreateArchive(filename, archiveFormat, compression_spec, + dosync, archiveMode, setupDumpWorker); =20 /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); @@ -948,10 +963,7 @@ main(int argc, char **argv) ropt->sequence_data =3D dopt.sequence_data; ropt->binary_upgrade =3D dopt.binary_upgrade; =20 - if (compressLevel =3D=3D -1) - ropt->compression =3D 0; - else - ropt->compression =3D compressLevel; + ropt->compression_spec =3D compression_spec; =20 ropt->suppressDumpWarnings =3D true; /* We've already shown them */ =20 @@ -998,7 +1010,8 @@ help(const char *progname) printf(_(" -j, --jobs=3DNUM use this many parallel jobs to= dump\n")); printf(_(" -v, --verbose verbose mode\n")); printf(_(" -V, --version output version information, then= exit\n")); - printf(_(" -Z, --compress=3D0-9 compression level for compress= ed formats\n")); + printf(_(" -Z, --compress=3DMETHOD[:LEVEL]\n" + " compress as specified\n")); printf(_(" --lock-wait-timeout=3DTIMEOUT fail after waiting TIMEOUT for= a table lock\n")); printf(_(" --no-sync do not wait for changes to be wr= itten safely to disk\n")); printf(_(" -?, --help show this help, then exit\n")); diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl index a583c8a6d2..c8bc02126d 100644 --- a/src/bin/pg_dump/t/001_basic.pl +++ b/src/bin/pg_dump/t/001_basic.pl @@ -121,24 +121,46 @@ command_fails_like( 'pg_restore: cannot specify both --single-transaction and multiple jobs'); =20 command_fails_like( - [ 'pg_dump', '-Z', '-1' ], - qr/\Qpg_dump: error: -Z\/--compress must be in range 0..9\E/, - 'pg_dump: -Z/--compress must be in range'); + [ 'pg_dump', '--compress', 'garbage' ], + qr/\Qpg_dump: error: unrecognized compression algorithm/, + 'pg_dump: invalid --compress'); + +command_fails_like( + [ 'pg_dump', '--compress', 'none:1' ], + qr/\Qpg_dump: error: invalid compression specification: compression algor= ithm "none" does not accept a compression level\E/, + 'pg_dump: invalid compression specification: compression algorithm "none"= does not accept a compression level' +); + =20 if (check_pg_config("#define HAVE_LIBZ 1")) { + command_fails_like( + [ 'pg_dump', '-Z', '15' ], + qr/\Qpg_dump: error: invalid compression specification: compression algo= rithm "gzip" expects a compression level between 1 and 9 (default at -1)\E/, + 'pg_dump: invalid compression specification: must be in range'); + command_fails_like( [ 'pg_dump', '--compress', '1', '--format', 'tar' ], qr/\Qpg_dump: error: compression is not supported by tar archive format\= E/, 'pg_dump: compression is not supported by tar archive format'); + + command_fails_like( + [ 'pg_dump', '-Z', 'gzip:nonInt' ], + qr/\Qpg_dump: error: invalid compression specification: unrecognized com= pression option: "nonInt"\E/, + 'pg_dump: invalid compression specification: must be an integer'); } else { # --jobs > 1 forces an error with tar format. command_fails_like( - [ 'pg_dump', '--compress', '1', '--format', 'tar', '-j3' ], - qr/\Qpg_dump: warning: requested compression not available in this insta= llation -- archive will be uncompressed\E/, - 'pg_dump: warning: compression not available in this installation'); + [ 'pg_dump', '--format', 'tar', '-j3' ], + qr/\Qpg_dump: error: parallel backup only supported by the directory for= mat\E/, + 'pg_dump: warning: parallel backup not supported by tar format'); + + command_fails_like( + [ 'pg_dump', '-Z', 'gzip:nonInt', '--format', 'tar', '-j2' ], + qr/\Qpg_dump: error: invalid compression specification: unrecognized com= pression option\E/, + 'pg_dump: invalid compression specification: must be an integer'); } =20 command_fails_like( diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_du= mp.pl index fe53ed0f89..709db0986d 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -87,7 +87,7 @@ my %pgdump_runs =3D ( compile_option =3D> 'gzip', dump_cmd =3D> [ 'pg_dump', '--jobs=3D2', - '--format=3Ddirectory', '--compress=3D1', + '--format=3Ddirectory', '--compress=3Dgzip:1', "--file=3D$tempdir/compression_gzip_dir", 'postgres', ], # Give coverage for manually compressed blob.toc files during @@ -200,6 +200,7 @@ my %pgdump_runs =3D ( # Do not use --no-sync to give test coverage for data sync. defaults_custom_format =3D> { test_key =3D> 'defaults', + compile_option =3D> 'gzip', dump_cmd =3D> [ 'pg_dump', '-Fc', '-Z6', "--file=3D$tempdir/defaults_custom_format.dump", 'postgres', diff --git a/src/test/modules/test_pg_dump/t/001_base.pl b/src/test/modules= /test_pg_dump/t/001_base.pl index f5da6bf46d..19577ce0ea 100644 --- a/src/test/modules/test_pg_dump/t/001_base.pl +++ b/src/test/modules/test_pg_dump/t/001_base.pl @@ -20,6 +20,10 @@ my $tempdir =3D PostgreSQL::Test::Utils::tempdir; # to define how each test should (or shouldn't) treat a result # from a given run. # +# compile_option indicates if the commands run depend on a compilation +# option, if any. This can be used to control if tests should be +# skipped when a build dependency is not satisfied. +# # test_key indicates that a given run should simply use the same # set of like/unlike tests as another run, and which run that is. # @@ -90,6 +94,7 @@ my %pgdump_runs =3D ( }, defaults_custom_format =3D> { test_key =3D> 'defaults', + compile_option =3D> 'gzip', dump_cmd =3D> [ 'pg_dump', '--no-sync', '-Fc', '-Z6', "--file=3D$tempdir/defaults_custom_format.dump", 'postgres', @@ -749,6 +754,8 @@ $node->start; =20 my $port =3D $node->port; =20 +my $supports_gzip =3D check_pg_config("#define HAVE_LIBZ 1"); + ######################################### # Set up schemas, tables, etc, to be dumped. =20 @@ -792,6 +799,15 @@ foreach my $run (sort keys %pgdump_runs) =20 my $test_key =3D $run; =20 + # Skip command-level tests for gzip if there is no support for it. + if ( defined($pgdump_runs{$run}->{compile_option}) + && $pgdump_runs{$run}->{compile_option} eq 'gzip' + && !$supports_gzip) + { + note "$run: skipped due to no gzip support"; + next; + } + $node->command_ok(\@{ $pgdump_runs{$run}->{dump_cmd} }, "$run: pg_dump runs"); =20 diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index 8b9d9f4cad..2a015908f0 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -644,17 +644,35 @@ PostgreSQL documentation =20 - - + + [:level] + + [:level] - Specify the compression level to use. Zero means no compression. + Specify the compression method and/or the compression level to use. + The compression method can be set to gzip or + none for no compression. A compression level can + be optionally specified, by appending the level number after a col= on + (:). + + + If no compression level is specified, the default compression + level will be used. If only a level is specified without mentioning + an algorithm, gzip compression will be used if + the level is greater than 0, and no compression + will be used if the level is 0. + + + For the custom and directory archive formats, this specifies compr= ession of - individual table-data segments, and the default is to compress - at a moderate level. - For plain text output, setting a nonzero compression level causes - the entire output file to be compressed, as though it had been - fed through gzip; but the default is no= t to compress. + individual table-data segments, and the default is to compress usi= ng + gzip at a moderate level. For plain text output, + setting a nonzero compression level causes the entire output file = to be compressed, + as though it had been fed through gzip;= but the default + is not to compress. + + The tar archive format currently does not support compression at a= ll. diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs= =2Elist index 2f5802195d..58daeca831 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -428,7 +428,6 @@ CompiledExprState CompositeIOData CompositeTypeStmt CompoundAffixFlag -CompressionAlgorithm CompressionLocation CompressorState ComputeXidHorizonsResult --=20 2.38.1 --Ab/7/En0vSesZYLI-- --HOBd0O/w67ZHsizP Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmOIDAIACgkQnvQgOdby QH2+PQ/+PfsAady4PmAeQP0ySR0HLRwtX0bqxaI9Y04sNhSQWkpPoQtoU0PBGT9f BWx5LUIGjMygPVAilgyOqyvZTWxusDHjGp3Avt8wzNnVPghOYUAWr7UvYq4BiF3C uHzxVd8zbv1scWdcaa4cQhuUNSyncfd8sNKXbc5MlEl7lfDU7AFNAlDZyjI7uE7G lYI4xXpxsYoQJ7HwoeBQzFlUGxSVwQQUn2J3AqwhxVuNRDFueESYc/jqA8dm8VEl Dlha2rMBE27pPW0Dr9C5H7/VN9UVOWUtMAmpaHeeVak8qCx/Sr/xrsbCOvevWuUM 7bBGaFt15cx3w02v3XwKQovrWbiwnDLS43SZx/oQ2VeVAmIemVmEFG6bMmT5ogw6 dx+/PicHJiJDvaw6R8osrnhnEcuuSVYSbY2xuFBn6+ubrBJHriz/SdB2w0Xmhl+h hOSB+9wmy4IHx9FDF0p4CFcHesjDxsQbHrcjUVNnXxLHJ1SHz1fcngokTxZrNXVw LeW+BoFWxuVN+56ton2AmvFIAA9yJhWl79pPfT0SDu4/Wy3gP3SPooDBt+v/TJFm rXU1Ezzo9gImIriS0x7ap2gr5kufUM8X0xHVJIXq/WSbea3vQF8LiMFFrokDVeOi ypFTXSeXgk0HaIwA5IxmwTCFT+mBrULxXMQxYcLPNw4/Hj7ziWk= =+DD5 -----END PGP SIGNATURE----- --HOBd0O/w67ZHsizP--