Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ly4dM-0002E9-H6 for pgsql-hackers@arkaria.postgresql.org; Tue, 29 Jun 2021 03:42:01 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1ly4dL-00046O-GA for pgsql-hackers@arkaria.postgresql.org; Tue, 29 Jun 2021 03:41:59 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ly4dK-00046F-4U for pgsql-hackers@lists.postgresql.org; Tue, 29 Jun 2021 03:41:59 +0000 Received: from out2-smtp.messagingengine.com ([66.111.4.26]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ly4dE-0001HK-SQ for pgsql-hackers@postgresql.org; Tue, 29 Jun 2021 03:41:57 +0000 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 9958D5C02A6; Mon, 28 Jun 2021 23:41:51 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 28 Jun 2021 23:41:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=paquier.xyz; h= date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm2; bh=V7QfCdBkjLitw/k5ckWKE1UFDdj 8TJg5Bridjy9ZQLY=; b=MJ3qhcfM8TQUgCs6UtfKoRAE23Bpuj+KccYlsMKnGCf E5/kBb3ogAW8zKQQ0N12U1gXD84gC9M0xPVtCupK0uJp2aRtuGLr2ghc0PdD23Rn TyHEAdR/c7ZHlA+bgm72ita61sQylpnb7LJITMD4m5HFIgkXHQbPOSE4fRfcfchW tCrW/zpY+tVEnyaMrVpprZfQIJK6ljMp2jsetODBEVRcoImtTd5lX6H731z0KRsx Yvhp0+nVkKSYgn/N7dnwybFuwZkYsFnxd6WZ+xbq+kXnYLgiU+ckxfC3NCZ0C1JN yfSWp1+Oxgr16xTCwMVLmtkI2eARiGi0ykM3G/SEk9w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=V7QfCd BkjLitw/k5ckWKE1UFDdj8TJg5Bridjy9ZQLY=; b=oD9B57NRavlOfLUlpmXL2M 5twxfhUwiKGPl2jEMIAXQkhMC++CWEa/FMAa3O2eoW5WnK6UCJaZtu2FC+KXeY11 CG+YQKTnnmevpxV7EYH5b5zAm3xIKI/CuDc9W5v+qatkg89wjCLZ6W2mRKzUaCvT DEUCfh5ecxC3DepjRiUCc8Acc2ipnPVymNFLUUZ8SkEU0rvTAy0r2pPBqKuWk8EE EgJJ5u0/y5oQ625Frw1IPVw9pp2tJ7QJtaC7eO98qRIsDpxdYOajQJK6vZ1R4j/u df7YVPgE2DgBM40w1UqUx/7vhRwQeJPkTETCX3dl5MO0TYTapEpD6yIdyUx56xXA == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeehhedgieelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne gfrhhlucfvnfffucdljedtmdenucfjughrpeffhffvuffkfhggtggujgesghdtreertddt vdenucfhrhhomhepofhitghhrggvlhcurfgrqhhuihgvrhcuoehmihgthhgrvghlsehprg hquhhivghrrdighiiiqeenucggtffrrghtthgvrhhnpedvgeduuefhtdeuleettdevjeeh heeiveeuieegleetgeeljeelieeuieehgeevhfenucevlhhushhtvghrufhiiigvpedtne curfgrrhgrmhepmhgrihhlfhhrohhmpehmihgthhgrvghlsehprghquhhivghrrdighiii X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 28 Jun 2021 23:41:46 -0400 (EDT) Date: Tue, 29 Jun 2021 12:41:43 +0900 From: Michael Paquier To: Andrey Borodin Cc: Justin Pryzby , Peter Eisentraut , Heikki Linnakangas , pgsql-hackers , Thomas Munro , Peter Geoghegan , Andres Freund Subject: Re: Different compression methods for FPI Message-ID: References: <20210615161456.GP31772@telsasoft.com> <74CD8839-05DD-40F5-9609-58B79AD2151E@yandex-team.ru> <5C10FAD9-DC15-45E5-BCF9-B179A1ABA82E@yandex-team.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="MuS8zOFCU7tKaZ92" Content-Disposition: inline In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --MuS8zOFCU7tKaZ92 Content-Type: multipart/mixed; boundary="QfBf83bjxXZgFzLe" Content-Disposition: inline --QfBf83bjxXZgFzLe Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jun 22, 2021 at 09:11:26AM +0900, Michael Paquier wrote: > What about the following error then? Say: > "image at %X/%X compressed with LZ4 not supported by build, block > %d". >=20 >> Also, maybe %s can be inlined to lz4 in this case. >=20 > I think that's a remnant of the zstd part of the patch set, where I > wanted to have only one translatable message. Sure, I can align lz4 > with the message. >=20 >> 2. >> > const char *method =3D "???"; >> Maybe we can use something like "unknown" for unknown compression >> methods? Or is it too long string for waldump output? >=20 > A couple of extra bytes for pg_waldump will not matter much. Using > "unknown" is fine by me. I have kept 1. as-is for translability, and included 2. Now that v15 is open for business, I have looked again at this stuff this morning and committed the LZ4 part after some adjustments: - Avoid the use of default in the switches used for the compression, so as the compiler would warn when introducing a new value in the enum used for wal_compression. - Switched to LZ4_compress_default(). LZ4_compress_fast() could be used with LZ4_ACCELERATION_DEFAULT, but toast_compression.c uses the former. I got check-world tested with wal_compression =3D {lz4,pglz,off}, and did some manuals checks, including some stuff without --with-lz4. Attached is the rest of the patch set for zstd, for reference, rebased with the changes that have been committed. This still requires proper support in the MSVC scripts. My buildfarm machine has been changed to use wal_compression =3D lz4, while on it for HEAD runs. -- Michael --QfBf83bjxXZgFzLe Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v12-0001-Add-wal_compression-zstd.patch" Content-Transfer-Encoding: quoted-printable =46rom f529a6908d4a502dc0615d149335033c5a82bc41 Mon Sep 17 00:00:00 2001 =46rom: Michael Paquier Date: Tue, 29 Jun 2021 12:37:04 +0900 Subject: [PATCH v12] Add wal_compression=3Dzstd This is still a WIP patch, that requires checks with its APIs and adjustments with the Windows builds. --- src/include/access/xlog.h | 3 +- src/include/access/xlogrecord.h | 4 +- src/include/pg_config.h.in | 9 + src/backend/access/transam/xloginsert.c | 29 ++- src/backend/access/transam/xlogreader.c | 21 ++ src/backend/utils/misc/guc.c | 3 + src/backend/utils/misc/postgresql.conf.sample | 2 +- src/bin/pg_waldump/pg_waldump.c | 2 + doc/src/sgml/config.sgml | 6 +- doc/src/sgml/installation.sgml | 19 ++ configure | 217 ++++++++++++++++++ configure.ac | 33 +++ src/tools/msvc/Solution.pm | 2 + src/tools/msvc/config_default.pl | 3 +- 14 files changed, 346 insertions(+), 7 deletions(-) diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index ccfcf43d62..02126886a1 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -172,7 +172,8 @@ typedef enum WalCompression { WAL_COMPRESSION_NONE =3D 0, WAL_COMPRESSION_PGLZ, - WAL_COMPRESSION_LZ4 + WAL_COMPRESSION_LZ4, + WAL_COMPRESSION_ZSTD } WalCompression; =20 /* Recovery states */ diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecor= d.h index e06ee92a5e..895970cd36 100644 --- a/src/include/access/xlogrecord.h +++ b/src/include/access/xlogrecord.h @@ -149,8 +149,10 @@ typedef struct XLogRecordBlockImageHeader /* compression methods supported */ #define BKPIMAGE_COMPRESS_PGLZ 0x04 #define BKPIMAGE_COMPRESS_LZ4 0x08 +#define BKPIMAGE_COMPRESS_ZSTD 0x10 #define BKPIMAGE_COMPRESSED(info) \ - ((info & (BKPIMAGE_COMPRESS_PGLZ | BKPIMAGE_COMPRESS_LZ4)) !=3D 0) + ((info & (BKPIMAGE_COMPRESS_PGLZ | BKPIMAGE_COMPRESS_LZ4 | \ + BKPIMAGE_COMPRESS_ZSTD)) !=3D 0) =20 /* * Extra header information used when page image has "hole" and diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 783b8fc1ba..1951d88ac9 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -355,6 +355,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ =20 +/* Define to 1 if you have the `zstd' library (-lzstd). */ +#undef HAVE_LIBZSTD + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK =20 @@ -722,6 +725,9 @@ /* Define to 1 if the assembler supports X86_64's POPCNTQ instruction. */ #undef HAVE_X86_64_POPCNTQ =20 +/* Define to 1 if you have the header file. */ +#undef HAVE_ZSTD_H + /* Define to 1 if the system has the type `_Bool'. */ #undef HAVE__BOOL =20 @@ -953,6 +959,9 @@ /* Define to select Win32-style shared memory. */ #undef USE_WIN32_SHARED_MEMORY =20 +/* Define to 1 to build with zstd support. (--with-zstd) */ +#undef USE_ZSTD + /* Define to 1 if `wcstombs_l' requires . */ #undef WCSTOMBS_L_IN_XLOCALE =20 diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/t= ransam/xloginsert.c index 3d2c9c3e8c..525b73fbb0 100644 --- a/src/backend/access/transam/xloginsert.c +++ b/src/backend/access/transam/xloginsert.c @@ -44,9 +44,17 @@ #define LZ4_MAX_BLCKSZ 0 #endif =20 +#ifdef USE_ZSTD +#include +#define ZSTD_MAX_BLCKSZ ZSTD_COMPRESSBOUND(BLCKSZ) +#else +#define ZSTD_MAX_BLCKSZ 0 +#endif + +/* Buffer size required to store a compressed version of backup block imag= e */ #define PGLZ_MAX_BLCKSZ PGLZ_MAX_OUTPUT(BLCKSZ) =20 -#define COMPRESS_BUFSIZE Max(PGLZ_MAX_BLCKSZ, LZ4_MAX_BLCKSZ) +#define COMPRESS_BUFSIZE Max(Max(PGLZ_MAX_BLCKSZ, LZ4_MAX_BLCKSZ), ZSTD_MA= X_BLCKSZ) =20 /* * For each block reference registered with XLogRegisterBuffer, we fill in @@ -695,6 +703,14 @@ XLogRecordAssemble(RmgrId rmid, uint8 info, #endif break; =20 + case WAL_COMPRESSION_ZSTD: +#ifdef USE_ZSTD + bimg.bimg_info |=3D BKPIMAGE_COMPRESS_ZSTD; +#else + elog(ERROR, "ZSTD is not supported by this build"); +#endif + break; + case WAL_COMPRESSION_NONE: Assert(false); /* cannot happen */ break; @@ -903,6 +919,17 @@ XLogCompressBackupBlock(char *page, uint16 hole_offset= , uint16 hole_length, #endif break; =20 + case WAL_COMPRESSION_ZSTD: +#ifdef USE_ZSTD + len =3D ZSTD_compress(dest, COMPRESS_BUFSIZE, source, orig_len, + ZSTD_CLEVEL_DEFAULT); + if (ZSTD_isError(len)) + len =3D -1; +#else + elog(ERROR, "ZSTD is not supported by this build"); +#endif + break; + case WAL_COMPRESSION_NONE: Assert(false); /* cannot happen */ break; diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/t= ransam/xlogreader.c index 9a2cdf888e..ca1df891c6 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -21,6 +21,9 @@ #ifdef USE_LZ4 #include #endif +#ifdef USE_ZSTD +#include +#endif =20 #include "access/transam.h" #include "access/xlog_internal.h" @@ -1581,6 +1584,24 @@ RestoreBlockImage(XLogReaderState *record, uint8 blo= ck_id, char *page) "LZ4", block_id); return false; +#endif + } + else if ((bkpb->bimg_info & BKPIMAGE_COMPRESS_ZSTD) !=3D 0) + { +#ifdef USE_ZSTD + size_t decomp_result =3D ZSTD_decompress(tmp.data, + BLCKSZ-bkpb->hole_length, + ptr, bkpb->bimg_len); + + if (ZSTD_isError(decomp_result)) + decomp_success =3D false; +#else + report_invalid_record(record, "image at %X/%X compressed with %s not su= pported by build, block %d", + (uint32) (record->ReadRecPtr >> 32), + (uint32) record->ReadRecPtr, + "zstd", + block_id); + return false; #endif } else diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 480e8cd199..807f84abc4 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -544,6 +544,9 @@ static const struct config_enum_entry wal_compression_o= ptions[] =3D { {"pglz", WAL_COMPRESSION_PGLZ, false}, #ifdef USE_LZ4 {"lz4", WAL_COMPRESSION_LZ4, false}, +#endif +#ifdef USE_ZSTD + {"zstd", WAL_COMPRESSION_ZSTD, false}, #endif {"on", WAL_COMPRESSION_PGLZ, false}, {"off", WAL_COMPRESSION_NONE, false}, diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/ut= ils/misc/postgresql.conf.sample index b696abfe54..1401e40ffd 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -219,7 +219,7 @@ #wal_log_hints =3D off # also do full page writes of non-critical updates # (change requires restart) #wal_compression =3D off # enables compression of full-page writes; - # off, pglz, lz4, or on + # off, pglz, lz4, zstd, or on #wal_init_zero =3D on # zero-fill new WAL files #wal_recycle =3D on # recycle WAL files #wal_buffers =3D -1 # min 32kB, -1 sets based on shared_buffers diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldum= p.c index d83847b276..c9f85a6a91 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -547,6 +547,8 @@ XLogDumpDisplayRecord(XLogDumpConfig *config, XLogReade= rState *record) method =3D "pglz"; else if ((bimg_info & BKPIMAGE_COMPRESS_LZ4) !=3D 0) method =3D "lz4"; + else if ((bimg_info & BKPIMAGE_COMPRESS_ZSTD) !=3D 0) + method =3D "zstd"; else method =3D "unknown"; =20 diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 6098f6b020..7a69ae7470 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -3141,9 +3141,11 @@ include_dir 'conf.d' server compresses full page images written to WAL when is on or during a base ba= ckup. A compressed page image will be decompressed during WAL replay. - The supported methods are pglz and + The supported methods are pglz, lz4 (if PostgreSQL w= as - compiled with ). The default value is + compiled with ) and + zstd (if PostgreSQL = was + compiled with . The default value is off. Only superusers can change this setting. =20 diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml index 61d0bc8c43..5b024a7bb1 100644 --- a/doc/src/sgml/installation.sgml +++ b/doc/src/sgml/installation.sgml @@ -275,6 +275,14 @@ su - postgres =20 + + + The ZSTD library can be used to enable + compression using that method; see + . + + + To build the PostgreSQL documentation, @@ -986,6 +994,17 @@ build-postgresql: =20 + + + + + Build with ZSTD compression support. + This enables use of ZSTD for + compression of WAL data. + + + +