From: Justin Pryzby <pryzby@telsasoft.com>
To: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: pgsql-hackers@postgresql.org
Subject: Re: Different compression methods for FPI
Date: Fri, 12 Mar 2021 01:45:47 -0600
Message-ID: <20210312074547.GQ8796@telsasoft.com> (raw)
In-Reply-To: <C57172E1-B9D7-4FDA-BCAB-76FEEFCBA603@yandex-team.ru>
References: <3037310D-ECB7-4BF1-AF20-01C10BB33A33@yandex-team.ru>
<20210228230817.GE20769@telsasoft.com>
<YDx0KDjQEaK+sfeB@paquier.xyz>
<20210301050327.GK20769@telsasoft.com>
<C57172E1-B9D7-4FDA-BCAB-76FEEFCBA603@yandex-team.ru>
On Sat, Mar 06, 2021 at 12:29:14PM +0500, Andrey Borodin wrote:
> > 1 марта 2021 г., в 10:03, Justin Pryzby <pryzby@telsasoft.com> написал(а):
>
> Justin, Michael, thanks for comments!
>
> As far as I understood TODO list for the patch looks as follows:
Your patch can be simplified some, and then the only ifdef are in two short
functions. Moving the compression calls to another function/file is hardly
worth it, and anyone that implements a generic compression API could refactor
easily, if it's a win. So I don't want to impose the burden on your small
patch of setting up the compression API for everyone else's patches. Since
this is non-streaming compression, the calls are trivial.
One advantage of a generic API is that it's a good place to handle things like
compression options, like zstd:9 or zstd:3,rsyncable (I am not suggesting this
syntax).
Today, I re-sent an Dillip's patch with a change to use pkg-config for liblz4,
and it now also compiles on mac, so I used those changes to configure.ac (using
pkg-config) and src/tools/msvc/Solution.pm, and changed HAVE_LIBLZ4 to USE_LZ4.
This also resolves conflict with 32fd2b57d7f64948e649fc205c43f007762ecaac.
wal_compression_method is PGC_SIGHUP, not SUSET.
I think that the COMPRESSION_ID should have a prefix like XLOG_* - but didn't
do it here.
Does this patch need to bump XLOG_PAGE_MAGIC ?
wal_compression_options: I made this conditional compilation, so the GUC
machinery rejects methods which aren't supported. That means that xloginsert
doesn't need to check for unsupported methods. sync_method_options also uses
conditional compilation, but a GUC "check" hook would be more friendly, since
it could distinguish between "not supported" and "valid":
|ERROR: invalid value for parameter "wal_compression_method": "lz4"
--
Justin
Attachments:download all (2 files, .tar.gz)0001-Use-different-compression-methods-for-FPIs.patch (text/x-diff, 20.2K) download
From 387fa7d10ecacd0c3c15b0bded769bddc234eb53 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 27 Feb 2021 09:03:50 +0500
Subject: [PATCH 1/2] Use different compression methods for FPIs
---
configure | 170 ++++++++++++++++++++++++
configure.ac | 20 +++
src/backend/Makefile | 2 +-
src/backend/access/transam/xlog.c | 13 ++
src/backend/access/transam/xloginsert.c | 58 +++++++-
src/backend/access/transam/xlogreader.c | 74 ++++++++++-
src/backend/utils/misc/guc.c | 11 ++
src/include/access/xlog.h | 1 +
src/include/access/xlog_internal.h | 7 +
src/include/access/xlogreader.h | 1 +
src/include/access/xlogrecord.h | 7 +-
src/include/pg_config.h.in | 3 +
src/tools/msvc/Solution.pm | 1 +
13 files changed, 357 insertions(+), 11 deletions(-)
diff --git a/configure b/configureindex fad817bb38..440d1e8ce5 100755--- a/configure+++ b/configure@@ -699,6 +699,9 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+LZ4_LIBS+LZ4_CFLAGS+with_lz4
with_zlib
with_system_tzdata
with_libxslt
@@ -864,6 +867,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_lz4
with_gnu_ld
with_ssl
with_openssl
@@ -891,6 +895,8 @@ ICU_LIBS
XML2_CONFIG
XML2_CFLAGS
XML2_LIBS
+LZ4_CFLAGS+LZ4_LIBS
LDFLAGS_EX
LDFLAGS_SL
PERL
@@ -1569,6 +1575,7 @@ Optional Packages:
--with-system-tzdata=DIR
use system time zone data in DIR
--without-zlib do not use Zlib
+ --with-lz4 build with LZ4 support
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
--with-ssl=LIB use LIB for SSL/TLS support (openssl)
--with-openssl obsolete spelling of --with-ssl=openssl
@@ -1596,6 +1603,8 @@ Some influential environment variables:
XML2_CONFIG path to xml2-config utility
XML2_CFLAGS C compiler flags for XML2, overriding pkg-config
XML2_LIBS linker flags for XML2, overriding pkg-config
+ LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config+ LZ4_LIBS linker flags for LZ4, overriding pkg-config
LDFLAGS_EX extra linker flags for linking executables only
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
@@ -8563,6 +8572,137 @@ fi+#+# LZ4+#+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5+$as_echo_n "checking whether to build with LZ4 support... " >&6; }++++# Check whether --with-lz4 was given.+if test "${with_lz4+set}" = set; then :+ withval=$with_lz4;+ case $withval in+ yes)++$as_echo "#define USE_LZ4 1" >>confdefs.h++ ;;+ no)+ :+ ;;+ *)+ as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5+ ;;+ esac++else+ with_lz4=no++fi+++{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5+$as_echo "$with_lz4" >&6; }+++if test "$with_lz4" = yes; then++pkg_failed=no+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5+$as_echo_n "checking for liblz4... " >&6; }++if test -n "$LZ4_CFLAGS"; then+ pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS"+ elif test -n "$PKG_CONFIG"; then+ if test -n "$PKG_CONFIG" && \+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5+ ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5+ ac_status=$?+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5+ test $ac_status = 0; }; then+ pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null`+ test "x$?" != "x0" && pkg_failed=yes+else+ pkg_failed=yes+fi+ else+ pkg_failed=untried+fi+if test -n "$LZ4_LIBS"; then+ pkg_cv_LZ4_LIBS="$LZ4_LIBS"+ elif test -n "$PKG_CONFIG"; then+ if test -n "$PKG_CONFIG" && \+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5+ ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5+ ac_status=$?+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5+ test $ac_status = 0; }; then+ pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null`+ test "x$?" != "x0" && pkg_failed=yes+else+ pkg_failed=yes+fi+ else+ pkg_failed=untried+fi++++if test $pkg_failed = yes; then+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5+$as_echo "no" >&6; }++if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then+ _pkg_short_errors_supported=yes+else+ _pkg_short_errors_supported=no+fi+ if test $_pkg_short_errors_supported = yes; then+ LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1`+ else+ LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1`+ fi+ # Put the nasty error message in config.log where it belongs+ echo "$LZ4_PKG_ERRORS" >&5++ as_fn_error $? "Package requirements (liblz4) were not met:++$LZ4_PKG_ERRORS++Consider adjusting the PKG_CONFIG_PATH environment variable if you+installed software in a non-standard prefix.++Alternatively, you may set the environment variables LZ4_CFLAGS+and LZ4_LIBS to avoid the need to call pkg-config.+See the pkg-config man page for more details." "$LINENO" 5+elif test $pkg_failed = untried; then+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5+$as_echo "no" >&6; }+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it+is in your PATH or set the PKG_CONFIG environment variable to the full+path to pkg-config.++Alternatively, you may set the environment variables LZ4_CFLAGS+and LZ4_LIBS to avoid the need to call pkg-config.+See the pkg-config man page for more details.++To get pkg-config, see <http://pkg-config.freedesktop.org/>.+See \`config.log' for more details" "$LINENO" 5; }+else+ LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS+ LZ4_LIBS=$pkg_cv_LZ4_LIBS+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5+$as_echo "yes" >&6; }++fi+ LIBS="$LZ4_LIBS $LIBS"+ CFLAGS="$LZ4_CFLAGS $CFLAGS"+fi+
#
# Assignments
#
@@ -13376,6 +13516,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5
fi
+fi++if test "$with_lz4" = yes; then+ for ac_header in lz4/lz4.h+do :+ ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default"+if test "x$ac_cv_header_lz4_lz4_h" = xyes; then :+ cat >>confdefs.h <<_ACEOF+#define HAVE_LZ4_LZ4_H 1+_ACEOF++else+ for ac_header in lz4.h+do :+ ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default"+if test "x$ac_cv_header_lz4_h" = xyes; then :+ cat >>confdefs.h <<_ACEOF+#define HAVE_LZ4_H 1+_ACEOF++else+ as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5+fi++done++fi++done+
fi
if test "$with_gssapi" = yes ; then
diff --git a/configure.ac b/configure.acindex 0ed53571dd..780791ae8a 100644--- a/configure.ac+++ b/configure.ac@@ -986,6 +986,21 @@ PGAC_ARG_BOOL(with, zlib, yes,
[do not use Zlib])
AC_SUBST(with_zlib)
+#+# LZ4+#+AC_MSG_CHECKING([whether to build with LZ4 support])+PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support],+ [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])])+AC_MSG_RESULT([$with_lz4])+AC_SUBST(with_lz4)++if test "$with_lz4" = yes; then+ PKG_CHECK_MODULES(LZ4, liblz4)+ LIBS="$LZ4_LIBS $LIBS"+ CFLAGS="$LZ4_CFLAGS $CFLAGS"+fi+
#
# Assignments
#
@@ -1407,6 +1422,11 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_lz4" = yes; then+ AC_CHECK_HEADERS(lz4/lz4.h, [],+ [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])])+fi+
if test "$with_gssapi" = yes ; then
AC_CHECK_HEADERS(gssapi/gssapi.h, [],
[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/src/backend/Makefile b/src/backend/Makefileindex 0da848b1fd..3af216ddfc 100644--- a/src/backend/Makefile+++ b/src/backend/Makefile@@ -48,7 +48,7 @@ OBJS = \
LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS)) $(LDAP_LIBS_BE) $(ICU_LIBS)
# The backend doesn't need everything that's in LIBS, however
-LIBS := $(filter-out -lz -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))+LIBS := $(filter-out -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))
ifeq ($(with_systemd),yes)
LIBS += -lsystemd
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex e04250f4e9..82cedb0a41 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -99,6 +99,7 @@ bool EnableHotStandby = false;
bool fullPageWrites = true;
bool wal_log_hints = false;
bool wal_compression = false;
+int wal_compression_method = PGLZ_COMPRESSION_ID;
char *wal_consistency_checking_string = NULL;
bool *wal_consistency_checking = NULL;
bool wal_init_zero = true;
@@ -180,6 +181,18 @@ const struct config_enum_entry recovery_target_action_options[] = {
{NULL, 0, false}
};
+/* Note that due to conditional compilation, offsets within the array are not static */+const struct config_enum_entry wal_compression_options[] = {+ {"pglz", PGLZ_COMPRESSION_ID, false},+#ifdef USE_LZ4+ {"lz4", LZ4_COMPRESSION_ID, false},+#endif+#ifdef HAVE_LIBZ+ {"zlib", ZLIB_COMPRESSION_ID, false},+#endif+ {NULL, 0, false}+};+
/*
* Statistics for current checkpoint are collected in this global struct.
* Because only the checkpointer or a stand-alone backend can perform
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.cindex 7052dc245e..256931a93d 100644--- a/src/backend/access/transam/xloginsert.c+++ b/src/backend/access/transam/xloginsert.c@@ -33,6 +33,14 @@
#include "storage/proc.h"
#include "utils/memutils.h"
+#ifdef USE_LZ4+#include "lz4.h"+#endif++#ifdef HAVE_LIBZ+#include <zlib.h>+#endif+
/* Buffer size required to store a compressed version of backup block image */
#define PGLZ_MAX_BLCKSZ PGLZ_MAX_OUTPUT(BLCKSZ)
@@ -113,7 +121,8 @@ static XLogRecData *XLogRecordAssemble(RmgrId rmid, uint8 info,
XLogRecPtr RedoRecPtr, bool doPageWrites,
XLogRecPtr *fpw_lsn, int *num_fpi);
static bool XLogCompressBackupBlock(char *page, uint16 hole_offset,
- uint16 hole_length, char *dest, uint16 *dlen);+ uint16 hole_length, char *dest,+ uint16 *dlen, CompressionId compression);
/*
* Begin constructing a WAL record. This must be called before the
@@ -630,11 +639,12 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
*/
if (wal_compression)
{
+ bimg.compression_method = wal_compression_method;
is_compressed =
XLogCompressBackupBlock(page, bimg.hole_offset,
cbimg.hole_length,
regbuf->compressed_page,
- &compressed_len);+ &compressed_len, bimg.compression_method);
}
/*
@@ -827,7 +837,7 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
*/
static bool
XLogCompressBackupBlock(char *page, uint16 hole_offset, uint16 hole_length,
- char *dest, uint16 *dlen)+ char *dest, uint16 *dlen, CompressionId compression)
{
int32 orig_len = BLCKSZ - hole_length;
int32 len;
@@ -853,12 +863,50 @@ XLogCompressBackupBlock(char *page, uint16 hole_offset, uint16 hole_length,
else
source = page;
+ switch (compression)+ {+#ifdef USE_LZ4+ case LZ4_COMPRESSION_ID:+ len = LZ4_compress_fast(source, dest, orig_len, PGLZ_MAX_BLCKSZ, 1);+ break;+#endif++#ifdef HAVE_LIBZ+ case ZLIB_COMPRESSION_ID:+ {+ unsigned long len_l = PGLZ_MAX_BLCKSZ;+ if (compress((Bytef*)dest, &len_l, (Bytef*)source, orig_len) != Z_OK)+ {+ // XXX: using an interface other than compress() would allow giving a better error message+ ereport(ERROR,+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),+ errmsg("failed compressing zlib")));+ len_l = -1;+ }+ len = len_l;+ break;+ }+#endif++ case PGLZ_COMPRESSION_ID:+ len = pglz_compress(source, orig_len, dest, PGLZ_strategy_default);+ break;++ default:+ /*+ * It should be impossible to get here for zlib and lz4, which+ * cannot be assigned if they're not enabled at compile time.+ */+ ereport(ERROR,+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),+ errmsg("unknown compression method requested: %d", compression)));+ }+
/*
- * We recheck the actual size even if pglz_compress() reports success and+ * We recheck the actual size even if compression reports success and
* see if the number of bytes saved by compression is larger than the
* length of extra data needed for the compressed version of block image.
*/
- len = pglz_compress(source, orig_len, dest, PGLZ_strategy_default);
if (len >= 0 &&
len + extra_bytes < orig_len)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.cindex 42738eb940..10e8766e01 100644--- a/src/backend/access/transam/xlogreader.c+++ b/src/backend/access/transam/xlogreader.c@@ -33,6 +33,14 @@
#include "utils/memutils.h"
#endif
+#ifdef USE_LZ4+#include "lz4.h"+#endif++#ifdef HAVE_LIBZ+#include <zlib.h>+#endif+
static void report_invalid_record(XLogReaderState *state, const char *fmt,...)
pg_attribute_printf(2, 3);
static bool allocate_recordbuf(XLogReaderState *state, uint32 reclength);
@@ -1286,6 +1294,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&blk->bimg_len, sizeof(uint16));
COPY_HEADER_FIELD(&blk->hole_offset, sizeof(uint16));
+ COPY_HEADER_FIELD(&blk->compression_method, sizeof(uint8));
COPY_HEADER_FIELD(&blk->bimg_info, sizeof(uint8));
blk->apply_image = ((blk->bimg_info & BKPIMAGE_APPLY) != 0);
@@ -1535,6 +1544,31 @@ XLogRecGetBlockData(XLogReaderState *record, uint8 block_id, Size *len)
}
}
+/*+ * Return a statically allocated string associated with the given compression+ * method. This is similar to the guc, but isn't subject to conditional+ * compilation.+ */+static const char *+wal_compression_name(CompressionId compression)+{+ /*+ * This could index into the guc array, except that it's compiled+ * conditionally and unsupported methods are elided.+ */+ switch (compression)+ {+ case PGLZ_COMPRESSION_ID:+ return "pglz";+ case LZ4_COMPRESSION_ID:+ return "lz4";+ case ZLIB_COMPRESSION_ID:+ return "zlib";+ default:+ return "???";+ }+}+
/*
* Restore a full-page image from a backup block attached to an XLOG record.
*
@@ -1558,8 +1592,44 @@ RestoreBlockImage(XLogReaderState *record, uint8 block_id, char *page)
if (bkpb->bimg_info & BKPIMAGE_IS_COMPRESSED)
{
/* If a backup block image is compressed, decompress it */
- if (pglz_decompress(ptr, bkpb->bimg_len, tmp.data,- BLCKSZ - bkpb->hole_length, true) < 0)+ int32 decomp_result = -1;+ switch (bkpb->compression_method)+ {+ case PGLZ_COMPRESSION_ID:+ decomp_result = pglz_decompress(ptr, bkpb->bimg_len, tmp.data,+ BLCKSZ - bkpb->hole_length, true);+ break;++#ifdef USE_LZ4+ case LZ4_COMPRESSION_ID:+ decomp_result = LZ4_decompress_safe(ptr, tmp.data, bkpb->bimg_len, BLCKSZ);+ break;+#endif++#ifdef HAVE_LIBZ+ case ZLIB_COMPRESSION_ID:+ {+ unsigned long decomp_result_l = 0;+ decomp_result_l = BLCKSZ - bkpb->hole_length;+ if (uncompress((Bytef*)tmp.data, &decomp_result_l, (Bytef*)ptr, bkpb->bimg_len) == Z_OK)+ decomp_result = decomp_result_l;+ else+ decomp_result = -1;+ break;+ }+#endif++ default:+ report_invalid_record(record, "image at %X/%X is compressed with unsupported codec, block %d (%d/%s)",+ (uint32) (record->ReadRecPtr >> 32),+ (uint32) record->ReadRecPtr,+ block_id,+ bkpb->compression_method,+ wal_compression_name(bkpb->compression_method));+ return false;+ }++ if (decomp_result < 0)
{
report_invalid_record(record, "invalid compressed image at %X/%X, block %d",
LSN_FORMAT_ARGS(record->ReadRecPtr),
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.cindex 855076b1fd..42253c11f1 100644--- a/src/backend/utils/misc/guc.c+++ b/src/backend/utils/misc/guc.c@@ -508,6 +508,7 @@ extern const struct config_enum_entry archive_mode_options[];
extern const struct config_enum_entry recovery_target_action_options[];
extern const struct config_enum_entry sync_method_options[];
extern const struct config_enum_entry dynamic_shared_memory_options[];
+extern const struct config_enum_entry wal_compression_options[];
/*
* GUC option variables that are exported from this module
@@ -4721,6 +4722,16 @@ static struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {+ {"wal_compression_method", PGC_SIGHUP, WAL_SETTINGS,+ gettext_noop("Set the method used to compress full page images in the WAL."),+ NULL+ },+ &wal_compression_method,+ PGLZ_COMPRESSION_ID, wal_compression_options,+ NULL, NULL, NULL+ },+
{
{"dynamic_shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Selects the dynamic shared memory implementation used."),
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.hindex 6d384d3ce6..5f8c5f120e 100644--- a/src/include/access/xlog.h+++ b/src/include/access/xlog.h@@ -184,6 +184,7 @@ typedef enum RecoveryPauseState
} RecoveryPauseState;
extern PGDLLIMPORT int wal_level;
+extern PGDLLIMPORT int wal_compression_method;
/* Is WAL archiving enabled (always or only while server is running normally)? */
#define XLogArchivingActive() \
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.hindex b23e286406..c8768d7dbe 100644--- a/src/include/access/xlog_internal.h+++ b/src/include/access/xlog_internal.h@@ -324,4 +324,11 @@ extern bool InArchiveRecovery;
extern bool StandbyMode;
extern char *recoveryRestoreCommand;
+typedef enum CompressionId+{+ PGLZ_COMPRESSION_ID = 0,+ LZ4_COMPRESSION_ID = 1,+ ZLIB_COMPRESSION_ID = 2,+} CompressionId;+
#endif /* XLOG_INTERNAL_H */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.hindex 21d200d3df..3d19c315d7 100644--- a/src/include/access/xlogreader.h+++ b/src/include/access/xlogreader.h@@ -133,6 +133,7 @@ typedef struct
bool apply_image; /* has image that should be restored */
char *bkp_image;
uint16 hole_offset;
+ uint8 compression_method;
uint16 hole_length;
uint16 bimg_len;
uint8 bimg_info;
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.hindex 80c92a2498..3a885dc0c2 100644--- a/src/include/access/xlogrecord.h+++ b/src/include/access/xlogrecord.h@@ -129,9 +129,10 @@ typedef struct XLogRecordBlockHeader
*/
typedef struct XLogRecordBlockImageHeader
{
- uint16 length; /* number of page image bytes */- uint16 hole_offset; /* number of bytes before "hole" */- uint8 bimg_info; /* flag bits, see below */+ uint16 length; /* number of page image bytes */+ uint16 hole_offset; /* number of bytes before "hole" */+ uint8 compression_method; /* compression method used for image */+ uint8 bimg_info; /* flag bits, see below */
/*
* If BKPIMAGE_HAS_HOLE and BKPIMAGE_IS_COMPRESSED, an
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.inindex 7a7cc21d8d..0a6422da4f 100644--- a/src/include/pg_config.h.in+++ b/src/include/pg_config.h.in@@ -899,6 +899,9 @@
/* Define to 1 to build with LLVM based JIT support. (--with-llvm) */
#undef USE_LLVM
+/* Define to 1 to build with LZ4 support (--with-lz4) */+#undef USE_LZ4+
/* Define to select named POSIX semaphores. */
#undef USE_NAMED_POSIX_SEMAPHORES
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pmindex a4f5cc4bdb..14605371bb 100644--- a/src/tools/msvc/Solution.pm+++ b/src/tools/msvc/Solution.pm@@ -485,6 +485,7 @@ sub GenerateFiles
USE_ICU => $self->{options}->{icu} ? 1 : undef,
USE_LIBXML => undef,
USE_LIBXSLT => undef,
+ USE_LZ4 => undef,
USE_LDAP => $self->{options}->{ldap} ? 1 : undef,
USE_LLVM => undef,
USE_NAMED_POSIX_SEMAPHORES => undef,
--
2.17.0
0002-default-to-with-lz4.patch (text/x-diff, 2.9K) download
From 672a9763121385f6a5b8153b21696e98e6ae724e Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Thu, 11 Mar 2021 17:36:24 -0600
Subject: [PATCH 2/2] default to --with-lz4
this is meant to exercise the new feature in the CIs, and not meant to be
merged
---
configure | 6 ++++--
configure.ac | 4 ++--
src/backend/access/transam/xlog.c | 2 +-
src/backend/utils/misc/guc.c | 4 ++--
4 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/configure b/configureindex 440d1e8ce5..b411ed1cb4 100755--- a/configure+++ b/configure@@ -1575,7 +1575,7 @@ Optional Packages:
--with-system-tzdata=DIR
use system time zone data in DIR
--without-zlib do not use Zlib
- --with-lz4 build with LZ4 support+ --without-lz4 build without LZ4 support
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
--with-ssl=LIB use LIB for SSL/TLS support (openssl)
--with-openssl obsolete spelling of --with-ssl=openssl
@@ -8598,7 +8598,9 @@ $as_echo "#define USE_LZ4 1" >>confdefs.h
esac
else
- with_lz4=no+ with_lz4=yes++$as_echo "#define USE_LZ4 1" >>confdefs.h
fi
diff --git a/configure.ac b/configure.acindex 780791ae8a..392784d55c 100644--- a/configure.ac+++ b/configure.ac@@ -990,8 +990,8 @@ AC_SUBST(with_zlib)
# LZ4
#
AC_MSG_CHECKING([whether to build with LZ4 support])
-PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support],- [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])])+PGAC_ARG_BOOL(with, lz4, yes, [build without LZ4 support],+ [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build without LZ4 support. (--without-lz4)])])
AC_MSG_RESULT([$with_lz4])
AC_SUBST(with_lz4)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex 82cedb0a41..321558a2c6 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -99,7 +99,7 @@ bool EnableHotStandby = false;
bool fullPageWrites = true;
bool wal_log_hints = false;
bool wal_compression = false;
-int wal_compression_method = PGLZ_COMPRESSION_ID;+int wal_compression_method = LZ4_COMPRESSION_ID;
char *wal_consistency_checking_string = NULL;
bool *wal_consistency_checking = NULL;
bool wal_init_zero = true;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.cindex 42253c11f1..6aa14694f4 100644--- a/src/backend/utils/misc/guc.c+++ b/src/backend/utils/misc/guc.c@@ -1269,7 +1269,7 @@ static struct config_bool ConfigureNamesBool[] =
NULL
},
&wal_compression,
- false,+ true,
NULL, NULL, NULL
},
@@ -4728,7 +4728,7 @@ static struct config_enum ConfigureNamesEnum[] =
NULL
},
&wal_compression_method,
- PGLZ_COMPRESSION_ID, wal_compression_options,+ LZ4_COMPRESSION_ID, wal_compression_options,
NULL, NULL, NULL
},
--
2.17.0
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: pryzby@telsasoft.com, x4mmm@yandex-team.ru, michael@paquier.xyz
Subject: Re: Different compression methods for FPI
In-Reply-To: <20210312074547.GQ8796@telsasoft.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox