pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedzstd compression for pg_dump
46+ messages / 11 participants
[nested] [flat]
* zstd compression for pg_dump
@ 2020-12-21 19:49 Justin Pryzby <pryzby@telsasoft.com>
2020-12-21 20:02 ` Re: zstd compression for pg_dump Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2020-12-21 19:49 UTC (permalink / raw)
To: pgsql-hackers
I found that our largest tables are 40% smaller and 20% faster to pipe
pg_dump -Fc -Z0 |zstd relative to native zlib
So I wondered how much better when integrated in pg_dump, and found that
there's some additional improvement, but a big disadvantage of piping through
zstd is that it's not identified as a PGDMP file, and, /usr/bin/file on centos7
fails to even identify zstd by its magic number..
I looked for previous discussion about alternate compressions, but didn't find
anything for pg_dump.
--
Justin
Attachments:
[text/x-diff] 0001-fix-pre-existing-docs-comments.patch (1.7K, ../../20201221194924.GI30237@telsasoft.com/2-0001-fix-pre-existing-docs-comments.patch)
download | inline diff:
From c506c8a93ae70726a5b4b33a1d0098caf5665f3a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 21 Dec 2020 00:32:32 -0600
Subject: [PATCH 1/7] fix pre-existing docs/comments
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
src/bin/pg_dump/pg_backup_directory.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0aa35cf0c3..dcb25dc3cd 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -621,7 +621,7 @@ PostgreSQL documentation
<listitem>
<para>
Specify the compression level to use. Zero means no compression.
- For the custom archive format, this specifies compression of
+ For the custom and directory archive formats, this specifies compression of
individual table-data segments, and the default is to compress
at a moderate level.
For plain text output, setting a nonzero compression level causes
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 48fa7cb1a3..650b542fce 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -4,7 +4,7 @@
*
* A directory format dump is a directory, which contains a "toc.dat" file
* for the TOC, and a separate file for each data entry, named "<oid>.dat".
- * Large objects (BLOBs) are stored in separate files named "blob_<uid>.dat",
+ * Large objects (BLOBs) are stored in separate files named "blob_<oid>.dat",
* and there's a plain-text TOC file for them called "blobs.toc". If
* compression is used, each data file is individually compressed and the
* ".gz" suffix is added to the filenames. The TOC files are never
--
2.17.0
[text/x-diff] 0002-Fix-malformed-comment.patch (1.2K, ../../20201221194924.GI30237@telsasoft.com/3-0002-Fix-malformed-comment.patch)
download | inline diff:
From 49f400b5b2e559cc2ed6be82e04a4dc4ba5c63b3 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 12 Dec 2020 00:11:37 -0600
Subject: [PATCH 2/7] Fix malformed comment..
broken since bf50caf10.
---
src/bin/pg_dump/pg_backup_archiver.h | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 177360ed6e..6a5a22637b 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -329,9 +329,12 @@ struct _archiveHandle
DumpId *tableDataId; /* TABLE DATA ids, indexed by table dumpId */
struct _tocEntry *currToc; /* Used when dumping data */
- int compression; /* Compression requested on open Possible
- * values for compression: -1
- * Z_DEFAULT_COMPRESSION 0 COMPRESSION_NONE
+ int compression; /*---------
+ * Compression requested on open
+ * Possible values for compression:
+ * -2 ZSTD_COMPRESSION
+ * -1 Z_DEFAULT_COMPRESSION
+ * 0 COMPRESSION_NONE
* 1-9 levels for gzip compression */
bool dosync; /* data requested to be synced on sight */
ArchiveMode mode; /* File mode - r or w */
--
2.17.0
[text/x-diff] 0003-pg_dump-zstd-compression.patch (29.9K, ../../20201221194924.GI30237@telsasoft.com/4-0003-pg_dump-zstd-compression.patch)
download | inline diff:
From a645d33c6bed149abae5a47adb40d24812e8b12c Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 11 Dec 2020 15:01:25 -0600
Subject: [PATCH 3/7] pg_dump: zstd compression
---
configure | 123 ++++++-
configure.ac | 22 ++
src/bin/pg_dump/compress_io.c | 475 ++++++++++++++++++++++++--
src/bin/pg_dump/compress_io.h | 5 +-
src/bin/pg_dump/pg_backup_archiver.h | 5 +
src/bin/pg_dump/pg_backup_directory.c | 6 +-
src/bin/pg_dump/pg_dump.c | 3 +-
src/include/pg_config.h.in | 3 +
8 files changed, 597 insertions(+), 45 deletions(-)
diff --git a/configure b/configure
index 11a4284e5b..240e536e04 100755
--- a/configure
+++ b/configure
@@ -698,6 +698,7 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+with_zstd
with_zlib
with_system_tzdata
with_libxslt
@@ -798,6 +799,7 @@ infodir
docdir
oldincludedir
includedir
+runstatedir
localstatedir
sharedstatedir
sysconfdir
@@ -866,6 +868,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_zstd
with_gnu_ld
enable_largefile
'
@@ -935,6 +938,7 @@ datadir='${datarootdir}'
sysconfdir='${prefix}/etc'
sharedstatedir='${prefix}/com'
localstatedir='${prefix}/var'
+runstatedir='${localstatedir}/run'
includedir='${prefix}/include'
oldincludedir='/usr/include'
docdir='${datarootdir}/doc/${PACKAGE_TARNAME}'
@@ -1187,6 +1191,15 @@ do
| -silent | --silent | --silen | --sile | --sil)
silent=yes ;;
+ -runstatedir | --runstatedir | --runstatedi | --runstated \
+ | --runstate | --runstat | --runsta | --runst | --runs \
+ | --run | --ru | --r)
+ ac_prev=runstatedir ;;
+ -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \
+ | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \
+ | --run=* | --ru=* | --r=*)
+ runstatedir=$ac_optarg ;;
+
-sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb)
ac_prev=sbindir ;;
-sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \
@@ -1324,7 +1337,7 @@ fi
for ac_var in exec_prefix prefix bindir sbindir libexecdir datarootdir \
datadir sysconfdir sharedstatedir localstatedir includedir \
oldincludedir docdir infodir htmldir dvidir pdfdir psdir \
- libdir localedir mandir
+ libdir localedir mandir runstatedir
do
eval ac_val=\$$ac_var
# Remove trailing slashes.
@@ -1477,6 +1490,7 @@ Fine tuning of the installation directories:
--sysconfdir=DIR read-only single-machine data [PREFIX/etc]
--sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com]
--localstatedir=DIR modifiable single-machine data [PREFIX/var]
+ --runstatedir=DIR modifiable per-process data [LOCALSTATEDIR/run]
--libdir=DIR object code libraries [EPREFIX/lib]
--includedir=DIR C header files [PREFIX/include]
--oldincludedir=DIR C header files for non-gcc [/usr/include]
@@ -1570,6 +1584,7 @@ Optional Packages:
--with-system-tzdata=DIR
use system time zone data in DIR
--without-zlib do not use Zlib
+ --without-zstd do not use Zstd
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
Some influential environment variables:
@@ -8601,6 +8616,35 @@ fi
+#
+# Zstd
+#
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+ :
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=yes
+
+fi
+
+
+
+
#
# Assignments
#
@@ -12092,6 +12136,59 @@ fi
fi
+if test "$with_zstd" = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compressStream2 in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compressStream2 in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compressStream2+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compressStream2 ();
+int
+main ()
+{
+return ZSTD_compressStream2 ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_compressStream2=yes
+else
+ ac_cv_lib_zstd_ZSTD_compressStream2=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compressStream2" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compressStream2" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compressStream2" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support." "$LINENO" 5
+fi
+
+fi
+
if test "$enable_spinlocks" = yes; then
$as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h
@@ -13295,6 +13392,20 @@ Use --without-zlib to disable zlib support." "$LINENO" 5
fi
+fi
+
+if test "$with_zstd" = yes; then
+ ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+
+else
+ as_fn_error $? "zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support." "$LINENO" 5
+fi
+
+
fi
if test "$with_gssapi" = yes ; then
@@ -14689,7 +14800,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14735,7 +14846,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14759,7 +14870,7 @@ rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14804,7 +14915,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14828,7 +14939,7 @@ rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
diff --git a/configure.ac b/configure.ac
index fc523c6aeb..7f7222159a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -999,6 +999,13 @@ PGAC_ARG_BOOL(with, zlib, yes,
[do not use Zlib])
AC_SUBST(with_zlib)
+#
+# Zstd
+#
+PGAC_ARG_BOOL(with, zstd, yes,
+ [do not use Zstd])
+AC_SUBST(with_zstd)
+
#
# Assignments
#
@@ -1186,6 +1193,14 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_LIB(zstd, ZSTD_compressStream2, [],
+ [AC_MSG_ERROR([zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support.])])
+fi
+
if test "$enable_spinlocks" = yes; then
AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.])
else
@@ -1400,6 +1415,13 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_HEADER(zstd.h, [], [AC_MSG_ERROR([zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support.])])
+fi
+
if test "$with_gssapi" = yes ; then
AC_CHECK_HEADERS(gssapi/gssapi.h, [],
[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 1417401086..b51ba680a2 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -13,7 +13,7 @@
* friends, providing an interface similar to those, but abstracts away
* the possible compression. Both APIs use libz for the compression, but
* the second API uses gzip headers, so the resulting files can be easily
- * manipulated with the gzip utility.
+ * manipulated with the gzip utility. XXX
*
* Compressor API
* --------------
@@ -41,7 +41,7 @@
* libz's gzopen() APIs. It allows you to use the same functions for
* compressed and uncompressed streams. cfopen_read() first tries to open
* the file with given name, and if it fails, it tries to open the same
- * file with the .gz suffix. cfopen_write() opens a file for writing, an
+ * file with the .gz suffix. cfopen_write() opens a file for writing, an XXX
* extra argument specifies if the file should be compressed, and adds the
* .gz suffix to the filename if so. This allows you to easily handle both
* compressed and uncompressed files.
@@ -72,6 +72,18 @@ struct CompressorState
char *zlibOut;
size_t zlibOutSize;
#endif
+
+#ifdef HAVE_LIBZSTD
+ union {
+ struct {
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
+ ZSTD_CStream *cstream;
+ } zstd;
+ } u;
+#endif
+
};
static void ParseCompressionOption(int compression, CompressionAlgorithm *alg,
@@ -88,6 +100,15 @@ static void WriteDataToArchiveZlib(ArchiveHandle *AH, CompressorState *cs,
static void EndCompressorZlib(ArchiveHandle *AH, CompressorState *cs);
#endif
+#ifdef HAVE_LIBZSTD
+static void InitCompressorZstd(CompressorState *cs, int level);
+static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void DeflateCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const char *data, size_t dLen);
+static void ReadDataFromArchiveZstd(ArchiveHandle *AH, ReadFunc readF);
+#endif
+
/* Routines that support uncompressed data I/O */
static void ReadDataFromArchiveNone(ArchiveHandle *AH, ReadFunc readF);
static void WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
@@ -101,15 +122,25 @@ static void WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
static void
ParseCompressionOption(int compression, CompressionAlgorithm *alg, int *level)
{
- if (compression == Z_DEFAULT_COMPRESSION ||
- (compression > 0 && compression <= 9))
- *alg = COMPR_ALG_LIBZ;
- else if (compression == 0)
- *alg = COMPR_ALG_NONE;
- else
+ switch (compression)
{
- fatal("invalid compression code: %d", compression);
- *alg = COMPR_ALG_NONE; /* keep compiler quiet */
+#ifdef HAVE_LIBZSTD
+ case ZSTD_COMPRESSION:
+ *alg = COMPR_ALG_ZSTD;
+ break;
+#endif
+#ifdef HAVE_ZLIB
+ case Z_DEFAULT_COMPRESSION:
+ case 1..9:
+ *alg = COMPR_ALG_LIBZ;
+ break;
+#endif
+ case 0:
+ *alg = COMPR_ALG_NONE;
+ break;
+ default:
+ fatal("invalid compression code: %d", compression);
+ *alg = COMPR_ALG_NONE; /* keep compiler quiet */
}
/* The level is just the passed-in value. */
@@ -141,10 +172,23 @@ AllocateCompressor(int compression, WriteFunc writeF)
/*
* Perform compression algorithm specific initialization.
*/
+ switch (alg)
+ {
#ifdef HAVE_LIBZ
- if (alg == COMPR_ALG_LIBZ)
+ case COMPR_ALG_LIBZ:
InitCompressorZlib(cs, level);
+ break;
+#endif
+#ifdef HAVE_LIBZSTD
+ case COMPR_ALG_ZSTD:
+ InitCompressorZstd(cs, level);
+ break;
#endif
+ case COMPR_ALG_NONE:
+ /* Do nothing */
+ break;
+ // default:
+ }
return cs;
}
@@ -162,12 +206,20 @@ ReadDataFromArchive(ArchiveHandle *AH, int compression, ReadFunc readF)
if (alg == COMPR_ALG_NONE)
ReadDataFromArchiveNone(AH, readF);
- if (alg == COMPR_ALG_LIBZ)
+ else if (alg == COMPR_ALG_LIBZ)
{
#ifdef HAVE_LIBZ
ReadDataFromArchiveZlib(AH, readF);
#else
fatal("not built with zlib support");
+#endif
+ }
+ else if (alg == COMPR_ALG_ZSTD)
+ {
+#ifdef HAVE_LIBZSTD
+ ReadDataFromArchiveZstd(AH, readF);
+#else
+ fatal("not built with zstd support");
#endif
}
}
@@ -188,6 +240,15 @@ WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs,
fatal("not built with zlib support");
#endif
break;
+
+ case COMPR_ALG_ZSTD:
+#ifdef HAVE_LIBZSTD
+ WriteDataToArchiveZstd(AH, cs, data, dLen);
+#else
+ fatal("not built with zstd support");
+#endif
+ break;
+
case COMPR_ALG_NONE:
WriteDataToArchiveNone(AH, cs, data, dLen);
break;
@@ -204,12 +265,172 @@ EndCompressor(ArchiveHandle *AH, CompressorState *cs)
if (cs->comprAlg == COMPR_ALG_LIBZ)
EndCompressorZlib(AH, cs);
#endif
+#ifdef HAVE_LIBZSTD
+ if (cs->comprAlg == COMPR_ALG_ZSTD)
+ EndCompressorZstd(AH, cs);
+#endif
+
free(cs);
}
/* Private routines, specific to each compression method. */
+// XXX: put in separate files ?
+
+#ifdef HAVE_LIBZSTD
+static void
+InitCompressorZstd(CompressorState *cs, int level)
+{
+ cs->u.zstd.cstream = ZSTD_createCStream();
+ if (cs->u.zstd.cstream == NULL)
+ fatal("could not initialize compression library");
+
+ /* XXX: initialize safely like the corresponding zlib "paranoia" */
+ cs->u.zstd.output.size = ZSTD_CStreamOutSize();
+ cs->u.zstd.output.dst = pg_malloc(cs->u.zstd.output.size);
+ cs->u.zstd.output.pos = 0;
+}
+
+static void
+EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZSTD_outBuffer *output = &cs->u.zstd.output;
+
+ for (;;)
+ {
+ size_t res;
+
+ res = ZSTD_compressStream2(cs->u.zstd.cstream, output,
+ &cs->u.zstd.input, ZSTD_e_end);
+
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ if (res == 0)
+ break;
+
+ if (ZSTD_isError(res))
+ fatal("could not close compression stream: %s",
+ ZSTD_getErrorName(res));
+ }
+
+ // XXX: retval
+ ZSTD_freeCStream(cs->u.zstd.cstream);
+}
+
+static void
+DeflateCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZSTD_inBuffer *input = &cs->u.zstd.input;
+ ZSTD_outBuffer *output = &cs->u.zstd.output;
+
+ while (input->pos != input->size)
+ {
+ size_t res;
+
+ res = ZSTD_compressStream2(cs->u.zstd.cstream, output,
+ input, ZSTD_e_continue);
+
+ if (output->pos == output->size ||
+ input->pos != input->size)
+ {
+ /*
+ * Extra paranoia: avoid zero-length chunks, since a zero length
+ * chunk is the EOF marker in the custom format. This should never
+ * happen but...
+ */
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ output->pos = 0;
+ }
+
+ if (ZSTD_isError(res))
+ fatal("could not compress data: %s", ZSTD_getErrorName(res));
+ }
+}
+
+static void
+WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const char *data, size_t dLen)
+{
+ cs->u.zstd.input.src = (void *) unconstify(char *, data);
+ cs->u.zstd.input.size = dLen;
+ cs->u.zstd.input.pos = 0;
+ DeflateCompressorZstd(AH, cs);
+}
+
+/* Read data from a compressed zstd archive */
+static void
+ReadDataFromArchiveZstd(ArchiveHandle *AH, ReadFunc readF)
+{
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ size_t res;
+ size_t input_size;
+
+ dstream = ZSTD_createDStream();
+ if (dstream == NULL)
+ fatal("could not initialize compression library");
+
+ input_size = ZSTD_DStreamInSize();
+ input.src = pg_malloc(input_size);
+
+ output.size = ZSTD_DStreamOutSize();
+ output.dst = pg_malloc(output.size);
+
+ /* read compressed data */
+ for (;;)
+ {
+ size_t cnt;
+
+ input.size = input_size; // XXX: the buffer can grow, we shouldn't keep resetting it to the original value..
+ cnt = readF(AH, (char **)unconstify(void **, &input.src), &input.size);
+ input.pos = 0;
+ input.size = cnt;
+
+ if (cnt == 0)
+ break;
+
+ while (input.pos < input.size)
+ {
+ /* decompress */
+ output.pos = 0;
+ res = ZSTD_decompressStream(dstream, &output, &input);
+
+ if (ZSTD_isError(res))
+ fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ /* write to output handle */
+ ((char *)output.dst)[output.pos] = '\0';
+ ahwrite(output.dst, 1, output.pos, AH);
+ }
+ }
+
+ /* write any remainder to output handle */
+ /* XXX: is it needed? */
+ /* If `input.pos < input.size`, some input has not been consumed. */
+ /* But if `output.pos == output.size`, there might be some data left within internal buffers., */
+#if 0
+ while (input.pos < input.size || output.pos == output.size)
+ {
+ output.pos = 0;
+ res = ZSTD_decompressStream(dstream, &output, &input);
+ if (ZSTD_isError(res))
+ fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+ ((char *)output.dst)[output.pos] = '\0';
+ ahwrite(output.dst, 1, output.pos, AH);
+ }
+#endif
+
+ pg_free(unconstify(void *, input.src));
+ pg_free(output.dst);
+}
+
+#endif /* HAVE_LIBZSTD */
#ifdef HAVE_LIBZ
+
/*
* Functions for zlib compressed output.
*/
@@ -422,6 +643,19 @@ struct cfp
#ifdef HAVE_LIBZ
gzFile compressedfp;
#endif
+
+#ifdef HAVE_LIBZSTD // XXX: this should be a union with a CompressionAlgorithm alg?
+ /* This is a normal file to which we read/write compressed data */
+ struct {
+ FILE *fp;
+ // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ } zstd;
+#endif
+
};
#ifdef HAVE_LIBZ
@@ -449,24 +683,25 @@ free_keep_errno(void *p)
* On failure, return NULL with an error code in errno.
*/
cfp *
-cfopen_read(const char *path, const char *mode)
+cfopen_read(const char *path, const char *mode, int compression)
{
cfp *fp;
#ifdef HAVE_LIBZ
- if (hasSuffix(path, ".gz"))
- fp = cfopen(path, mode, 1);
+ if (hasSuffix(path, ".gz") || hasSuffix(path, ".zst"))
+ fp = cfopen(path, mode, compression);
else
#endif
{
- fp = cfopen(path, mode, 0);
+ fp = cfopen(path, mode, compression);
#ifdef HAVE_LIBZ
if (fp == NULL)
{
char *fname;
+ char *suffix = compression == ZSTD_COMPRESSION ? "zst" : "gz";
- fname = psprintf("%s.gz", path);
- fp = cfopen(fname, mode, 1);
+ fname = psprintf("%s.%s", path, suffix);
+ fp = cfopen(fname, mode, compression);
free_keep_errno(fname);
}
#endif
@@ -491,13 +726,14 @@ cfopen_write(const char *path, const char *mode, int compression)
cfp *fp;
if (compression == 0)
- fp = cfopen(path, mode, 0);
+ fp = cfopen(path, mode, compression);
else
{
-#ifdef HAVE_LIBZ
+#ifdef HAVE_LIBZ // XXX
char *fname;
+ char *suffix = compression == ZSTD_COMPRESSION ? "zst" : "gz";
- fname = psprintf("%s.gz", path);
+ fname = psprintf("%s.%s", path, suffix);
fp = cfopen(fname, mode, compression);
free_keep_errno(fname);
#else
@@ -505,11 +741,12 @@ cfopen_write(const char *path, const char *mode, int compression)
fp = NULL; /* keep compiler quiet */
#endif
}
+
return fp;
}
/*
- * Opens file 'path' in 'mode'. If 'compression' is non-zero, the file
+ * Opens file 'path' in 'mode'. If 'alg' is COMPR_ALG_ZLIB, the file
* is opened with libz gzopen(), otherwise with plain fopen().
*
* On failure, return NULL with an error code in errno.
@@ -519,9 +756,19 @@ cfopen(const char *path, const char *mode, int compression)
{
cfp *fp = pg_malloc(sizeof(cfp));
- if (compression != 0)
+ fp->uncompressedfp = NULL;
+#ifdef HAVE_LIBZ
+ fp->compressedfp = NULL;
+#endif
+#ifdef HAVE_LIBZSTD
+ fp->zstd.fp = NULL;
+#endif
+
+ switch (compression)
{
#ifdef HAVE_LIBZ
+ case 1 ... 9: // XXX: nonportable
+ case Z_DEFAULT_COMPRESSION:
if (compression != Z_DEFAULT_COMPRESSION)
{
/* user has specified a compression level, so tell zlib to use it */
@@ -537,30 +784,57 @@ cfopen(const char *path, const char *mode, int compression)
fp->compressedfp = gzopen(path, mode);
}
- fp->uncompressedfp = NULL;
if (fp->compressedfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
}
-#else
- fatal("not built with zlib support");
+
+ return fp;
+ break;
#endif
- }
- else
- {
-#ifdef HAVE_LIBZ
- fp->compressedfp = NULL;
+
+#ifdef HAVE_LIBZSTD
+ case ZSTD_COMPRESSION:
+ fp->zstd.fp = fopen(path, mode);
+ // XXX: save the compression params
+ if (fp->zstd.fp == NULL)
+ {
+ free_keep_errno(fp);
+ fp = NULL;
+ }
+ else if (strchr(mode, 'w'))
+ {
+ fp->zstd.dstream = NULL;
+ fp->zstd.output.size = ZSTD_CStreamOutSize(); // XXX
+ fp->zstd.output.dst = pg_malloc0(fp->zstd.output.size);
+ fp->zstd.cstream = ZSTD_createCStream();
+ if (fp->zstd.cstream == NULL)
+ fatal("could not initialize compression library");
+ }
+ else if (strchr(mode, 'r'))
+ {
+ fp->zstd.cstream = NULL;
+ fp->zstd.input.size = ZSTD_DStreamOutSize(); // XXX
+ fp->zstd.input.src = pg_malloc0(fp->zstd.input.size);
+ fp->zstd.dstream = ZSTD_createDStream();
+ if (fp->zstd.dstream == NULL)
+ fatal("could not initialize compression library");
+ } // XXX else: bad mode
+ return fp;
+ break;
#endif
+
+ default:
fp->uncompressedfp = fopen(path, mode);
if (fp->uncompressedfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
}
- }
- return fp;
+ return fp;
+ }
}
@@ -587,6 +861,44 @@ cfread(void *ptr, int size, cfp *fp)
}
else
#endif
+
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ ZSTD_outBuffer *output = &fp->zstd.output;
+ ZSTD_inBuffer *input = &fp->zstd.input;
+ size_t input_size = ZSTD_DStreamInSize();
+ size_t res, cnt;
+
+ output->size = size;
+ output->dst = ptr;
+ output->pos = 0;
+
+ /* read compressed data */
+ while ((cnt = fread(unconstify(void *, input->src), 1, input_size, fp->zstd.fp)))
+ {
+ input->size = cnt;
+ input->pos = 0;
+
+ for ( ; input->pos < input->size; )
+ {
+ /* decompress */
+ res = ZSTD_decompressStream(fp->zstd.dstream, output, input);
+ if (res == 0 || output->pos == output->size)
+ break;
+ if (ZSTD_isError(res))
+ fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+ }
+
+ if (output->pos == output->size)
+ break; /* We read all the data that fits */
+ }
+
+ ret = output->pos;
+ }
+ else
+#endif
+
{
ret = fread(ptr, 1, size, fp->uncompressedfp);
if (ret != size && !feof(fp->uncompressedfp))
@@ -603,6 +915,35 @@ cfwrite(const void *ptr, int size, cfp *fp)
return gzwrite(fp->compressedfp, ptr, size);
else
#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ size_t res, cnt;
+ ZSTD_outBuffer *output = &fp->zstd.output;
+ ZSTD_inBuffer *input = &fp->zstd.input;
+
+ input->src = ptr;
+ input->size = size;
+ input->pos = 0;
+
+ /* Consume all input, and flush later */
+ while (input->pos != input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_continue);
+ if (ZSTD_isError(res))
+ fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ if (cnt != output->pos)
+ fatal("could not write data: %s", strerror(errno));
+ }
+
+ return size;
+ }
+ else
+#endif
+
return fwrite(ptr, 1, size, fp->uncompressedfp);
}
@@ -625,6 +966,20 @@ cfgetc(cfp *fp)
}
else
#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ if (cfread(&ret, 1, fp) != 1)
+ {
+ if (feof(fp->zstd.fp))
+ fatal("could not read from input file: end of file");
+ else
+ fatal("could not read from input file: %s", strerror(errno));
+ }
+fprintf(stderr, "cfgetc %d\n", ret);
+ }
+#endif
+
{
ret = fgetc(fp->uncompressedfp);
if (ret == EOF)
@@ -641,6 +996,18 @@ cfgets(cfp *fp, char *buf, int len)
if (fp->compressedfp)
return gzgets(fp->compressedfp, buf, len);
else
+#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ int res;
+ res = cfread(buf, len, fp);
+ buf[res] = 0;
+ if (strchr(buf, '\n'))
+ *strchr(buf, '\n') = '\0';
+ return buf;
+ }
+ else
#endif
return fgets(buf, len, fp->uncompressedfp);
}
@@ -662,6 +1029,43 @@ cfclose(cfp *fp)
fp->compressedfp = NULL;
}
else
+#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ ZSTD_outBuffer *output = &fp->zstd.output;
+ ZSTD_inBuffer *input = &fp->zstd.input;
+ size_t res, cnt;
+
+ if (fp->zstd.cstream)
+ {
+ for (;;)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_end);
+ if (ZSTD_isError(res))
+ fatal("could not compress data: %s", ZSTD_getErrorName(res));
+ cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ if (cnt != output->pos)
+ fatal("could not write data: %s", strerror(errno));
+ if (res == 0)
+ break;
+ }
+
+ ZSTD_freeCStream(fp->zstd.cstream);
+ pg_free(fp->zstd.output.dst);
+ }
+
+ if (fp->zstd.dstream)
+ {
+ ZSTD_freeDStream(fp->zstd.dstream);
+ pg_free(unconstify(void *, fp->zstd.input.src));
+ }
+
+ result = fclose(fp->zstd.fp);
+ fp->zstd.fp = NULL;
+ }
+ else
#endif
{
result = fclose(fp->uncompressedfp);
@@ -679,6 +1083,11 @@ cfeof(cfp *fp)
if (fp->compressedfp)
return gzeof(fp->compressedfp);
else
+#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ return feof(fp->zstd.fp);
+ else
#endif
return feof(fp->uncompressedfp);
}
diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h
index d2e6e1b854..f0ce06f176 100644
--- a/src/bin/pg_dump/compress_io.h
+++ b/src/bin/pg_dump/compress_io.h
@@ -24,7 +24,8 @@
typedef enum
{
COMPR_ALG_NONE,
- COMPR_ALG_LIBZ
+ COMPR_ALG_LIBZ,
+ COMPR_ALG_ZSTD,
} CompressionAlgorithm;
/* Prototype for callback function to WriteDataToArchive() */
@@ -57,7 +58,7 @@ extern void EndCompressor(ArchiveHandle *AH, CompressorState *cs);
typedef struct cfp cfp;
extern cfp *cfopen(const char *path, const char *mode, int compression);
-extern cfp *cfopen_read(const char *path, const char *mode);
+extern cfp *cfopen_read(const char *path, const char *mode, int compression);
extern cfp *cfopen_write(const char *path, const char *mode, int compression);
extern int cfread(void *ptr, int size, cfp *fp);
extern int cfwrite(const void *ptr, int size, cfp *fp);
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 6a5a22637b..1a229ebedb 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -60,6 +60,11 @@ typedef struct _z_stream
typedef z_stream *z_streamp;
#endif
+#ifdef HAVE_LIBZSTD
+#include <zstd.h>
+#define ZSTD_COMPRESSION -2
+#endif /* HAVE_LIBZSTD */
+
/* Data block types */
#define BLK_DATA 1
#define BLK_BLOBS 3
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 650b542fce..6e55cb425e 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -202,7 +202,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH)
setFilePath(AH, fname, "toc.dat");
- tocFH = cfopen_read(fname, PG_BINARY_R);
+ tocFH = cfopen_read(fname, PG_BINARY_R, AH->compression);
if (tocFH == NULL)
fatal("could not open input file \"%s\": %m", fname);
@@ -388,7 +388,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename)
if (!filename)
return;
- cfp = cfopen_read(filename, PG_BINARY_R);
+ cfp = cfopen_read(filename, PG_BINARY_R, AH->compression);
if (!cfp)
fatal("could not open input file \"%s\": %m", filename);
@@ -440,7 +440,7 @@ _LoadBlobs(ArchiveHandle *AH)
setFilePath(AH, fname, "blobs.toc");
- ctx->blobsTocFH = cfopen_read(fname, PG_BINARY_R);
+ ctx->blobsTocFH = cfopen_read(fname, PG_BINARY_R, AH->compression);
if (ctx->blobsTocFH == NULL)
fatal("could not open large object TOC file \"%s\" for input: %m",
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 8b1e5cc2b5..f21eb021c7 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -533,7 +533,8 @@ main(int argc, char **argv)
case 'Z': /* Compression Level */
compressLevel = atoi(optarg);
- if (compressLevel < 0 || compressLevel > 9)
+ if ((compressLevel < 0 || compressLevel > 9) &&
+ compressLevel != -2)
{
pg_log_error("compression level must be in range 0..9");
exit_nicely(1);
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index de8f838e53..da35415c72 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -346,6 +346,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if you have the `link' function. */
#undef HAVE_LINK
--
2.17.0
[text/x-diff] 0004-cmdline-parser-for-compression-alg-level-opts.patch (242.3K, ../../20201221194924.GI30237@telsasoft.com/5-0004-cmdline-parser-for-compression-alg-level-opts.patch)
download | inline diff:
From 81e88ef3a63d1304e14134be14f41c2865c61226 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 12 Dec 2020 00:40:39 -0600
Subject: [PATCH 4/7] cmdline parser for compression alg/level/opts
---
src/bin/pg_dump/compress_io.c | 153 +++++++++++---------------
src/bin/pg_dump/compress_io.h | 17 +--
src/bin/pg_dump/core | Bin 0 -> 1257472 bytes
src/bin/pg_dump/kwlookup.c | 1 +
src/bin/pg_dump/pg_backup.h | 19 +++-
src/bin/pg_dump/pg_backup_archiver.c | 46 ++++----
src/bin/pg_dump/pg_backup_archiver.h | 11 +-
src/bin/pg_dump/pg_backup_custom.c | 6 +-
src/bin/pg_dump/pg_backup_directory.c | 19 ++--
src/bin/pg_dump/pg_backup_tar.c | 48 ++++----
src/bin/pg_dump/pg_dump.c | 144 ++++++++++++++++++++----
11 files changed, 278 insertions(+), 186 deletions(-)
create mode 100644 src/bin/pg_dump/core
create mode 120000 src/bin/pg_dump/kwlookup.c
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index b51ba680a2..10db22ff88 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -86,12 +86,9 @@ struct CompressorState
};
-static void ParseCompressionOption(int compression, CompressionAlgorithm *alg,
- int *level);
-
/* Routines that support zlib compressed data I/O */
#ifdef HAVE_LIBZ
-static void InitCompressorZlib(CompressorState *cs, int level);
+static void InitCompressorZlib(CompressorState *cs, Compress *compress);
static void DeflateCompressorZlib(ArchiveHandle *AH, CompressorState *cs,
bool flush);
static void ReadDataFromArchiveZlib(ArchiveHandle *AH, ReadFunc readF);
@@ -101,7 +98,7 @@ static void EndCompressorZlib(ArchiveHandle *AH, CompressorState *cs);
#endif
#ifdef HAVE_LIBZSTD
-static void InitCompressorZstd(CompressorState *cs, int level);
+static void InitCompressorZstd(CompressorState *cs, Compress *compress);
static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
static void DeflateCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
@@ -114,80 +111,40 @@ static void ReadDataFromArchiveNone(ArchiveHandle *AH, ReadFunc readF);
static void WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
const char *data, size_t dLen);
-/*
- * Interprets a numeric 'compression' value. The algorithm implied by the
- * value (zlib or none at the moment), is returned in *alg, and the
- * zlib compression level in *level.
- */
-static void
-ParseCompressionOption(int compression, CompressionAlgorithm *alg, int *level)
-{
- switch (compression)
- {
-#ifdef HAVE_LIBZSTD
- case ZSTD_COMPRESSION:
- *alg = COMPR_ALG_ZSTD;
- break;
-#endif
-#ifdef HAVE_ZLIB
- case Z_DEFAULT_COMPRESSION:
- case 1..9:
- *alg = COMPR_ALG_LIBZ;
- break;
-#endif
- case 0:
- *alg = COMPR_ALG_NONE;
- break;
- default:
- fatal("invalid compression code: %d", compression);
- *alg = COMPR_ALG_NONE; /* keep compiler quiet */
- }
-
- /* The level is just the passed-in value. */
- if (level)
- *level = compression;
-}
-
/* Public interface routines */
/* Allocate a new compressor */
CompressorState *
-AllocateCompressor(int compression, WriteFunc writeF)
+AllocateCompressor(Compress *compression, WriteFunc writeF)
{
CompressorState *cs;
- CompressionAlgorithm alg;
- int level;
-
- ParseCompressionOption(compression, &alg, &level);
-
-#ifndef HAVE_LIBZ
- if (alg == COMPR_ALG_LIBZ)
- fatal("not built with zlib support");
-#endif
cs = (CompressorState *) pg_malloc0(sizeof(CompressorState));
cs->writeF = writeF;
- cs->comprAlg = alg;
+ cs->comprAlg = compression->alg;
/*
* Perform compression algorithm specific initialization.
*/
- switch (alg)
+ Assert (compression->alg != COMPR_ALG_DEFAULT);
+ switch (compression->alg)
{
#ifdef HAVE_LIBZ
case COMPR_ALG_LIBZ:
- InitCompressorZlib(cs, level);
+ InitCompressorZlib(cs, compression);
break;
#endif
#ifdef HAVE_LIBZSTD
case COMPR_ALG_ZSTD:
- InitCompressorZstd(cs, level);
+ InitCompressorZstd(cs, compression);
break;
#endif
case COMPR_ALG_NONE:
/* Do nothing */
break;
- // default:
+ default:
+ /* Should not happen */
+ fatal("requested compression not available in this installation");
}
return cs;
@@ -198,12 +155,9 @@ AllocateCompressor(int compression, WriteFunc writeF)
* out with ahwrite().
*/
void
-ReadDataFromArchive(ArchiveHandle *AH, int compression, ReadFunc readF)
+ReadDataFromArchive(ArchiveHandle *AH, ReadFunc readF)
{
- CompressionAlgorithm alg;
-
- ParseCompressionOption(compression, &alg, NULL);
-
+ CompressionAlgorithm alg = AH->compression.alg;
if (alg == COMPR_ALG_NONE)
ReadDataFromArchiveNone(AH, readF);
else if (alg == COMPR_ALG_LIBZ)
@@ -233,25 +187,25 @@ WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs,
{
switch (cs->comprAlg)
{
- case COMPR_ALG_LIBZ:
#ifdef HAVE_LIBZ
+ case COMPR_ALG_LIBZ:
WriteDataToArchiveZlib(AH, cs, data, dLen);
-#else
- fatal("not built with zlib support");
-#endif
break;
+#endif
- case COMPR_ALG_ZSTD:
#ifdef HAVE_LIBZSTD
+ case COMPR_ALG_ZSTD:
WriteDataToArchiveZstd(AH, cs, data, dLen);
-#else
- fatal("not built with zstd support");
-#endif
break;
+#endif
case COMPR_ALG_NONE:
WriteDataToArchiveNone(AH, cs, data, dLen);
break;
+
+ default:
+ /* Should not happen */
+ fatal("requested compression not available in this installation");
}
}
@@ -278,9 +232,30 @@ EndCompressor(ArchiveHandle *AH, CompressorState *cs)
#ifdef HAVE_LIBZSTD
static void
-InitCompressorZstd(CompressorState *cs, int level)
+InitCompressorZstd(CompressorState *cs, Compress *compress)
{
+ size_t res;
+
cs->u.zstd.cstream = ZSTD_createCStream();
+
+ if (compress->level != 0) // XXX
+ {
+ res = ZSTD_CCtx_setParameter(cs->u.zstd.cstream,
+ ZSTD_c_compressionLevel, compress->level);
+ if (ZSTD_isError(res))
+ fatal("could not set compression level: %s",
+ ZSTD_getErrorName(res));
+ }
+
+ if (compress->zstdlong)
+ {
+ res = ZSTD_CCtx_setParameter(cs->u.zstd.cstream,
+ ZSTD_c_enableLongDistanceMatching, 1); // XXX: allow to disable it
+ if (ZSTD_isError(res))
+ fatal("could not set compression parameter: %s",
+ ZSTD_getErrorName(res));
+ }
+
if (cs->u.zstd.cstream == NULL)
fatal("could not initialize compression library");
@@ -436,7 +411,7 @@ ReadDataFromArchiveZstd(ArchiveHandle *AH, ReadFunc readF)
*/
static void
-InitCompressorZlib(CompressorState *cs, int level)
+InitCompressorZlib(CompressorState *cs, Compress *compress)
{
z_streamp zp;
@@ -453,7 +428,7 @@ InitCompressorZlib(CompressorState *cs, int level)
cs->zlibOut = (char *) pg_malloc(ZLIB_OUT_SIZE + 1);
cs->zlibOutSize = ZLIB_OUT_SIZE;
- if (deflateInit(zp, level) != Z_OK)
+ if (deflateInit(zp, compress->level) != Z_OK)
fatal("could not initialize compression library: %s",
zp->msg);
@@ -683,7 +658,7 @@ free_keep_errno(void *p)
* On failure, return NULL with an error code in errno.
*/
cfp *
-cfopen_read(const char *path, const char *mode, int compression)
+cfopen_read(const char *path, const char *mode, Compress *compression)
{
cfp *fp;
@@ -698,7 +673,7 @@ cfopen_read(const char *path, const char *mode, int compression)
if (fp == NULL)
{
char *fname;
- char *suffix = compression == ZSTD_COMPRESSION ? "zst" : "gz";
+ char *suffix = compression->alg == COMPR_ALG_ZSTD ? "zst" : "gz";
fname = psprintf("%s.%s", path, suffix);
fp = cfopen(fname, mode, compression);
@@ -714,24 +689,23 @@ cfopen_read(const char *path, const char *mode, int compression)
* be a filemode as accepted by fopen() and gzopen() that indicates writing
* ("w", "wb", "a", or "ab").
*
- * If 'compression' is non-zero, a gzip compressed stream is opened, and
- * 'compression' indicates the compression level used. The ".gz" suffix
- * is automatically added to 'path' in that case.
+ * Use compression if specified.
+ * The appropriate suffix is automatically added to 'path' in that case.
*
* On failure, return NULL with an error code in errno.
*/
cfp *
-cfopen_write(const char *path, const char *mode, int compression)
+cfopen_write(const char *path, const char *mode, Compress *compression)
{
cfp *fp;
- if (compression == 0)
+ if (compression->alg == COMPR_ALG_NONE)
fp = cfopen(path, mode, compression);
else
{
#ifdef HAVE_LIBZ // XXX
char *fname;
- char *suffix = compression == ZSTD_COMPRESSION ? "zst" : "gz";
+ char *suffix = compression->alg == COMPR_ALG_ZSTD ? "zst" : "gz";
fname = psprintf("%s.%s", path, suffix);
fp = cfopen(fname, mode, compression);
@@ -752,7 +726,7 @@ cfopen_write(const char *path, const char *mode, int compression)
* On failure, return NULL with an error code in errno.
*/
cfp *
-cfopen(const char *path, const char *mode, int compression)
+cfopen(const char *path, const char *mode, Compress *compression)
{
cfp *fp = pg_malloc(sizeof(cfp));
@@ -764,18 +738,17 @@ cfopen(const char *path, const char *mode, int compression)
fp->zstd.fp = NULL;
#endif
- switch (compression)
+ switch (compression->alg)
{
#ifdef HAVE_LIBZ
- case 1 ... 9: // XXX: nonportable
- case Z_DEFAULT_COMPRESSION:
- if (compression != Z_DEFAULT_COMPRESSION)
+ case COMPR_ALG_LIBZ:
+ if (compression->level != Z_DEFAULT_COMPRESSION)
{
/* user has specified a compression level, so tell zlib to use it */
char mode_compression[32];
snprintf(mode_compression, sizeof(mode_compression), "%s%d",
- mode, compression);
+ mode, compression->level);
fp->compressedfp = gzopen(path, mode_compression);
}
else
@@ -795,7 +768,7 @@ cfopen(const char *path, const char *mode, int compression)
#endif
#ifdef HAVE_LIBZSTD
- case ZSTD_COMPRESSION:
+ case COMPR_ALG_ZSTD:
fp->zstd.fp = fopen(path, mode);
// XXX: save the compression params
if (fp->zstd.fp == NULL)
@@ -825,15 +798,19 @@ cfopen(const char *path, const char *mode, int compression)
break;
#endif
- default:
+ case COMPR_ALG_NONE:
fp->uncompressedfp = fopen(path, mode);
if (fp->uncompressedfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
}
-
return fp;
+ break;
+
+ default:
+ /* should not happen */
+ fatal("requested compression not available in this installation");
}
}
@@ -1005,7 +982,7 @@ cfgets(cfp *fp, char *buf, int len)
buf[res] = 0;
if (strchr(buf, '\n'))
*strchr(buf, '\n') = '\0';
- return buf;
+ return res > 0 ? buf : 0;
}
else
#endif
diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h
index f0ce06f176..ae37be08d9 100644
--- a/src/bin/pg_dump/compress_io.h
+++ b/src/bin/pg_dump/compress_io.h
@@ -21,13 +21,6 @@
#define ZLIB_OUT_SIZE 4096
#define ZLIB_IN_SIZE 4096
-typedef enum
-{
- COMPR_ALG_NONE,
- COMPR_ALG_LIBZ,
- COMPR_ALG_ZSTD,
-} CompressionAlgorithm;
-
/* Prototype for callback function to WriteDataToArchive() */
typedef void (*WriteFunc) (ArchiveHandle *AH, const char *buf, size_t len);
@@ -47,8 +40,8 @@ typedef size_t (*ReadFunc) (ArchiveHandle *AH, char **buf, size_t *buflen);
/* struct definition appears in compress_io.c */
typedef struct CompressorState CompressorState;
-extern CompressorState *AllocateCompressor(int compression, WriteFunc writeF);
-extern void ReadDataFromArchive(ArchiveHandle *AH, int compression,
+extern CompressorState *AllocateCompressor(Compress *compression, WriteFunc writeF);
+extern void ReadDataFromArchive(ArchiveHandle *AH,
ReadFunc readF);
extern void WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs,
const void *data, size_t dLen);
@@ -57,9 +50,9 @@ extern void EndCompressor(ArchiveHandle *AH, CompressorState *cs);
typedef struct cfp cfp;
-extern cfp *cfopen(const char *path, const char *mode, int compression);
-extern cfp *cfopen_read(const char *path, const char *mode, int compression);
-extern cfp *cfopen_write(const char *path, const char *mode, int compression);
+extern cfp *cfopen(const char *path, const char *mode, Compress *compression);
+extern cfp *cfopen_read(const char *path, const char *mode, Compress *compression);
+extern cfp *cfopen_write(const char *path, const char *mode, Compress *compression);
extern int cfread(void *ptr, int size, cfp *fp);
extern int cfwrite(const void *ptr, int size, cfp *fp);
extern int cfgetc(cfp *fp);
diff --git a/src/bin/pg_dump/core b/src/bin/pg_dump/core
new file mode 100644
index 0000000000000000000000000000000000000000..bdb556ec717e50744ccc2a7256ba9f90cb41427c
GIT binary patch
literal 1257472
zcmeEv34mKw)&J`PRF;|oYEi(~g3Js|r<t|WN-?FK(qcOeU8DtclFnonnB_6al$J#T
z0u}@fg4j<$Bg$u8&?tzNMT39Tx?xb1x(7G<iA!)r5&A#(p5MuG^Im3(EmF;c&M)Vl
zdzO3d^6q`heV2EvSUP#qBukE|)@fFOvaEnGiZEccTk}9q1ui?hC_~ZD-yU5=6#Y~-
zw5%VAxq}w}WL;4PAASdcpq1P~Dk-4;&=EMeRE|?ED?NpO70GFP0m*QvyIg2;Wx3Gd
zrE)g(2^QHqSg}It4;^ts4*4ttRr$1>qot^PCzjfCj$nphkv$!kEcm{!KmC=<p+kY#
zNg=IBPN(D`mp0^(-O9a|e-+8;c#2%=H}3YtZjqcWpEaHJrT0?K2f2(PCw_+TqsX3)
zi^%z(a`)3gt1pt%nrzrh{<c(3?0LmS<PwMf(X}4D;^G&3dnp%yT*{CWKW+RdDj)YF
ziKiha{-w(0bV?9%@gqvhhwPs1MdX}0rE+4=z<hrQiM@%->GC;{OL)k|J<1n{oUftO
zPbr^e^Dp?T{M7v+FV|TrXOv6&W4?R|*s~`tr|l&n=QrfU&+MEsKXqJ0&UdN1pU&}R
zaylUed-3;`%85O%xa8&TE0qgCP_QVTdYs6?UfPheAtz|c`CiT$vW8p$a)L#2Ixa!T
z+4q<FDfO3DHAY8~oTfu!?}1V|&7?nG<Alg%eq1Ui_6*Fo7f0+(TuzrS4!N9%T+*X_
z3CJaWTI%O`<rBI1^QCe|xr*bd>pdy<_EIhdxuhW%fTMy%^{3;KgIv~-6F;5uc>$lt
zkRf@@pGwAg8}<Z??CE}MBf&YgR8H*0Azvh?{q!RtXUHMDWtID>?d9c;FSR#eIlENO
z4?)2qKXqI}NU&#>${|_mTd>fsk-=}s*{~;AWKWkbfrOwTH=eyDGKBV0E(JMfFXhsZ
z^FRNJ=ZfcNHtJ0f*;HI~`LaiH00M`;vQ*AHF1o!~M{$PGUdq{!vqF2+0W0}&PwGuj
z*$R@g%HPGSR(GiUhp9g}ihoPmdxk9Q6#Do2^DXPVQzf+=1DJZ2b4pXvoppMy96X!n
z%%P1ZY5Et)miofUntoCT)UslcXsjz1-sr;|JN5LtOUEe434bus)q6o#qm??OUh5J4
zIzaU2=VwgQ>CKRvJXuA9rtl|Ny3S5Pdi>+YSb=meU@g4v*-Dwz%0^;rt%lUW0sUh)
zG1WgawoYGH&)ATI%$7B`e`F{!*BRY%@y0FZ&y9|D&vizkvA)qrbZ%_O=^7Z0#=?Vx
zbH}2ia|cJd!-I33zUYO6b2knQ(=V;co-=27WKML;a5w$ZYqB_LUw+2&HR7Apx%i(q
z{v!QQZPHK1)p#U6%@W(_59nsro@d|jD#hY4oqm8Z1NnP^D)#CKV7-c_L@Q!GxX%^Z
z{vk-Wa9)`vH1K7><S?&eC1$D))gjAjp|MKSb*%W99p!0V2*tp03~);ww?XMjpIyZ?
z&U)6h;ESA~D$gzCRL6?tOC^RU3sZt8Yj<-ko~L=OEwQmIti7?8UL^WtZH={ZucXV`
zIl*h_eo2=#uDNCv8f#LCsU>SvtT_)AJ+cPHnlhhX){5rZPiU;|@-dgSUXs^RuN3{D
zEv(J*`7>B^nQJJau|~@KBWtEOuVwQ7gSN27$>*0fkGX~s+Q(~#e47MqVGWW`mo*92
z7}91fS#y-FAui={)LbJ79p^dulQJ-XwlK%fr-Qb#oc}4wFLU$k1Eu4&(3ndfEAf}P
zb(ZJ6uaR`n7UshF{4zH-=ej~;F8gZH1KLXP+)?_R&{mSu^Z79r&<V42ZX|O-b8aVe
zj^}RqdI4?acy3lJ{(!bH_c}w;WiE!f)tQnmb0*AzY9w9eM401<{(t@4u@6o^|3NAS
z2R6>#+}_sJ)-q>sV0divoW9|)xs*5@jdt}83`U}Jni`tsHAF`&mHJZ6Iyez8U*t^D
zuHk_RM(^L4v3q353CAW9zG&Brl~+&vsIlqEUj#5GJlfqquqoo*4!g%M#fz5ZV(Kvs
zjSWo=jg8(8zsQ-KSpR4w+%tZZ{%skHe0yO_1T{p_Kj%Uf{?>-ZmKQow|92^JAK8m)
zW4sKwFGiOU)RhF8t6we)7CEN_<FpXivj^844Bi1u?x`Kwc>X|-z0mF+8SWj83@-1q
z)p|AJM236pRhl49G&UL@7>?OzuCgEFxuxI%Ty|I2)p|BSZD2B0^@`=4>(&!pZ4>`l
zZ6be-!L<dHC;rFv`tl-HQ>qcDMxYvjY6Pkgs79a~focS*5vWF>8i8sAsu8G0peO>D
z8_)kw!Snygw{oTc@=kkC{;sB_p8r3L{=UvSQ9m!Co`H~jCidfEgP`2IRF@@{+;;Fy
zlzXv)a_d*2Jaa)m%Z~IiP7;*oIFy`z{vZeu1Ali9UnmtUmeclpuxjEO6hIKqL@0ae
z+FNRnpD0OPT8cAq6MB~oe;}u{shpwBoIkX+6#XzKDDhO@seccHZ{iZ-sY6iZRY&kw
zu^#pNmrwB1875u@zXV0@ERti-w3>1MK#jE;_PpW}#A9cIQt#@PLR-t+!5-WqrTe9G
z+FH~th9+HZ^_q#Q!81D7S<gN)<KmzG?x06s{_ew8f9TPXhp)cpt;>|($_J%S<xz*q
zk8tVVA0=Aq?WTi1IP<DW$4*vhpQL{&`d540^~YNCK6S;zzy9&B<J<qz@Mz?bq26_2
z-~2zm<JiyaXjswo@ZXL+IPmFn?!M!1cvz2wjN@Snnj5phBmIZukXwE~704}pJzWfR
zOTX1a|1_$0xBMTc;JBq<?_pOxr(_#tRpikg`nx^!%<%B@Xpj8=@JLt9+0FkmJ>t{l
zk^fa5@qfT0K6M`cT;fr#r5<|rc%)z8;pcPIkX_5`VfO?N{kDhxK94xGdzAMt9(rba
z__Ng`|2I6!b+Cv3zwz+r3=cgIdDy+*BmD;+`qy}r>wb^8je7WVr-z<TdD#7;M|!V^
zo?AWgw|SIzu}6B=BmGAn_3|N){Oa`P)@~CX@%gofo*@rC(>=;N=AozF!~c2@e?IMz
z{vi*$^&WP=;gSD`9{K-7W2;--T0Q(}_Zass@z8&TNBTEC(m(8B_gx<O8$JAar$@Pt
z^@z_A9{sD+qaL5}NPn+~pS>RG>RC9O{+Y*N9_{c;l+mre)_dqb%On5qJ@RKf%2nf$
z|1^(u+e1&6N4Y-lQSa(Lvs-!p<WY|^Jlezg9{Cd<{v7P#&j&pGnd+hEJP*6I9)AA9
zBW|r8`aj{3ews&nKEuPGOFir!?2-RUkMvC*cGaTEtsd|8@PE`p|1}=^pYrhkSr7dS
zJp4Jvqh9Xx(7)TmpMZz|CwjE2`#k*ogGc@!d9=e%dz5R;L(eRa@}A|9{%Q~X)02~?
zoIUB#>FtZA`xi~GJ$<@;`t+HnPxqZZecDp>9C?WTT~F`m)L#AEp1ah`0>pdQ<c{ex
z-#LZy-Y~UedhPWGEt&4W=HNxs+uwG`8Pn5;Pkz!>UB_Wlj-5Jv=iy7HXAVDo`Y!sH
zJ^UQXZdo%)ZH{cjZhmghZS>n(Ha$4`nnR~gJN+=tJABx(o*|xEgIRB!o}D!L1H@c(
zn3Ag{-u3T!ZqN0Kw`_V~@(8h(v)mHmS+_j5=d+5Z<vy(CY$8L%`!Ml7sdydJeb-Dr
zV|r%V<jW47?pvgau5@2TdHnA!msjh^B=H_2o;>5P^4~*gSxWmV{bqajYI^}9GsL@j
z+jD!SQ+--Xr~BSHnc{qdisAKBDel)CM3I{`WgXQ-?V_SOpa}30AUJx2{C?kadrqV_
zt#n*JdCByd*GyS5-G1BDOAehrbCIsLMTae--^=j(+@f^l_j=N?;`ZnEyn=qKL!}K;
z+ImXctW!8GPH7KQ+E`KA&6MVU|8sj<kftuRY^SvIDDCs`32mdJv~4U=ZR5=8&Xh;o
zo7$qo)cXC;q$~5m=l0~^FQBv~^zU%TvUXB>gno1Tqu9N3inejOYJbaA`&*`LtehSg
zb!*GIoNGx>j`XN1s2<e_R3lK0Ks5r@2vj3bjX*U52Sx<qBmACp04QGHe8*w>-TPaC
zcK|;Mlzb_Jz8CaEz;6N{2mT!RJK*1e2Yra`%>au2<3OJXJjLKQf^G-C0k{-+7Vtb^
zA8-`71sDgu9r#Y*dw}AP;B82!<*Gg;4s!qSPVi;!{Rz<Wj`7`~Wj%5qXt^sa{>wYA
z4}mZ5+Ddukey)^D-f#aV^2__}-vvD#{{9g3%RxT@TCHEyu^Y6!ub%@g>obY_Y|d^;
zoQ?x6@s)S8rGC`9P90KD*cHi=dXV?&r9Nz(+p?ry<l2d}i&@}HeaS8x(q3e{JZU$w
zPJ0IBk+qAobGnS74{85u8Kw?t|8zM+AJPx#GJrlL{|TU<hyD{mPx~;p@7IAo0`y6s
zXMnB&{R+_Px~Mvi16{|YbpmLRd1x}yPX#UIS`1pseI{rr=UJeooa;eLIU}H@-JcJd
zE|cg(+P(AxY4_3(q}@wDkajQqK-#UeV`;b2j-}m7JC=4U?RYc%k#;QYR@(99;7dDt
zJ7{S)?*c9D<`&S>Zf*xH{oq5OrJa8qw6ycPK})~>0%+;iUjr@u`Z3VbuYU$w`qh)5
zrQiGkwDg<5fwoZ}dq7J+Ipj`mhthA404?oS+Nrc#X{XX|rJYK<m3AuaR@$kw+asZ0
z+U+dR67Sc6mU!2J7W)mLrM<R*miG7t(9*s;KuddmBWP*QXMq-fH-MJ%y&1H$-(Jwt
z{v6QK{w@YB?e8kk(*E8FTH4<>(9$112wM8X$3RPexEr*zp9ero`}s0xX+MvEmiF^F
zXz5Q`&{99Y1}*LRDT96*w6vGMftL6@4_eB1@c(lAlJXq^TFQ4cXer-oK#PB~K})>r
zKuf&+pe5e(Kuf$&11;@&DQFp=))@TrKuh}^F!;lu(VeV|Kui0*)S#~hE&bzK&@x`$
zXwbJ9^0$GO@gxaa`u}!=p8_r8%?{8qKBPg*_^=bSj1L*mG9K;%E#pNNw3K%@Xz8zi
z04?L=UqMU!p93xZYwAb1eMo!`11<giNYGN=<3US2P6jRIZ38XkJsq@^w-dCK_Z-kt
z-d@mB-cis}-nW95@?HyC%6l_tDenhBOL;#ATFU!b&{AG$SAy{!9G~nxjOjaGrsv^b
zq5)3zF&XDfOklv1^sgB7Y7Br9*N>t@NWZg!?1A*VUruEK8F%D)I~ixB|I0YE1JC2h
zxFh_(Jili|X`c2c-j7(u?e?j>?vQ@59()-mz5xGZoRIpKxCbueed16|D{(RJWhCbb
z*#=VTP5QsIqfi~&mvn17)8cms1(k73>`Ob7e9!LT@=@G%((|B02sCYDrpqsW<v{=S
zFt+!1(65$?<7(Jvf8}{cQ(<c6<$AB5Y+GDo@b=^M-Z}9W-UBFj?bSLx{<>>5reDeV
zvi_fH+84Q5W9H4*X$*etCXI>rU&ZN<@P41E!*0-Y`WQ}6U&EL^h0{ZK!|tnDuXQ%t
zk557VpD;gr7N^^@-mdlKmLR>J^*EDRKHJa!#6Q6N+ya#6k8jg*_RE-WpTv3+%Xtr=
z#8S?0O=Eqz4#>Ze^XFd8>GAX4rS<q$Uav85Cd;SpgZyiuw-0*G<^1-$xjg<8nC~3J
z{LE2Y54lOq55_re&Q8wf8({nCB<oMtaC-1HEFU<X%a^!;<$Y5*4yoDj{}b>F{#$o*
zy`>Igz1g?3Twn^zC5Ab_{Vd1JpW=3qp2zi{yp{REhnXL`i}(1;A%6ZO`|tZG=kp=H
zz7_1Je;Vt_T+Mb<hawJ;_l22n-3dF7V-LYUvEKM3=BI{u|HWXM^QBOZ%p18rf}iH}
z?B%RKbO>V><<9(u%N0aDJBvBL6X*2o-?)4d*AU|Ddx+DMbGct65NGQ)_RmH+5@9ZH
z==ZEIbPwzI-v<4EW<7!ITXj21BVIQ8O>Q>FCwmv$u_tl=${fn|<b&TS>93HF4{|x<
z??ZW(GX~E``44CLz>$c{?QF-7{^@_7{qjpZzQlC!S@zriEvB<8InJ54vHk%1aRBva
zpTX(&BD9-nY$t>EkX^y@$yuB)gZi?+$aM1m*k1B}rbCd+Ab)TM+AHMZovbf_aUk&*
zmd{|EO8_nO*W7m2>ojox%stGsA9`fGOr6a2BebC1fp#hh22PVb_5x~O5I$6v+uy|J
zbtAhKM#9HD_#46hhzI|4@KYZAZt(B*;4cRMLGaDfQ4jdTx3d2xzZd+6J>-{y{|yg*
zKlm9B{`ug44}6e$SdCvl_u#Jr|E`7Z{yX6R$;19I_)mH8JHh{lhy5t{m#lENzXts2
zXS?%b;2-S4Ukm=x;DgM=Mfmk95B}NU+aCJYfnVn#zX|+C5B_HG-!Kk;sfYb@ApZ_r
zFE-m>2l$cY96ykGcmsYNex^JBP2jIG{7Y@+d8OD38~lX9cMQHMf4RZ`gdu;6!GFx)
zf5hNt4gUS$i@y>8O#iJ%Joqx7eZtUxkD-5q!5959Pkzcnzs!H1Gvs#}`VU`UT0i#n
zrSYF_@PBE@ul4AkvQAh4c{4t;E;-%MpE2~G;~_8Wm~KNpdu^%zj=}%2q5qu*e}}<;
zpTSQX{Er#@2Mqp$27jl)f85}I)8PN!;6H5e_ZWO>AF|FmWJ78F9&W_vc<{~ok#%8%
z2Vd5S>rX1x?=$p^KVF!=gxA|%{LtgZ_Yc0#w6q^DIZ2Oy-;fu^1aZk}ywC6iv*)wz
z$^Y6nKZ@(8`1iiizW=nuSG`VOPZ9KEZ*#$kSl&Y?liIiOp35?>rfOKPiiH{rPNsj8
zY0y#ERo_a3-!BN&kvN6(*+It8BPX#xw=#y#=Jd>MjLEHxEM59{of>p;;nVrTT&<@x
z2fdGo@=3Z^T}i&=n|QBxuYAt!yeE6=Iv&?VZx#100=e3H*ZLYyOXzgt`&Hu!{Fm|N
zJpW%->}qjeVi7n?CcNs3J$(;QeG-t8=>L+D^^j7stPo|*Ir(NjPnbFV_wjim&`^55
zNZ!r)1NXSgS!+u3i{AJ-?&-NM4?QBdk1+7sQu{faH}{eHm%N(XxuxZw1&__|jz~d-
z9~tgix29p;+NE>ag{AR|63VjTIFHzi`F!GE%9sM?-pF)j1!JO<F|TG7(;;9Mm<S@@
zS>OX_pe2jl^p{uZ>z;#$@yt2jUGfI_hq`j6er-T{{a0D8{_DVuf!_x$_(Ra+;LYHt
zfU=%D{QRKy^Ju|grkBAVxo#l6+8$;8Y~YE&lYq6rlY#ZX2B06<1QfmPpicz`fJ=bO
zfGdDO;9B5%;Ca9ha3in>*bD3jo(~)Z4g($FDA131Gy<D|&A=95E3geX4=Am5KIl5!
zs7pfsl5cZduitu>t`GmO7$rWb!`A3@h+8`hTGF#fzwKzwzwJ22^{tGVd5p^zFvb@$
z?l_k*w}r9xGRExXj5D_}?)nttv^3~P82?<mR{Q6>h*A2Z^am?-iRMdO#_JE#K5fX&
z2A&9PK!NLlexyqr7Gc3E_5<Mm88?K-v;P4odiXQ!_fFt9ff?Xqz+J!}1GB&<fxid-
z6=>bfdS3?g0gnM52b=|b9k32K2iOQ~1<nU91TF%00G9*L0-gg50egYx14n={;3dGT
zfj0ne0e%2@2k=hd=YVP8{{w#8zzpa|fvxwT-2kP2PX#Tw0`z&nLExpp>w&icKLOkc
z{3USeXW3poFbEt3CV)v`3YZ3FfLUM;Xx+<teLx%N2L^yapaZ-a_zB>{z{h~Qf&T=~
z{2bc}07Jm5fZKuh0Mo!}J5X+5JFp)(2y}qfeVl(Pa2jyI1L_M}v}^}G19&9xDB#h+
zR{;IM1n?H%`+*+;-UYlH_&MMMz=wc4f!_iC5cmt=Z-Iy3&*ks~mjOdS2Y3~58}J_B
zkAeRL+7Gb4CBS9C6~G{HEpR>XJYWde1MCM50v%uscopzw;D>?t03QZE2K*)P&%mh<
zvcE?IPXs=Jjwa)$|4|;-Z+VuVEE;e8l;?}&`q5U%-z@k&*0=UkXY2FA!*kYa-0|@Z
z6*1F!uI6uhtXpGlagWBEZ~^t1HB2YAGj6Z#)%@*sjOlOoY5F0YSfzXcl>2z3k5@i9
z|5#t(_#Fg11o$%GVZiCYmjh=2j|3hCJR0~4;46Wz0?q^;13VUZ9PoJHYk@X!7H~GO
z23Q9?8F&h?9ykZs0Gtc-0~>)&z-C|zuoc(_9IN4m^~Eo!9e*s#q?aL|#8=Mm>yeIT
zX<djPQhy=vWqvhY{mJ$0OCfg^@OoeZcoQ&e*h?aPHqytlCktJ*Ka@IlF<1{k&lRAh
zJQ<{~MEZF8<^IE;AvfiV+%BgArvZ-u9tC_A@Y9BU8|fdJpnZ9+?mWm{3w$r|R^a=A
zcL0|mFrNVZN#MP}&jTL>J_I!DKZEqwPtaef|KCFH8K9NsxJ>~b1T^d4hjdKi^ZUZV
zd>&+dC-rATZWd57K8hbw|9<f0GxH)Z`2|HkE|XiEA$Jq-Gr-4y-vj;um;*iyocs`%
z-v^ulJQ8>m@KwOs!1n>guUYuvwO+6CIY2KsK9g8}#p15TYc~Wt78WUbJtcOoTiLOC
z`C>D9?edjnSr@Nbv0~BMvgDPkIy=@bTD`$kv9e>$nnh=HtXa5N*$FNRcC0qDbS^5l
zvS`VY)g)<VUE8suBe;xAnT(Yki`J}L-LbNxbFIl*zJwB&FI~QP(c0y!I*Yl|`Yh`#
z{0yG4W&?d-wsYx{<*OIIzz-2dV%^pU5$~0_KlvtLAMg_3HsGDWM}bcO2bS>t%_#6~
zz;^*Z348$fst(Ry54;2TN#Ngsr!D1tD}fiE!SwaOJAt19-VgkLz{x8)|1rSV1J4KE
z2z(g$IPh8Ek)15R5%><^hk*|Re*k<Mcr?B%yA*gf&;h;;xE+{UYh0&8J<E9Fb^TDz
z7ZsX-`l#{7!uN{xca6LIBi$G1U;5+ED=ywNFgLQfdvL5LqCaZfrIVwSZ7AG8A3V-C
zK=CJ^puVQ8b92fEF`$R=d_g`Rek5@pH#?g%#@i=Zi%*zbb~#;LCqdIJQISuZ&c8%o
z9i9)8dAU3nBXo%pB7T0v&Mq@;Z)o4y{JeqxO^nHPrOzLv3_5MlUbceJ=L2RS=ZC!<
zXkYB$7jz%e%WmjaZhyYDrPr^`{J9OK?bLj(B6LnE-;_7$e0`b>4?1|Rp+De}(IffA
z?k`Wn4~dJQhkFNn@{GU%UxaD_su8G0pc;W{1ga6JMxYvjY6Pkgs79a~focS*5vWF>
z8i8sAsu8G0pc;W{1ga6JMxYvjY6Pkgs79a~focS*5vWF>8i8sAsu8G0pc;W{1ga6J
zMxYvjY6Pkgs79a~focS*5vWF>8i8sAsu8G0pc;W{1ga6JMxYvjY6Pkgs79a~focS*
z5%`abz(d#vD{xlnz7iqOaU<OfKoa~kFblLzIfG7~!1mJEZ^YDFMz7}lHn5C8PC^KB
zaYH^}&`E>N9LC=-6ZQ7I(tSiE-9*_pD;X}{*V}tH4dFP5i|CX6+k`LaFE&bfCRu7%
znz_N%y<`0&oxw9!%x#D^&}K(xt!QX#X=##ei1(X&!+Fh9EVV}s#brXFQJG}V8Kpwe
zw+va<{kNZZf}+!?ciGoQ@rxDPAHcpQ0n1X~M3aC1pB2(6S<Bo{==&_U*IJc0fg?Dp
zwEv0m&}>f6`(Er6>6QIF)OX4jT9dm+hDRfV13g3?+Mo;kYHKR*)kdt|BA(ihIL}*5
z#1U&~ORU9fZNH_y33kxh$fj6iIJ|K%qLdwRo&NgDk=7(urkI-|!|Lm4#8Qp)C@YX>
zja)Pw8KpEF#jIGkFDmJlb&L{G8<CT~-r?|2q`;afelMZ4O}Z#kn_Jq{KF$X<wzagY
z4Hu8pyC7*tx?-aPeSO;MB=t?Y89hCNUEL!?L*e0`u4pVgs$4z98QpU6#x3&AyJPcv
zBQ=!h=;|FC?k)~VBSK<LZEEwYCVo(3V{5Z=Wolz%quPSIfb-iLiJml|{`4x}=UZxz
z#3IR-c0;mVYiRN}sm;18OR8<X$rejA`rDhBo~I?lO5Dq)%26!Ytmr0xql)9Cijpcv
zK@#O9Z<Q~VBwL$Hy%n$Lw`oc52(`5~sTda7M8ujkR<hYLUu~FPkZjToW@=+|`?wLB
zFE+V${OvkIjV<FwsI^&Qi+Y8(O@8G|#rmAj^(S7!W|O}~X{uDt)@H61krbTYrt7m1
z+j(Z|Yi(pNTicX(lbdwBn*8&$Mb-bf1-Zy^ZHgR6r%kuSNd>vG*52B}HNtXJ8wK0i
zRYxj{PFuOB937EpZBz85LQprV?&TuJu5orTXo|V`$@O0lW9x10Mr$pLOmiz6oTuv>
z5pA3|&*)h$ay)W~9NV9#+g(xJyU3}?O>Go8SZ`6<isahOcs93k&9mGT22#^vRD_El
zSAAX(o^mTKv^5t&Zlxl~owK>U$#B)BGC25oA9ZL|kyT-f8clg{FqqQ#L^-80n#}Is
zQWhND9H%ykoVKHznjRZ<Q!5wJ<8DDn7t3g$<wEVcL*y;#j%Ns!7mG(H5>k%sMX`8<
z$;(ZwSh~rpVzDtd56ik;^L&+AgcQ0(qpV;pT(ihN@q)GZbtBBnO{`$^bq~l3O|W39
zAW|SHpgEqn6ijy-vZcD$-V{uqP4aRRDp-rIDdig*b1PW6hkTYLA(eeEJZ#dPGcPwW
z4;v*MY|PEWvaZrHPlvv_S<k5^YZYeTU4-=TC_;KkQY1thp<0%UkaCD7m^^Gu)RHcf
z2)SBviMp=e=6TJkY<m-RJ(lL>CKh!)(oW3_xkbI)!)6ZPJUy4#3lDil+$?ev^RQX>
z<Gj#>JZ#mctf|fIs4cF$R(*aenv0Z`thI$>2sxhnG`F;>_FP$xD_Z2(<(5`GY8U1{
zF7~*tMNUP8+E}ZSFS6Hao{U=cxm?OOg{R!yhV|J<x8bt0ZL9VtFNJd_nbQ?faMMMQ
z3&d{H#6Xixt@=b&<fqGNnj_DGG9#f9>NpokmPLkRFtx?6*KvE%#&lg3<R;d}T6Fd0
zjZLVHwQwD@`1LtsFFfS7nU|ZOhk8zs7n+!dyz-<4imI=@@UT($@4Vc^JZxkS#h9Ci
zE~Bb$|E+#Ko=$lD)}2&@CN`=zx9KxQs~=+)?9MCdX=*|La$MtRBCULtgF0lRpQq}D
ziELSiY;IEuCpW25dj`76?8eP-hNaxXTrM(Cg;@-#G8LV=ye6S4UM!+!0D3UfBa<$W
zhmc;Q7K94URT7$Cm|e(xXk7PjM%AH{3Cj3FdXZXiZ(^;jNtFUZIOUCRsksnvI#2TQ
zdA#Vg=#>vFVOW`frJ{O5Paj{%Uo6C(&lD<at*tF85IAkf`GBPgr;lc{wem%mDUB_>
za8gxWB%_;Axg~w}h75HiEjzxHZVW|Ih1OSCY?b-fGLK6&pXW^l{|YU({4}E%B~#k;
zB7;C_pp%ClFpHMTE^^8U38?V{;&dgUdAx~2i|!#LpfVQ8=>itcz%KSw3opp&!K@Ld
zQ5_R~S<gjIo!LlEhlY!({L}+SMSE&IE6C{+eq(D((Qv7z$u9O(b1cY-pGsS$c(xW-
zz?6B7dU$SZ;c>TupL7ynd#Dc8F{ym|l2WDVMXOW*<WzM`rFg1nRrV8sgq&_KB&Wxp
zihe>sL!o?%Eb_Cw|L90eY11>e$@)|`Wu6|9io7g4t+wiOF{GxnyN+39RoAMoE<;GV
zYPm5NISvU=E~$%avdqb{qBSF<u6aIZl5r(mR5TYUjv%CT?KkP`tCgkHXn-qPll5s}
zO0#+0w)~2Pu2x84)=Of#4;1DCWwYgG-HITVpYzgXG(9*JI;%_Bbf@F#H6qetUhQ*{
z(p^rZ5D}vxmrLpDLZR{zF{H{PqE{vo3S6~oZZry39uYk$hgfk$jM^@5Tl#VZ#seBv
zxO=zglYddSago!*CHFIoM%2B{^@@v}9xn58s;#Nv#OS6ja$E=ca>SJOCPhtcY2?e8
zm6}Q`*EnuU$cn68x6q;n?P5?1vZX2W_%a9?)B!CREUWfb-RLnSc#S9J)^wYPRDOh*
zr<Vy8z0`q#++GbSx|8$3rR3a)l(Oc@cmKrDWF9-`o7cTv#B_aOb~2emM3VqD6cu7w
zF4n3BmAo|`ocXe2kyu%0;7Njr$t0(#T`wbx8nug{3WEp=PHEFik4hF>a9y8gUrJEN
zv&_}<66zaqBH$IzvgXpF*C+)sjxJxwE%LMMwAIq8dS717T-fOuW_eWfZAKCD4rh7c
zbl(#}g?bn>ruOCb^xUafOb4@#PqKyPQg$t*1$O~C`m)j?MdsqNT#|1H6z=Box}H{Y
zdR(exvt6AI$VPk9<WmFmbF%KqMYp*pZB!X(A=;$oa|>k7&lh7=$qs6y`|oNr_0boM
z=stO@E8HC$*fbE^LTRK<-=F6{)})xVXHVf4x|-)t>ZNRxdR2z}4`t~)mU{4EYGYHY
zY8?kPHZ|(FP9r0uV}p@sky`yz32&l5<prR`>QRkAH3HQLR3lK0Ks5r@2vj3bjX*U5
z)d*B0P>nz}0@VmqBT$V%H3HQLR3lK0Ks5r@2vj3bjX*U5)d*B0P>sNUTLfkVxWs1>
z+=r*#WmAB9JHH2jy>#`el{Vdtr<7=HG(0dIv(H?$yi+ChM4ZTQkG-nXt{vHU{y>kt
zupn1wpS`SOb%#obZE+&Rs_Ct<7j-VNM<c!6gW+gY=dU3(k<GM!Ky+YaxMsnE(MX?W
z>YSz7G*y)4_14IcJvKZXiN<PaR{*tNK%E_q+OgWrb#?W24gIfa@apXdy}Tdm9U17+
zW}MNHZk1g&6v!8}<PCp2Vy{}gq+_*x`Ua9GS?hc$&YBu~?Xr$eyCzy=cdS^`VfRFO
z!`*`v37w;(bIF1Qv%CAlqqFOoGdeODsjH*FEML1!X<gpAXvOli8|+1EY$q~0R7=A3
zqazn}4NL4gSFN=>)-PYPc8y(Yqly~yes+gr;lYu<285t{ButM>bVq8P0jFVLcp&DC
z4s43ntC~Yb(ytsD8;sR1UPS&<1$7!uQ!TyVVo@=a*;@67o(&n4QqWJT{d}sflirb$
zI#q5Cj4HTYivkuqJUSZQ(iQIOQ=uu;17f5brbuZOn(>m2k4v@dYT8%zUWjFJ2={xP
ztETJJj?ktW<&~aKEvfX;f~R)px)m!_Lr{_MNG@d0SFOrP%~%<<d+Ze*OV{SdB;C$*
z)3>Q3=&`9s^;>F!s$)>^RD>Fb6sgA)=BkmXNN>?Vlg~}U(XowMm%q-o!qM2FHikJJ
z@@Pz3I?ZluYi@70iK*>~dDRmsi~6JX4XGpks@3@pUe>>9upN%}j*Jc!8_6e@SW;#v
zS1px~OWH@foItuEHrm@eIx-Z!aF7jpWr~eZmO6OlG|>2{!n=Hpy@W=f<(;(Q3!e&#
zhnG5QaHLlacMBF2H98u^c?eXaqOM^Y3{@@Dt}>jx-tK9j^8lYUh(Y_n3=EUg;f)0j
zhq}P*9w{0nqE5Hk1t#AjNRmcBC)^#eW2h3Y3_2x|rNIJUk2@tP^>$bfG^3HBkxh}>
z5<&})I59dqMeA$2`y<^KbTPSbaHKmtSW{=DQtft!dm=q_?m>YeY!}8{H$Buo7j<s1
zYl{um;RLf{)#61fs6oc)Ov$5msfN7GN=nH5;zdh3mWUpG$||lLT{tdVy&Ws|LzN^p
zLd~{IRiKnd=eS^iCIEcaP>oQ|R%MMDQw7WkN>2agd0lrJZW*fbq`MY(KAoj(9^tBp
z49f|7fa_!++BMKaI}*{8i(S#kg=3N7?nu42=iwM_7_@OL7O}&zafcdqcOpZLtFsuP
zWRMBR8p!(a*iaqEjcKX_-N_4~sg<)}Q6J=1ysyul?14Jj7Mm)ZV`{o<>bd2pjh$X1
zlbv#>s6DIm#Kfm6GzzN8nC>n#X1{3hRt-W=5-QB)%8R$ZXBvCo3Mk-vIZrSmt|LEA
zT3sEPp$YeJU!+G(4eHezr7sfea%fYp3nE+S>`ucgjVF3&E+o=wN1Y-D==3p4qXG@}
zdPY(p^NJ#yolzd4Ro0CoW5Yd4xLBo5%%7HYFSol(#|?~yyl`?GMSFN&f?lpqT`~&N
zTt-$}`H@kl$(Z5SW1~L1YUA!oWK^VeTy0HQEtP0+tSfbseUEkPIh1x&p95t0p)-Uy
zRVc%$r`a_Q4YMUvvha{i|EpD^&Pcx(QUXV<B%{mRjg9Q90hMNLI0LB)*XOQ*VT!=0
ziohE6XOsFvgP2`2r2h1(KdM0aMWA-*zwKH<wY=2`R3lK0Ks5r@2vj3bjX*U5)d*B0
zP>nz}0@VmqBT$V%H3HQLR3lK0Ks5r@2vj3bjX*U5)d*B0P>nz}0@VmqBT$V%H3HQL
zR3lK0Ks5r@2vj3bjX*U5)d*B0P>nz}0@VmqBk<CQz>LZ4N8*~-E&j<m%ev}=Cu%$h
z?p+W1NYL*At-e{Wj*o*roJs3m(Ae?c`ien+-Jl-<4e=`cn<Fq|(!be(mza5p2VQMe
zp9j&0dVSQQS44;VqLTDqf$D=I>cFsJSwS8*IMuSUt4iq^Q<*cje`F{!7as?i+u-y$
zv=<1m5+{`AZ+qtWKYitL%i43yYc(DX#n1kg)6W23?8RZP(?jmbf3VznhMdUvA$|F=
zrS`TU{YnqH8;~x32B2{1-`L)VJ>))X$W4RXVwCU89&(Qxa)(L*VDCvoE^!LS^^b;}
z#Pg)Tu%Fi7fzX#kJP$#-luv!GMIEm|`tgPwb1kdRkedU>WY{|${OA76cGIx8+>kpM
zau*{mXM5P|H{>KPn~{F0!I$=VCDK=3X6tq)<+~Z_w;6KM&i~g)7rCz?{a&EN!$$x3
zg&`L}hP07i<Wi`=A9~3B*+Wj`_ZV_AtluQ{E*TA+j{YL`F7Zr_BRAVa?mLE@+IL7D
zPtv&1MqgaA*K<2n^gM%ZFQQw>k@-cmGiX_AoRIN(<H*RMt#(SV2I%G7o(6e+wy?tt
z?ZZ>DcZBSZLvIVKEqbDOM^wGtx{ws`wj_G<9`#;ufmGU=QtI^P6NMytO;_zZP_SD3
z!u)>O*M?o;pwiz(`Ct5_wvOXF9~gF=!2v2dz3$wl{I~r6rY>@_XKaWz{}{~6`;fmM
zxEa_zI6#{ybkXT=q=!y^*1EMz+pVJUTI>gre>?Cqz$M{WWKC?#V8mL!W>vkta>)iu
z_djW8BA-D1-N1hUX(OsgUu3kahxEh-DE`*Suqx??Fb;}*2Kj4G=JK}zmy`VHrtl#9
z;EY6LeWRqr`t7qk9?AL5hy4A(F(7Rt5gntK+Q*`iQDs`}1;O72+zzB{o5<a=crE@X
z+Y|XX^6vsZSt?ID)K*zmV@nThR7GuH4v|kI|4fV<4ZvuGw%nq+4v)q9X<wd;b<JCa
zk90|Yw2*&2FedV`@NiFfv`1|Orgo)K`?b)9V0zxek!MeWaA+1#`nC8eDEvadu=IMv
zMut3p(CMR%X+$iKjroa3K54)Hh=2dSKkE4iEodax#RfC=+`i<4Wx~rMU1$?!ePpI*
zZ)AHa-7AkNQ>Duzki4GbDD5e88rPFG7YLCnOi`PU;Q<;H=&V5OH5U!Uy8CC>*wp@M
z0}<5=bT6{QwCh96j`a^j?cRaGNCTbCVxwCYm>Mh6;$OCb{hDOt7XFpp@73+_pBrYc
z`~HXj@(0N#eESV7ACmI5F}(;G=MD~RoV&Tbt*fnN&fviC*ycHX!(%FOI2xtCJ><~t
z9?_A8CTpa|ru~#I2=~z$abP(5Y0TfP#2jtkhW;JLb!s{YJGp^bdK}N&huvMxrSTTI
z?Z>nHHefq&J+L3R3%DEj3~)1WD{vS5*$vzYlsJutJD|tRx6?>}26|^AeHyUV$XEMX
zE}yAS=w;xy17`xKdC2v9q>H|Z;`)80XQKA@cfMtadjRb-c?Y-CLoDtYP7AlcDR3lX
zv`6tTfqZH`_G<b!iT<f;T=u(JpC4!e6@Pj_{dGS+c`M8LpjY}&x)$xi*P#1PZH)WP
z%%;+QB3h;U?Ev4*H(5DM$9rdF_=oW~DE5oei@o+0rMvx{b_<t7{K%cg{?;CJnfBjJ
zW^dBA-(c(P>v3L}>x!ZyXVjPY?lN*|kwp-69q%kJDX~_VKg&2K<(2VGa(eL}l?;1{
zF|?Pfzuo<^V$lyGeeBf(I{j?WW`0S3FZf3f3>=~KNc#0izxl>n^wL_=uSEJ$Mt<=@
z;wf}7s=AQ+uy7tocXB({=TG9#Di)yMq1MuNF8r*KE@Hx8Fq!+yKQ~<YsrR~^{}Paw
z^O4BQeBQM^$oY?)<&&Rf<<EDS!})xNawRU~e&8%&|E2y-dpj4DmM;al<Xc%j0rUfd
zzz{G4%mQ;j2N(xtpf?Ll1Et*KVG4T8d}$*+2faQ?2iihIfqglb&(tS$0DM2tC;0^>
zBQWHVZe$ym4DKU6;~ILAG6WF!0NSUM<92$8b%ORFek!*=xqp*3+M~FcLcYT&p^yHl
zaaRp6_IFrc2-xynrd2;z=R0$qL&^Hc{hISxuSffbe_O9cyNDlAa=x23zqFr-9y!l!
z2j9#m=exfh&F4GQpCHO5`V=bP<rwdL*Zu=8hxj4m|Gqlk#qYAUD{-9faQYVatgKOA
z;ycb<MI#l9R2<*rd{-n@F+Y!ogK8|4e1fjyC+nYhei?7oY6&Cm1oTI{T=L5}EcGJ+
zG}A@Dy`0;dncqwgLy?T@qR-4{&Icx%U#?e*zZ1pJp?{h4gD)dp<_EWfHuJy3BVF?U
z8hn{INV=)#a^y4ZO8Sq$mwAThc@*g~&ye&dCK>=E-~XNa)YIy^jFem0e?ht#Co}(d
zk<ZL8=`+t`{SpsJKLY7eK1rW~bSa;tOZ!V=UL^fl{1x=$$1I>9>-ik6Td3P*7xfh8
zry^I3N_SpA*6C?nSKuXnVLq0C{vfb#bY#pS`9;MFjo&5u4W6k#2fZm^v_IT4a?!ri
z9}IAR_5u4~;D!3HhZri;ER-t;y$-OSt`w__L83o}^x`m7ijU|Q`C>WwT}i%YSXs9$
zX-}a=Tz(rk9En{tGI{}L5`}~>)Smb&_I#(ay#(yhKH;$u+MC-nDE2bQSLBLro2ra%
zN6$u16?wNjO4du>^fwKE13+~hexO@l&nr<a_TtD_%&cT@U->%=21AQEzFGJYiS^SZ
z|3dxQ;Qzb)v#{=o!;ev?Te~9hOaGhvllCcb@xxE*4>F_)tHj@uL=ctgTh_Dr^`W27
z4}Q#dH{Z_lMwus=^8vB9dl}cW&?49WewJSkd<Obvg02N_18xWI0JZ~{0k=VZJ8&yd
z?2d<<p~uX()kxm~y}N)rfx8WRyA3&0@3aqa`JX|)UBI0la<#~B>e*X*=04K1w|4he
z{%MF;0PQp3a66Ujp{Cue(H^DUrWbNMRcpd$=-<ofpSnkDH?h7XFxA2|u_|5<osIcX
zNY;n&Z~fP~-mT<s^?XjQhnnj?rC%MgF57tq`&SEOE<Klo1=TGc*F$q+UjztMzn5dY
z^`X6(%PW337xMMc^pu13eN55wLa~2{&j{PFJ}kKysFY$IBl^X6y*?~iiE)nnc;)-L
zj&B0x-~QllG%b3}b&jNOTUk0@iQKj?u>4kFEpQodJ#Z&*7jQSQAGjH~6aMT1?lAll
zf4p!z^qBk|M*42(ord&hK+iPt&HQ3%d`!LV;MW4D0iQASh}`;d^hv()aM?c6GoGCn
zi{FlT1<*c&&D>7C#;b(U9%a1BAYFC;3+P||de)JwF9Zx6#k4A4rTaGz*KxkIj2G~)
zy$0i5n)wqQuQspXezfb2rTt09tGhhLEBjx(o|Qa?{gm-au7B*S>si}yJ*yw%l|09z
z;;W97QC}tFRmqhd&i^RGUjMxg^<}R6KZkU=PB0AG%rEJ8fiLTRNxuW>vhJ7kZAh1O
zzobh6cr>@H#rgXeKmU;1{fjKF4_K?XeWY<eL*zU2@~7YCCT~M708HY#fyi&j%Wu5P
zO+ExUsUOq-p1l0&pLUa%{8Iw*i&5;$gG*!mBb~uBR^&4Yzh@w-zU7opmmk7!YiVg}
zmBb<x{@mD*GdH-p*VMlVdF>v=HNv<RbZsxw>ye&AdT)3z8mW7|bp_;W2(P0%Y6Ahw
zT1zM8&`)Yg&i8-VV_8x98lh!HV)Stk!qUSz?<Ac+1v!71=^)SnuCgLEh0jhj=dVXJ
zHt-dq)HnIL78~kk0SixbmD<ho^G~|D^S$CH^5ez7RBM3}`9ezRFX0=HA?7m)ACD;J
z)8&Wo$7?@weNE&9MSksyHJcjg@fY(D{&@NePZE}D5c%-%mQn_&MZf@_x7)Rv+v5|!
zwe*bRIU~anYtc|-lpfriySShJ_Kk?LC-B4fKWN8mxLpa~L^Vw3-(_&?P^N$fojL3{
z{k)H`?RhUN=?{{wP@d&3PV@SGE-~(;llLyn<7It+34L9T9@LEL2UWXb^z~2u)b7G{
z@=n2JR&5>Ko%$dYPqK^_J+~p+5IkeeSt}a6nzENmFDP9<luDAf3|ZFwx1Y#2=ZKc;
z|LWOBE=Dv$k5=|{Md?FqL*a&@NH{u1pQKX{*=}j_``d<^Ed6cJ{rTO?n&F!I;fCR;
zLm%Yx{C=1IAl1J5YGppA5)b|S_C`m2IMtF5m+qJ6r6=-jt-?3^_~~jq)r~LS?X%Cn
z7DY54j~nqz-M>}O#J?qASy$4-OZIz8&r>R!`s^V+;h{gctLTyg`BmB^`scfk<rN*P
zJFzBvGt;ThGiHD}pvtF?hZwB%PR<wjCS&>=K+u72fd&R10Syd%8#FMG0Syd13K|&r
z4(P8k2E_i?kPZxd9W*cizu>lYA%3VB9CEm2wbBr;erGV=N<OvjueA1;_c0`Y7VmW=
z@cu#yI7>2oo6plZpd{z{IFBcRCSQ+m`Xz?0-V3^96fE4Iko+d*^}`=8zSn#~`lITI
z^!#V6#VUMAc)#>V{y3QDrv(dLQD-dr@MLK>q>p*o#ou1mxhoB4mA}2551YiZ=y~-#
zPyLX1B{07m8P;D)`)~V{c0MK&FAMED=h41}pYh<U_fhou(JGzq&cH&}cDUbXG2ij6
z<#|s4=m3|U&FQ;<{p*>22Do(t(=*Ru+z#wN7x{o&QNJ_ajC{cMw=lgExW0?&-N4Nu
zrl(<kzYSQsk<)how;s#uiJ8YSZU?p>&-6~<`qwhO8@SnKdfF_;ZNS>uOz!|LJAvt4
zz-=coUHdxZ11>uW`GEa3Og{tMTFdmzI>sc>e=^f);IdPg&H%U9Gu=LiaVKzn1Jk>K
zo98k;&Cj?ESlh_-4&bsTrgs7Ro0)zFxTA&XWvz@E;QBVEcLO)iV|rRU<2GRJe5Q8*
zmn~p=7clgCrgsC^JE(Wy<_l5pqm0{twNcbNa9ND$UBLb^rk?@EH!<zIh%pK5-;8{~
zty`F$c`@U5VEZLZ?*y*Dl<D2T&6hDf?X8U4fVG!1y#u&8j(k@jAF%dH<O42y8`Hah
z{Z}#l3~=k!OwYWAaXYa6?M&|kZrzG}*CHRV{W_*)z>)n6gf5KJ4g2R?xO$#k;ZwL1
znMzka_pDuG*V<*DQkK^h$A*V#@l)G95~gRUx+Ap{na`Hb@AvjoxYpJ%kGJw%;SC0T
z6X>S4#wM$=sl9cc{@X0|@>lR_DyxrYK+8K$JJzs%lWz9Un-5J6UI{ByvDItIbrG?X
zZx12XSIE4d+QSQr|Nj0~?_Rgkz5Z0M1pFnASKzv|a&sPCoL0}fs-)04+&)yg9F^w(
z{tcYp2h3r9oy7ISG*IbMcDB*=3)R-MMd^yCZ0w<Hg`P(r+p&QGU<eooDtTqcAK-hU
z$)a?{Q#LBuNh5z2{VoeSaW30W0hKOgXG*}bRNv1Or7PZeM=;L%9AE;N0%m|pkFwKE
zsIC{~iqaKtyu)_|+wlW~KnIurDm}{1Z~T0ZC{>g`o(;v%Ab$?%yApPRL7?a|=fhs>
zJ(&-C@tv`a^xn2*dGbfMT)c71`JVhsim#N>bCYt2f5cy0xRfT`qV*=`>0b87^FKeI
z?<%?8mA}3%>rqpGe&q}MX=A-0o>Fz{L&~GxX;<%xt9QM-)GPTG*OTzQ><NFo^6YQE
zdI#-Q^v{e(-ajsG+21_BpUZ1z5WbY(i?6Q7S+pONd6@G4M|ph~8YqqL{*KRh`t{xh
z)Xu%)C*}9zkEefs^QrCXeHFa?I{^G|Qu{(2DE<%Q8}ZV<DgI5HBHgi(QA@wKvU=4@
zTg@efZ^plY#u&9K5Wd+yYU*Pu%ZL|$(6DdH*I6g)`zv1Z$+wj9^`*O__-7-feA9n>
zsFZKo55+3;lb2NHXRoZxx34SZ)9oI5#JD@(enK~v@=gDf+e-PS|Je^z=Gz~s%nyCC
zGC%p*Qoi1bra1l&l=4mg?T1VGrvIUDROTnYQ<<OrL1n)EL@A%P!O{EcByqkC?Jnh;
z{s(Y9-m87eb$L%d?i+aWao@meK49KIkokaIFAQAD=cl5LfrcBXkoC4+Dyh>&Vbh+y
z+A1i2+e~c}1j&M`{l5O(>y@6%((}N(9$|;0HU(adxOOeBmF)LW|F<snRj@}T+MVlG
ztWXK+=A~O2GTQonHC2h~S$dN!d)Qvlv2?9{=BnkL>Qt%}bUC8~o1!)a)?U?V*XkZg
zcUe`S>7J|Fifd8l5>0jw(hE?$H&;!u)|v$i=v1M(`Me|=9owk)@$%Q%r8C(4_)?mj
zr}F$*<@vL<pZIb=N#76O6du(##AD&Xk-q(O{fh5!dtcBj+<@D!_ri@Kh^}4m)eET)
zIp5?f3O`EXr6)c6`F`bu27xs-^}P+z(M^?ZKo;HoAJI>E@QwV=Rcq~z^~=}n-Bn<#
z`=ZkI7xfWUUt@!@E~?u?)XaEh9^v(y<Pg3&|7dQq^b+2SPnRC_BRXDub#dY{GUCOb
zv$@RwzQK`=;X&O0TWIi2|K$3ymwuDq9|=2^)+eTZnFo34|AFBj#Xs8L6LBKLJdZN%
zOM4c*Cf~F_Ui}PhDUF{ge~kxU?!TJ$$^Vl0+i*axulqK#J<2C{Cic($mY3%90ejaw
zCo*Of#@T=6L5ID*uOsd1g<`%xe2d4CAW-z>`Quza+}r->rC-{!$)|RzpQh1M5n|fN
zAnOCkZ*Ssuy$iS<<LWZ>@7+MjU&U$!suB3liGYk(a(*^TCVVqp=ogEx#<P)dG$wWh
z#r{j(|Dt^~@#r&`V50t)+|QE26N-FMQpwMMl`r!j2lJ`@KL2^~<_<0whW6<i<wcpv
zz1Yt&<l6)8`|kJlK>M_Y={+OQ2WD%!YwG{ShhO*o<hI<Y7k#2yG79WJLfS+8suyW}
zG?DS4JpB8+ukU>;eX%?4{%ZmynrEpOk?BZx*6I6qi6CQWHKTtmqxB}n%-M{|4UEpY
zjDa^Z`nnjiA;#23#&{27u$R&9XUq*SrY~Sj3^9gA82uMAT2aPKj4`>1(b>!xxR}v*
zDP#6B#?<AE@hcdEZ)3EtX3Sl~nBK~mxQ;P&J){4fj8=j%b0cH&CPwEyjDhzu`u>M8
z`##3hZH)2vGX|54_8pA54>6|iWK4X7G4xSJ|Hm1vyBIT{WK5<QozE}^?qT%Z%b5Kf
zW9sva@%tHr4>H<cWXz=*(_dmtJj@vS3Zs7~qxE&h%r_X5-(qyW%@}x;(f3`(>|>0n
z?=!}CF$RCgX#bco_Y=nS&lnR=Fou4?=>HX?^(13vH)HZQjLvTv1HWhV<ruS1Gp7E;
z82<}n@NbOv-x+iNU`+p$G4VWO2=}x7lli`)HI*@Q5M%NXM(1UWfx{SmM=)lmGp2lu
z@go_7M>E>5WX!#aF?|eU;x&w+;~4#~Wwd57W@a-cPh@mXVhq$W`c7udp2C=#!x*2-
z7;I#;n;CO0jOjMUL_1?>0i*v^M(Z@j%o`Y!ix{27jDZeD-x-YAWsIpa8RIJ$gPn|a
zkTG`_V|oo^VjW}XY)1bEM(bS0%z2FRJb@EpI<S$^*TX2!6Quf>jt?*fFJP4C339_s
zrya(`D5E@2;NQfwwV5%qg;Aa-a4usya5<yz3PyRJAaxbf@oN}^TN&kfg4{cpPT#<o
zco(BQPvF0aY3n_VnOhj;c>?EsOb2db^u3=^o+n6skm>k`7=w2*%JT%d?M$aX#+dj7
zqdZUGPcdzMhB0$DqdZUGe2(eB=NWzXGiLETLFx-k$J30#FEPsV1i3FWo&G9g;%kiZ
zJc0jPOk3Y(%w!nlc>?D#rUTz+^zCAl=Lu3jVmkg4#^BEw<#~eK&zVmDk}>gXM)P?B
zK1-JW&9B$q#q%jMhw#T+?>d9(jk>`weZHhSJUH0ZPoFgzq))hr{6gGkNM`stra!^<
zgfIH#dcF7|{JgyJe4_C8R=$fKNK&^@z2x=7Nc5tt@J|*0VSgAa1mT<Z8z>a&?v(IN
z|K)z1@J)GnMM3zceR)Me_@+GH(d-@+zG=TsJxxK&PvM*L`tC86QTX(ptbR^Q>ZfBx
z$KtgVB;lL#tCw`Fwol(+3*WR)Pjb*rIeV$_O?i3yLilF+$u)idP57q#GJDnPCCfV(
z3IEiErT*h<RGq>%<@sr}a7_588S-H?0O6bRgJ=N4Kh0=Au%B;VraaZoC^exx|J2h;
z{nw8PP$toD%B#l$R09{j8UK#e9k#mNFMLye6b~2(-;BRJUm$!_UY;)yz8QZSD8}d;
zC^f<tdAYwJj~57^;;#>Be>MgP;aj*b5(4UHlIeqG!WaD_ztE1U$CL8qhq^K_<;@Jj
zAC6oU-qdGu3;T(xkx>;`z2n|9M&G-sskfuC5r;Mv?THo=Ezu|CNufME9!2x*(K7D$
zsE?|tYa=4RaZ4-`CU>N~^v;uhj7RupJox5#p+B4Y=`oQ*ZM)VHd6PdnGDKT099oE{
z;a@{+v^O>qz0j1G`d+)dlg5*!tA($h%P8{SMreg^;l6qbXx9wX)YaK5@_fU7t-fXx
z%Re7VpV;bC-#P0Y8}3&3sp+j^{qD2KoAC(`=_hFN@iogY&)kT-$(Ltty!fi_)FU`v
ze0knR_|%k2o*$&<R6|ci<l}GpuZBZ4B8$A~AHrK`e@6WFH9tx38LR!Jldxmzr?0Tl
z4jm=6q_c?sW#xl{FmN{EGqj)3`2pIGMeQeq_l&{T@9wCrKrD;50+Dn<^=`m)`gbk;
z=_Pr7gk<UJCnN)KW|-eomG`CN2ES-Om`1C^D!zP)GuU9PcAR_DO65M=4~E8l^$3i(
zxrBZ^p!S2Iy&Kfw6<_8S`2X(qv%l@{tNZ0%?VGor7#Qw}Y^Lx1h4tRo)%X|M*ZzwC
zBz&OPS68pE7bb!JlbKEfgQqZ^1v>RiTXPr_K)ZqI6fiKC=?pOBXF3P8=P?~`Wb`#L
zCV~EDrqjS+3)86~#w=)C=<_+<*Tv{uz!(^2v<4Xyz}%a`Z)eN^L-U!=0pkmp_Pw4l
zc_Cxs4Zty^gYQ3;=`=97km+EQ(GCDNF=oNfg72Kh=@#@m8$ow7CV+Mi(}4(M1{msP
zItPsRF`evZ^batmfkDJAa1p0xfGOzDAbz0{=I4NM2l_^#59mXDe6W{7c~dB_6XX0A
z;ucuMbOsnYo#`AfzL@Fw7RJzK@PWQ1O#3coOoC2=&R)vt*^eUM^^E=w;3bS{&}q=Y
zrJNoVdC<-oobFu4Xf0z*0PW>Wr+}F=k?*a@2edCoK42!!wC_!f{<9g=z~FkOvp{D9
z(}{B!?Q<DZz$Ekp&g1mJU5uemGA7>67+(Rr66t3#hR{EJE1CAa4f)=}=(I9gZHyV%
zNg{u`lhggcB;@^9ae5jUT*Y+oYQ`++ENCal>CQEb$u33<<xZ_;I<=KCum*T7V+M2v
zbZ9N7hpuDHfzE-BujBOiI~bE+Vzdw!3vtW65%O0shCqiv2T<<;)N2B9vf-}{e^c{V
z-XCI21A}3vv%usxknaY@B-(NCosdJi^Df9CJ&|DA0;X<c+6HFc&2#{myNT%#(Dxpu
z<2N(<Z(&R#J@{Ux(@1x=F`Y$v;(wU7fT{N}Z38p6G93WsZeuzG^u3?y`0b4T4=^T?
z9!xTwM!NGsrn5*-+`+U3Onr!H8<_bp(*a=aPNqXZ-$$5^|1YC|J7W^*!H+VXM!NGc
zrV}4$w1BBkFl_@fcQKv&B%?3I82=Qb|I>^~qz6C4bQ<Z--AreZp16l;3z+&W(>5@3
zFVg{F?sH6sfWFT&9pAy|zmGAA^x*wWr;+YFz;qVri3gdsfT=GqZ38o3WI6!MrI`)^
zeduTLhdAB;{}_W0Gp0d1UuHUs^u$+~wt%UxGHnAhJDCmub6;aR1oWX_#lOz!{%<k{
zzr~ma?L5MC7U_v^Gi?D=8K!Mu=24~tz}$D34gr0SF&+OdqyKx1Nu&qA&vY8;&f`pH
zk)GJav;|E4fN2|;`61H*VD3juhk(AHFdhFfqyMLjNu&pV#&jC#PL}B`(i2ZGZ2?n1
zXW9m4e!+AAnENHuA)xQqOviu4=zo$iiS*!Zrqf7w{*UP_(i6X7+5)DYV%i2~e#>+K
znEM^mA)xOMOviuE=+7}Gksf@S=`_-vKQf&~dg4z^Tfo$xnYMwMzc3vD=Kjib2<Ur;
z>G<Cm{eNdnB0cyIrqf7wo@F|V^u#}zwt%VUn6`nL=a~)wb9<N$0ezF^=y`wKV)RdD
zOd>rvh3PcXovBP`k)Ak+X$zP-m}wiBIfUr|FgK0q5YTrh)A5%v`VV7FB0YFG(`lqT
zM=+g5dSW`$7BKa4rfp!x$8-Rgo56Gl=sSw(_>qkMqZyM(559uwG}4_{GMz<w;#Evr
zz|>5pZD8gYrUSs-tC<b~eaA8#e+{GmIL0K>gU2(SM!NG_rn5*-*i2i%)GVfLU}iSc
z0buR~rb9sA>zIz8$ml<bF^TkG4by3)JGD$_k)Ehy+5)CdX4(d3PGLF#%+)g;0{R-5
zj?ZEA&t*&^J?LjTjdZ7x=`7L{O-x(BR5Q~yFw??x0GMlKIt291V>;f(=x=9CB0V^t
z=`_-v1x#m=o_Iae7BF=x(>5@(km&$0cN)_npfA95{0)r$MT|+L2Tx}@jdW)*(^;e^
zmN0DrQyom(z|2yn1HjxFOoxEJ<xIzyG5XJBOd>t_MyAt9cUCZ+MS5Z-(-tt*$+Qj3
ztYSI<%mtYa0e!2Pj-SQoU&EM0dT=e%X{0;rn9d?S@g}A%VCrn9ZD3|S(*a;^1Jfa(
z?_8$i=P>%uV@x7F_-3ZlNO#`CbQbA}E~YJDD#Wx6%!HW^0CO9e4gq~VOvk$!{Sn3_
z(u2KBr;+aTF`Y$vqMvCCm>OW(24>D@IsnXFz;p=c8)7;>$mkztOd>rv!gLzxj>B{o
z>4^)Owt%Torfpy*%5(sji!mJn`Zh5gA7k`i#F#{Sa5K|sq&r)f<Z=G1C?>bqUip
zFmox>0buAdrgPT-L1(UJn!b0e&xaxK>AS&vU2H25bUMzoADF#@=^)U08`I8}jP_NG
z38V*LFNJg;^vB=M`O?=i`hnT&m<|H1>zQ`m!D!#Wm_T~qolK{Y9(otk8Kj%<-^ulC
z6Xp8%xX-_*#(7=>o9UVQA40-w2xri}h3n|wRdoG5_yF&_DBn>~+0={O^t}W920p~{
za5c$YL;v1R|F+UUb)O<iH<d;2g8|FBn)Eu*E8kyG3h1hHJ#j|6M|vW8h4@jdS0yRE
zwmhs*KbpUPCijyQ*ne5}Q<V6b_h)23Md6$Gv+4C~eLqC_ro4VwAy%7zUctN{tzW~X
z8-QN&>ORa*Xul=loA=|@`_RRF)4s_!?`KaU{Y_2njiTR_SNvlA=JNyr+{g5iZ)|Fr
zXSMqC`pxoNe)`Y0HhRgASALUk`d?Ol<9P(d@1ZwTz3fk-@~i#RgfHz!?vE%wZ8zB~
zeAB+-t9t^%H{-A4LoZPa-;_Te0TRB%U+zCE^Xd@3DQ|18W!<O(MF+M?whl_3sK-m$
zxAO?lN9tUb^-9o_K-Ytw0lE=1bya;V1r5?V3-qx_SO;3gTO}QQ2iq6=+Xnx;hQ7x^
zi$6^%mKS=SL7!^S=YOg+{|IQ=cQj`3F99w7#SQu@gWhV$UvJPi8uTp&eJg0GpQORx
z4qE)V%b-7P(4RHv&l~iE2K|she;KsI_iG0Ih>`!h2LEw`|6|b7J|;o&D=DFc_FbG$
z`|ro8GW?$|JwN$~ukQa(vh;&jb=e)vSK^8{sooxKBiVWMubuw+AfLl|E(1(LE)5jB
z%LA5mg|hqkQoHKAJv1zzZCU(87MUtN=tY}lfkILc@(wToOyYYAX`t-VJ4-~3=M~g`
z9_oZ5e6v5(qf_dv=9~q26^G*I5$N#{wXp-aPJ6NRj*Qe<^28jSgw)O|iYX6qbq~_h
zH=LTcmwz6TpWBj6bcAo-zxH}QL&hiCheuajULJnS_-TXA^Km{&0%d=bJl}{HKkik#
z#)~hz5C~sAA5k2ynBHlo-nkHd2&j)v(Q{Vy>S42bTRj9sdqwIW>WL`*qnDo@TCgCW
zg`N`{)Fd84qjxb!!^6?ukx_aIN1iNfu5Y2IIG(_Gtj5neYX6G7nl?C%KUmrh#UF)7
z6%+U!BUv?uCm|>LZkj~%EYg>D(Z^#57tmBWfNgoQhQ9ZZzPGClSiey2pPC=2Lp3=5
zRrot_Kj*h0C;q*h`rKC17pS1Gpr!C@z-nxyDd3inq3>Rb!?h}Y74&se1r{{%uYm{H
zzoelrNgp=7PQ}ktU*kNhxy@8(4YW4X{I}lD8v6c8&-A=Q#qW#W@mopHx|s^aKO1rq
zzs7)NU9aM2=u_jCijQT{q-CSkZt9DnUPFeycaXjtRQz1@N&V7;ZuxE2MDGCfB_a1N
z(9++=dmcjZc?X8@Q2ya-QZDivW1|r@*_ZJ`>=yX)#J2l>HT~gef2I9g6kjx`aX}8z
z-#0+-Qd7E@eZ?=9|6kO5L4LSoyh)YwMO;6ptTsqj?^%j`)X^T~?ax7+i})q-O$LAC
z=)iEX{|jCCqCl1ZRP-;=FQ3=q=YGONmLcz2e?{vR<vpF2^ZtL?kT>J^ZSZeIMZA`9
zCOyY?LBO)UM2$c5ROvqdYBpJEy_rJ(1keHIfW9xXz5p-;j02tC@X)~EmVA5U&1^=T
zD4vUMZ=q#ogX6h=_(_`kyTsF!|GB|8&-cGG_#5dIuB=OYTxisnDc}E4seQG7m{x%E
z=%~Rr<u@CA{uGzipxf(}2H%w5Xf?LYr%^-7Z^}P6j{HGiGW=K12W#`(-+cz(wBKsc
ziiu7cGH|KqbBoJwL0;zFvOlEwEBbQs{)b`DT#uvK=F2PcQeP(Dyq;k4$CJPG($eSa
z4>WmmJ-xU7oAQ%LLGkqpGkzxDjK5xw)2ZF8|9t($XtiH7zM1kXzg*g0P5bK%zUjYt
zz0Z_4`KJAlq2HA6G5F^7n0|wA%IoLbsXs}4%=Rz+Ncg6_$v3Zu$okex-sH>m5UC$E
zKT?O6{3K@Q`DXl0zUjZoH{~V2m;KFOEo~oW{lyKw=#%o5)xYt6j>$LWw;KA*`g^y*
zH|1|J_-6gR&)|!^tOs$qFCTxZKYd8~Y}C&T^+66gRC`h1J5+1!r|7qh_wS0Y3sbW9
zf&4W6E;)ANJEB><KbQt4ZsGb#0mXiAz_NZz8Zt<qsQn!BtLq!Z$CI${doSDb1A{;Z
zDE7~%Mc(g7K?3Q;*V~Krn!i)XPnW|j)pmp1*xnsVF&!E3b3h;XexTUz4_MakDZhjC
ziIzWs{AT%|R}RsU!S@_<K;L%}51`n;Bw$&8p!`9kPt?AH{OUCURes-Ntp8%tr;Y^p
zDPRVe1B(4i1N7|?%I}B#MC}KWU*eww{cF^}13Cdr0W(0ce;L_-n(X_&$No>$z90Fe
z{29={fc+q72bch+fMWlxWdDz3KZo>*+V_2*{Y7<?20qkv7<Kr;4+0%v0x0%Rr}h1x
zD1Qp+(*D&oz)D93`KA2v$Jrl=e@^rvJpc>=#lCu%=Fem(iFC1F$+qI9kzeeGcCmda
ze-?ZT={7I`6#MFa)?Y|pT=L<2<m1^-BEQ%V{DAHMAL>61eimqfZv(~tQ#3LDD;WzR
zUA~V#p8YuTOZ&Hf$o9Vm`$_QAz%0;0KC%B>vi~<S7C`z$?T3(G+1HocepDL&IQU6m
z8khx&{oj%OXGn()`H9*OAivnpf>z&gQAY@L9GC>AfnxvnWdH9<;K%I$MD5$iFZR=*
zrTqm!hk$Wl5-9fnK=%JZ=CeqjsD0}v?625Qf)@KW=m0PTj044fj_f~6_R~n8sQoPR
zi~aadOY7ePZ36?q5K!ztP4@pu_LE4TsQonZi~Z2gO6_OCw~%fF13<C=N3#DM*^eWA
zqV`kBFZP4kQu`V3b3h;XexTU@6WM>B>^n%GsQm=;i+%qSrS?<cXMj1N5BbFYpUFO5
z>al`IpQwEY`Nh8P=cV=&;HQ8YU=Aqu|3dcF1j2{!Cr#A8ANj?82J|tcP8~te4ln^s
z0XbJ;J!3tF{d9q8%$sDrVHx*F%H;n=zS>H?iT3ChJEne_@5_AN<cr+Di?8NA#rJ<q
z{~Wa)2wg1l^6#ZyJ~ggZDj(P1$XE>P#<KE{^S!u#cm1h(d!_QJ_u{)Q8i@6GMaLXx
zWK=dnmHzQkuRnUxIyxAMI1=ZH){k7@)SK-`qS3CxhxO>&Gm(YsjgR<AJT58bTrSp%
z#+!d}ym*i%U6txfwa3v&_olrqulO%=N%$!xc(M3u9$l$?&S<o!YcR4|e`?SrzAyFm
zI9~bG_Y@<W9ojWX-`SAx=dbU))XVp)aq5r17pdNMcSfQEv4N3cE~2!rmwNfsI^z}e
z&#Ql^^2J8SBGf+V%U*@!Kq=oF$>lR9R=%N;q5}RWl~3Q(8VYZwab~zL)~_q^Kq=pN
z-!oP1lkeGe$0D&p3ICJIH#{~J86D`Rk6uN(2ckU09O&hf`G~5&vhsC}hI<A!7X};p
zb)c75{4XwVly($U&#tIL_y@xILaiYxoiBRA)D;JYBE=2%Kc#%sT<LMW$Uxt47Zo$y
zb<t?pp$9Hs^7U8T-&_lzTOs|ye$EF<eHGP5ls?)`HNnS$;=dXXq&|uZaG>~4?vIh*
zT|?9j28R2j6}{y9gTz<G->m?`f2ohhMa@S)f+rPlp!i?h{)-E6p!iSmcSZ&Wy0;vl
z*UMD=Q2_nC-htMB$o<iXLlqEKYwxZRdcT1#J|8&qDHVUW0vtf?e?s32^_nkQ|Kjz-
ze^p+^->m=#Q2g}^I{Gyg>AVM6`*$nA0Tlmegm$>y7#Y;pd8G<G{U76fTZC`6U)i_M
zi!b};dGTexJ1@TV)>8Yb{pSln*Pms7Gm$svKk+L|<xT%(A2^XW`Lf@d7e8=qW&N^`
zo5(Bo`||~$`~NqT%A5YnzG)(F`X~FQdGTf6G%vpFo94xreba=m<6l$&**8u2rvI{U
zn($5kW#2R}zU-Uk#g~24y!f(jn(!(9-6Mm8`s}OTBawa6@_h4tA@)xbzUja0Yv{$7
z{nWhpvY(n4U-nZIzKVaL0M!2Pt=xWWbnu5>$3N)yfYnaVp80Al>z(v|R4~oY4-`ND
zCGSV6_r4&W2M75@&kWM@F4B{5(WCYvuq@V}SAZWWL(h?<Cqa5LE_%ct^`!$-PtMSD
z6zRE<^!OesFPGXo+RGn5@=LjnCOz*~@o~{3{zOLW>P&weL(eNn&rK>mE_(R=GmrQf
zdR|F-ZdUPe(Ifthj*MJn`ja#Cyo&U^N5u#4eH7PwUthS}Gd}<1_lhJwGfB@aDn3Xr
z)+7F??|PX2IEJ2MNY8s!d|dR<N8M|^{7D&lUQK$osrb0)5r16UkD;fD9xVJH6`$w0
zyv5~GUzzvv$B+CHpVyE-?^E${(IfuQtIPFe@iFuqOL}fq@o~{(pSfzemp>^(&vB&Z
zHWeQiJ>rkr_}=s<XXt6B_}s4I^L%-H)HkQS{P82d#OHYO=lv=^E_%ctd-)pM^v5yu
zyq5HQK*h&Xk4OJ9^w^{)sp8|Q$NhcfoS|nH>G`0F&z|!5sQZ?l@j-rx&ur3jhl-Dj
z9*K`&af}>8&;5s3)`wJlJoR|Q$Ix>E`SW2FA5T5xMYyl8*cbIXXXv?){JB%b$HIR7
zMbG`x@H6NcALN(#oJjusuZoY09*K)!u}_Ynr-j<{M^t<~^?1a`(DOR-XS<4zrylZx
zCP&4-sNXq5&q<``qbfd=%HzYb?*90ZU*c0kdOoJ&<D$ncF5;JC=&2<=A6M~l(NoEv
zl%c1N^n60a$3>5N0iJiI*(h%0F=yyGne^PH;xl<K<AeMXpHoQBCsll0^i=Z4G4#}v
zo=>Uxxai?HyVqCB&@+ehq*Q!d^r#&IEbC&mZBJQ%a)zE(YR{im@tIN{pGyAtkzeA|
zK>mD2#m7ZYC4U@4&x40p*4-*TE_y1qA4AVv^5-5E9~V89>MLjH@spm<s`yMTk546k
z{KzlyX(T=Os`$9*sTdzaPZR0+oQjW&o=WwVGW0Z)o*gPaE_y2YlQZ<Rke<)0_#9Lo
zpGyAtkzeA|M(yxE6(1KpmHcrGJ+0)={VF~#dMedd%FxqBdLB^manV!BpPZp*9_e{d
z#pmGi_*C-8kNgs!c@&>7sQ9?(snnhwLr**T^F<XO7d>2GJP%>t3V&0Ep82FFt>WXN
zr;<N8L(c-z^N@<qA?5L@<c}ZuB|fhwJzrAsanYmRf2aLG^nh3>5&v=wJ*SeM|EJ>P
zrbml-#K+LHkn}vP;^U^Lf<HM!PdnB7msNbGmB+_wZ<}vf?*90ZU*dBb`STSO9~V89
z+OuQm+2XUTud4XC=uyvsdL1c4&&8x?r;3k@9<F!N)YPWlIw(1EhMr4E&(~CZURE9-
z+Ln%FJ>rA>5}!9ve7;V4f-ZW*AHiaW979il^n8Q#Bs}%F`;#*CEFwMMBt02VJ^Y@3
zu`lX(&d_r@>G>Av@f}(opNjnp`6WJ!NzWrHKAw8q{c#LEOGwYRReU`4c*Muh(?NPN
zDn6ci-2KTJdX|!&M^$_d+spVMzr^PZ((@e^A5T5*{y2u7Wu)i3Dn6ciJmO>MSx$N$
zQ}OZC<L*z+&~ql~`JRf;;d>b$<d^uok@S3D#m7^RyFZShX9ekbT*b#l5BCL+_!xRt
zlAc{EJ}!FP`nmM8oS~<a^!z}@=ZNz7RPx7<{1Ts4r00h!J}!Fl{xteq#1BEo&=Vv*
zKT`2=(NoEvl%eM=((_{#9~V7&f5eF*%o%!Clb)Zb_)Oo+_#nT;XASB3sfv$_p1ePe
z{`MkQ^iRjovzGMyOvT4jk4O74^sFO2Srs2oJ?`x}XXtqo>3KrM=jG+`;c~hA<41mp
z&)KBs=PEv)delBq`E7vmVR8&T>q*ZqRD4|Yu#m_4)X=km^!!rA$3;)RzVcoae&h^2
z=a8OXsrdNH<HJu1d&CF%B|hhpo?ol@xag_mk7MXLkMumL;^U%+=9x6nboVD^=y@~g
z*{$N^qNkESIYZA|NYDSN_{=De4@cGAA3yR-e7Z=_Z&Z9-^i=Z4G4zB;&r>QsE_x_F
z)IZ(*Nf~;=r02IPJ}!DH`I9sBY$QFuQ}H=+FXMy!5}$6;^LrH^7d@5yaSS~@q~{MR
zJ}!DH#>dbTAw4-29~V89{K*-5dP&dI|DU~gfs>=Cx5pb%jC#=#e4q#nq7anF?9M)c
zz=ZG!VqheI0mR+SW;fYnAH(kE!FvTo6p;ZGqlgZm7)4|Nl_;Vs2!aR<qKI<Q5fo8G
zNAQ6n@;l$E^X;jr>7MLlz5jdt{B4+g_tfLmsc%(RS9e!;i+{Fl(mzbE{#g#5r^G)I
zJdO0^Jf0Qcd0PAv!4tH!;)Rnj{m0`O0?(hsKM_2tM{RZVfloc2mEifa_~%vi{;~O^
z^`w|y{j&-@e-Z!0@<bmmg6%B7@(hFLui!~C+`Zp4X-;&b^<+Gr5%4?%p56$aD8HzW
zraYce@ca!tg$SNT<Il9mGX|c&gQpz9(@0P7x?2C_!1Jv5hvBY&WUWD5|H*hfCxPc5
z;-3f})uUE@BX-K;83)gE;-6Ta==PfScqYK}Pw`JIPfY)LeXV~c!SlTMhvBY&%&XR8
z_OFb`vl=|Xi{YOLo+!VlkET4HHQ?C-JcS4zn<vIU9?x3vyZ}7q2%bjQH-hbJ{j&}{
zTY@La@EZRZLD70L9?!|(c_DauBX}C=neuo}0ndxTQ;6Vc=pT>gRPbyCo^k|FBR#<z
zYW;H>cwP*iB*SOyACKpB@XP^EZv;;xJyRaf8Q^&dcnT3b4gKTsoC%(nf~Oq8(@0OS
zL#=<N!1FTjBpE(q|9Cv_0ndMdr#FJ9k)A1!=PdAS4W2>-PecEBJZFRF<=`nt@HEmB
z>{#oc_k!mY;7Kxk#{Thm-UprpczPpv8tIwxc&^CcdoJK9FueY~UeWvew8ygnJllY$
z9Kq8_Pq0(1e{KcOw%|!Je5QIbOurM4pSu1_*VmQrx9?4<yw;Yd@p%70SI@}Ms60L^
z@72{!<28RF=KTW5-#W|uZL`TgiB}8ECO_U?AC-N^&-YBuGXLQ4Sl?uX2WIMjlRu&b
zXPSScufK2EGCbepb%7@H*D083{-IGk)$A9JPS{lbq0#=~)$-tQwF}QQKi|h(r(mY}
z&2yl2xo56lo+X`4{iXbg$-a@CN79UblYjD*8itwX$NrDU68na0GB?qFqTegp3_&w3
zU)~2E;r<!-pEZ5skrXp6U!U#vHOw^s>ebN$$V~I&`x?s!@IESk2bgJoEDs)@7_E2V
zO!F@tTNhFK%=`aXu1A&;PW1Gzo|qgP(ZkP-+fVcBI%_4EseaAhGcb~ybXHq!D*uH$
z*^=;D%;4KtAASB7LGUUZw-dj!$5(e=W6y{1QtkHo>(eQwSDp#HFX+|a$wu(l5~JrM
zIgjUa(DNGb<Rf?-J-B~dyA?g24d8h#cuEmGjr3GJo-crBJMqtxjr`M6cfU8q^y;6h
z!Sg!tPXtdRJvoo(i{N>^_$Pv=>L1M9yshZ*Tmzo%#Xk`|jr3GJo@>GL2Jz4D8~CR!
zo%ZAeDW+HdTo0Zd#6J-{jr8O^o*TfkqxdI+$MH|AuczqoTn~<&@GC{|G}2S?cy0hk
z62HVB;{4+)u{g!_>YuNIV`u!b5j<EgHyi}%E_uJbzvVoh8^Q5L{PGbz);~Uj#YK<j
zXIlrsF5oFe@Yv;hE#`gxK3m1(`2k+A_$KfqDoy%_>D50Q!SQDNvJpJ8pb%dwlh0Yy
zZ8?wUYv9-wzkCEwL;rX@-vq~Q_?04f?E1#GHnYsbQ&jPIz6FlC_$B_>q<@%R{WA@Y
z-SNvt@K8^CWdHGaz73AI;Fpi!Y3Lu1=dI{JZv{^&g2(nBTr$V^Cq4HB6^~~R@azGe
z#8dJ9iPDu~diBqDz_BNO*$5u1r_+~WaL(ho2ON9hmyh649co>|qQ`R^IQGV`6v4yx
zs=H?_d=-!9c5tNdOFZ4Af0$nV^IdSX;Fpcyp`N;9vgpZqJa>Sj6~BB0kNQQe<kzj}
z@m!DZ45YzRir}$&+Uxp`a8*2>^YFY_8+a0bYSKSUul_j&>)qPHla1h^o>b)i<?(zE
z93A-OBY4y=YSlMliyqHi;ONA!6v1Qrb8GbY?D2db99{S&{@kR0m|p#JH#oZS%SP~6
zJ?Th2IgjUFaO{I$K7vR6qPE6A9#07z`{GxM;IVqzBK1@}p8LSDAAX6y)ceOR-~T(s
z^y;4<fMXth*(jdY4)kTq@AKq5o@c<ZKYsZLo+!U4U(w^)8#nF_08c4`$L5LFQ}K9O
zz>@(_;;;4oX{0B`^y;59c;<sA8^O~^PtN1{5_k?2|3vUK(o^(!t_9Bm@lOO#BRv(5
z=T|rfJ4pQVOq2d$diBqB&~vc(CxWMuo}9<CVcQ@$MEn!M)6hR2&t2d-RQwacgY~ho
z{kh`tT#tG!6#x9KN&hgt`sW7l947vW;A!X|kLN4kd7Jnrf~TQ>Jf0iDbGY~?f~S$5
zipO&kc-}7l`FoT8VS4q?&gh4W#6J-{jr8O^o;QN$2=Pw@PecEBJiCDBNbyeuPa{1Q
zkLN{r5qwts^K6s;VS4q?*P!Pe;-3hfMtX7{&w9*TjuQVw@HF&~$MbRUEEfMn@HEm>
z@p!(1dCSq_pMS*rCrVdkf16(Y^HuO1BRmJh@z`*yt@>|@>6Pae%;%2<Pd1h(dc8o-
z<GB?)?*vaimM2<I(c^gw&QF(srxeR$zl&b2k^QUU@f-@C<HSFidjHrQ(Rxx$ul_k4
zJjaWFB6w^&$@RLny6Y`DkEb6zCy0Myd7||cJ)Sk-d6)PnmdEI+TmM_}c;?}|y(fx)
z=GXhj)+<_1is{uq2Y~0@;-6R^qo>_(ubjtoBzStnKe0T~dWs&;Md*jU;-6R^qo?kJ
z#1)U{TIlH${~Xw)f0$nVQv}aa@lPy|(NlM_oAY?C15dyBCzdD1KOWDw!Lv;K6U!rd
z+Uu^bRy>|tz%wBJS<s|^m|p$!9q<f_e`0xzp1O0(oX2w;c$SNQVtHcx<MBKIo)zMs
zSRSLN?s{g$<M}aohQvPy)%&O6_`~$-p9jISQv4IaQyqWsF>P<lc|1P_&nodx1WzM9
zMUUqp@C=K8B6u7<_2ZAn^E2>_h<^@l(mzbE{wafJRQwac<LId$e>|RFfM-nn6T#Ea
zKOWC7!IKmJMDRFz>c=0C=U3o4N&It2lm20P_0MDA85jRV@Hl$v#~+X9x8Ru&|3vUK
z^pD50HJ-zp6#qo<IC|=?wN*TxC!lAw_~+0j{loO?pRJ&0jrb>m$Lfih|9CvVgPyhG
zp9mga|7nZ59#!;so&?W2@lOO#BRv(5=l9?_S^TrG-aph6dHp%X^y;5KfaetPPXtdR
zJvoo30-jUFKM_3kdR=$q^}3?R^GEQUCjN=wX{4v(@w^!Ae!BSQuzLSkJzd>V?UiDB
z_0Ln#bB6dQf~S$5oX0Z<dd?L8MDW=5>WI8PR`htz#dUxw@lOO#BRv(5Cl8+Yh=1Nz
z?;oqDGg42A>D51<&IG|(;va^)>(m<Q$$31NgXe7VPXv$E(-o<w$n^7gez3D0tmKs`
z<~i?au6HZ*{3?vM>m6jN+h)BFb^r@LYQCS&azlRqeQ;j<Z1Yb}tQqK^9JAkWcb>PN
zY5D*A`A=FsFn+Req5Hh}%=OPMzg;gQ&2QI3vUz5_-sIms|4FO+mf?lSShiJND<?jf
zssC5Gxi_c($$Cw3f6RJP-GAfOo35M~8(nG$Hn08uZ{*MQ;R9{%a@!gEU)I|#m-o?c
z=6cQ9u2-AwdfVBq_Y1$h;W=QY{nvgg$a7v3{eS7nlLLKq-k<sYGunsmhIrbW+5i4e
ztXGbxz)b!BzrEfV%Z=BLjjyV!!A$jQet*T~O!NP5uQ#6UdfmvyvQ6wivt4id|M&H>
zvffq>9C63TO&ve+Ed)FH+RWoG*2@m|jSLNM?)M<V^|no}_k9oU&&jqUe%s($`DgI^
z9L^hxzpOuxvF9JP^8=F^|CE^igUBzN@_j9Z_c4O663d6g=~UhIl*F%W`G&8(yHojY
z0^jGsmtuVVdSF~1jaayrb@-$n%6BvPz5u>l1HPaog;cd$-r<w;DdqbXQeG|oYrv=H
zaV}%Y;dAwU8+>0B|Hbm*`f5l0`sc)BRsXqsw}S5)@gL)({1>G!8^d=S_`W3mi{NYY
zzOsA_-*>@xt@tm3Pj#uS@m~zzo!~2q|04J_e-vNh*U|pF3w+m!{}?}G|2cd?M@x(L
zm%G7tz4)&IAD$mxp<31ET|NXV-@V|IHNC|~d`jZjlEWw0bCmBs@O@eQ*Cb!!@v8r%
z80EVkd|wg&F}~h^(x0*R)Z4NSpR||q{Rn)r_Ba>I7kxc4AH(-!@W~qFVgz5K{#=UT
z`w93qivJ?`wBM<9tDN{v)qgGM09xONz$a^fQ;e_opUYDHJL~XCsPa7wzORe_n&iv7
zd>w7!`22J5-6Z~NlCR|QIr@G9zHf;C8t|opq*SS9OZ>L#Kes<W3cj1ge~h28{~SIk
zNd5OK@X1>0Tr8jI&vk20^D%tC2A`~{F2?fJR5Ju6hfgk*slMNUPu5mfBKXW}En?o^
zn0TV<KexU&W$;{%_>b}K^~q37#2;CY?|q9c|0_BF$lqt@6>2k#SBqsb!SmQIXZT)N
z?}8;AW|MEdYJC6L==i`eJ~@m(2JK$~F3koR{3QR#(X1Eld*gKtX4?k;Zh@b~uXEzJ
zw9z&B-@|xD8RPPQFe3hS<QD#mBI370yy`dKGQ-JeAise%lX6}wc3`w`>F|IF8dx(h
zIw@geYe(_jLi{n=w|qkEph<16>k6NYE6HPQeP1L>TRPg_Oe}S730m9UN-XuC9A7rD
zKk+MphbD958Dca!GPI<RA{N*c3<Sp$zX5pu;^fc?#Obz<4k-_J1t#$zWseb!FG3G3
z$5oc#$Y9yBVSGPhWCS1h?!j}C<CEbP<{<pS{9ZMgdIndI_6O_9k5^c>Lr;5a3Z(?C
z?LljIXG_qBnV+q%)PJ;R@u7)htM!))*smtW<6uD#zF@ySv80!RcXV*j)|T3ZSp3t|
z)zv=N!5CNOzJ*xzq>UcDP_DJANXpgrC@{UWhqTXUiA9g#tBH;NWn*i<<luf(_y%Ip
zBk4DK`0K=~2cCg{peJ3`Bl%QM>YY{pNO(K3=;`m79NX8yEhv9~2jjg#ZS#p$&p{qO
zgqV8T&`5Y$Rckuv>_qPw40UW|I@3?#_nV1DM+bD=>R@?Z=5}J)M@9FY#H!<N58q3y
zI@-I9j`pCtts@wMkPe)Kpj`2b)O+EQs$Wp;U=gwC=|aF_VoATxKH~$&604pQh*eLI
zhx>?C58lWMznFg0g}<dE_q4ROw+O$Sdp^kY;veZx4-<<X%qP1ZC6@H!pT~%G410oD
z^vL!H7q_*w{E;}+W7^5|8}X3nF@6?)*;X9q_y>4|*y!o(YMXGd=vhOodQK)bdQkr9
z4qiJny6jBiP)}>n(k=UWTgtW<<3*3$nR<rU^q)z*8;IEYr!}?Z@m8<VkAAxqvC)I_
zUq&o?q`$qw!E!u(C2^=HEqY9Qwc+eb{IkH_ug+z9qX*@kPb_+*JufCUdQko)4wmvS
zB^Euh-xVAz<y=7=>S?1M=xW1bNg`;p=g<R%U(#=PLbbgF??5bi1n=VDQH(pg5{rKX
z?@lawr2p;dU>OJYCZ?WrD|Yf!(ApAowzL^BZhCNBNNm~%IKg>9!gL56dqyYQk`pHl
z2b>pVnO^O~ZvKwe^rH@z^YO=sP5WZv{u^RL4}U-DV7R;I55%E<bT;Tu2W?$;Z+w^2
zf1q~-v1#AdmedF_`&-+Di{YO&Zhn-1GO_3xni%Uj!@<&@r-*k(ylfE%Q4{`3`pW?P
z(>E$}fTVo#Cn?XM<1zNteai=u?_7BBQ3oe6sYuRCqD5M^I<Z<GssBre4IRe2m%A9e
z31an62adS#4-PG@U7cnCt@26#K9}jOe>zj=6C3}a{udKV`?R<8OkUz(8E-#9tkwIJ
zi{Z}#vFgEb7kWBV^VJcwqbtlLaF68B=!#t5IA%_vZxAvtJ~TOj9VY16<=s~QE7>6n
z_t^QU+GTqr2lj&dVC<yK{J~k7{adDGdnD(|*S7FQDa<WuS^^&$?He9jz84w=Q3De-
zEuxdT-IJI@DcB>~H$L8Xvb4?wCQhTv=NdeBVqh4zs*@+J9>XgxhKKN;h`!;u>vm5j
z2gk=olB-8YG2faC5}6h5p6r`QP9k>qJ(9cOe}fq(HNu~RsFfVSmgmOD`X#x9)kv?I
zh9w7*M;>+1!AB(zJkEN=jvt5jSpP|T$#|0W@KMBUuXgwx?bXtbV_j>iv)#-ERwAz%
z4`J_MdiJ06-NbhPO1IrlEa`E0S^9v35fnT`Z2C`ITWi_D6Dx-%9w83<59Vvo(}^p6
zEuCF0_^yet;JiUQpoPI+Izboj{ii<+j7Ri`p^2W!)B9|H*qMIQ>*pozSLb-~df!^F
zzvmgR3t1bU6mSHRdE2*H9``~$9o!=so)3omKSHoy_2Eb(AMiRthRbCGxq;DT$s><Q
z&K+C2(hQbizTLUUB4G05+yKVW-3E6{&OhRy<oLj#dDn(XzZ+<9FMeQD-kY<V*|AKd
zNm-lBAScPb(B4P=I6V!wW3V`=w8Xw2SkJd*Rln?KvW*P&+drItWs;3us}-}QaD<lh
z*D;OyS?!+vw!Ng=OS&|%&L27)Y|aaaW%rQzQO3bi#DT=zzuM#sEfu6Y(>V6%5ktz6
z{IXrn^xW-&q74qpc#cFiobCrxaLO`RdDeG&+Ul*pek0dg>c4MLW_soMDh@pItZ(q`
zdLD#gv*Gsd>c2^*SDqWeb0>H*5j?hij2`XAAyKc#a{;c?$g{Omu{_aw3LZ}$Jo0Sq
zbS#h2Q@0+w?D1R(9(lGlxTW4dwqDVCl1#7uxd=S+Y;7i%$LL9ij^Q7@9?!+#xkvmH
z%M-1q;PHF}Jo1e0bS#h2Q+NHj?D2dQJo1cgaBGwPVS4q?CE$@~bThF$Mo)X4e>|R#
zfk&Rvor>VGd1L(J@vH~W{o<bp9;ZLot;a5VJRb+o55+&<srOGKJxQil|9k>GKN9~$
z@Hl$v?jQDgJePv!0r5{PPmF&&o=<}3$Ksz@9%-*mq^jM@9?xY+@gR8ex7GWn(f(Cr
zdiBqz!1ELElwx^|p1Sd;;_-YMJU<0b;`ZkBq?lgy6u|Qkc(SoPMo-=Nlk<2k2hV?t
ze`0x}+pFmDTmhbk#Xqq;Mo-=NQ}KAN1kcaJKi{qQPow^vVtVz@XTbAw@lOO#wLf>(
zjXyb$=d<7`i+>_`8tEx|JXe9|7vi4?9!F2z_*3zCJ_nvh#6Nd5=^v(7|7-xyqvD?k
z9;>H1?{W9AoX7Kd@cdHz6TxF%M-<>0Ak07ft?2Q50X)AF|3vUK(o^wxt_IIz;-5R~
z{Ug^aFyq2al_2TsNin_p=ZoO^wfHB3r;(nV$8!yM9vA;a@K`<3@0TcgJYNFOZ^S<l
zJdN~JJf3U8^IP%H_u~B1F6VhjRlB8_Uj0)<iYLTB5j>6b<UF41!1FutPXv$kPt5ZL
z9?$jQc~bO@)z_<`9;VlN-2k57i=GG`s|RC7{cF{NqQ~=P@caQhr3juzdMX~z*D|=L
z2%bdF*6Un;KlTqeFQK03=Vh4wGV<-r<{TL9IetF=?&b3Z882SXAN71idx4w(od}GJ
z-&Fp=@t#ADnnixOJ_pw}UH<$eGGWWT!F~rJqxGksGPHNo>iOCI#Cm@A0Pz;!l6HK^
z!MLxHe%QfFhnBbhoLH}abm1l%{%=jCv5yVe@0$vb@-I2bwzFK1k?<3VRZl;$rXM60
zJ<HpBCRaIF+G51PK}UCbj9B$_<9yNRX`>#-EC1ufs^>{!bN-0>KJBEJ^W(n|tAC#L
z@N*u1o><RcQ@B24^t4C&XWe+!KT_Y*iA9fGFF%`D^`Aqmdd~Ck`3{DE+AknhJ#y32
z=;=^Bs3op{2!GGy+F;H^RgbjS*2LOg+Y+mu*AT0o?LEAMhj$`YJ-AX1J?8x0_Mi6J
z`yuO@-dxXv#><FBk36q(C9&v{?J8o?Biq$3M*CjlV7WqiEwSp6>%pQ2UviA>&vPeh
z{j(>r`X^1S{^=kVJ?8p{gE8#p<~dmO9zd*ma1RZ7aH~l5Fy7R+t2MQeSo|a7=gq{T
z2m5c^txo#(RL|t?#G*&~*If>lEO&eOUSidg#{CV^)2;nywe<(}v@9ex?S=9e5sMz&
z4w_i(q?huKCD!&j!NVtdxQAHvq^zD6yads<7vqf{l>Z>H=#ln*m{{%~$Z_vcH$8%W
zO{{u;=i%RbxI#=l@`j2QT%%5RVaJynKtqAtl^k0udc^-L)>QkCls`gj^dQXyv1vcx
zwZy9D6c3-~;WLO;5ALM3$eIEfe)`ZqM2Gad3e$;SWPRG7iABdK(mqQp>17;vo>+94
z@qMkWhvKaqEd6c{vFb=$9p=WTRRSVx%0I>QqDT1OM=W{-pGU0eFCf<PKjPpqEV#JD
z!BXG##Ht6KSo~scR9ijOek|oB*44JttBFOAJT$#MvF>L(5sMyK@9{<lWBwVu$-%<E
zD>3!7b;^BV+}qH@gtq^gOou5GUVNA8Jgb_o)7-$oSS9m^7Fi60t$f1X=6<q2pRuv<
zCG%^4d(^?A>(|8U=O>8O&lL|pMXdGcPGdzx3P0QdO<|{$`^n-D;gfb)akArAgpClZ
zo(Tub_`a4{^2>HAvFMR)%EfJM?Pn2(dfK3+#b|12Pq&);MN&_#-!n|F{q8wp?RQ(A
zV)ZJ1F|p=<8L{XY?CP0JI9Tq-ZbKaEX@{0p)swE%b1u`Xp7V*d{EHoo{jB|C#G+@Y
zt7rUDVr{R2hp!+G^>k1V&H~!nx=nkz{@L!-YI{le4#c8I%G<@kGC$jmSpD->V$maC
zHb^-*=uEe?5{G(ZRl4m%Xs@;!|44l|FunK(<*&YmSoBDLxxvAr_eNsXa}%-pXWGNJ
z5QloYs0ZU$ds~aqvwCu{OZcR|U8hz3Bgfx)#G)q`Oa==ayu4@m@S((#U$(=ERZrHz
zQs1M9Lp|NpW7^9uAkp@MKht+Gz4%A;+)b?g=Y9|Wm{|Su5V7cy^2-jE^(BuGQ%^hh
z^-fuY)YjQu+g`Y9v*h$z|C~sydip)QoY?rMt*v7fvFMTGXwJbhe;6kY^~k<1`@GyF
zZcF)k+8<|n_0N;Us^@73qy8O#B^EtxsQ<IXqDRUP&am~<_Im+ws7Lm7(Su`6TU%$%
z{)P7HSjY6L=X7GxBjum%V5$H6iA9f$FXs_!`FRgtL>%goeM|JndLK+KWi6f4e>&%!
zS#2*Ve`{jVBlDMS9W3R)mRR)k<$~1q#G*&Gq=(-~9O@A*ZCLGN`cK{d-m#wP#XnO2
z%ZNpflz%0$9xp%V#7p^KAXYtJ@^Fz@^*~1l-pYjIa3@X$<mlzh->}|f?o`!3QvROA
zroB*J+QXg1TK>Ksp6B5Mh(kR%mO_twS>DcXWpy9=lFZqp{FTA}E?A0I#ncARzb2&i
zo1PCm$?{bH)5K=~L;k;d@&6z;^^p$ro@)PZZ|#|U0kP`GdM?p#=6&kXu)l$S9n(wu
z%62-j>N%TO^vL-Bem5TFe~4K0NdL$?Sp0Jlai|CV-|E3qQns+GXU<vG_7y!_6RVzW
zi8=4<c&!uPp6;1^J+bJK@^^G_(AnOVB&MD;&KG3u2<~xr;KH)>H(S2?e?8Nye=Z|7
zdN80|Ni2G#{-1N=rTi}tYybO_i#yx9i^QQG?2DoYYrOH|{dzrf&vyKS^7kYbJ?$7E
z)5M}j%I|dI5fto0Z1mv!v-`WavooC`4)tJ16g}x6jdhcr9+bb4>2-X#nOO8l`L{Y)
z%Kxs1?;=+J-0R`{h(kRcMo%Z^lxggQvI4kzeuwKf3*TGykCeZN*tFMZ(6X3V+80Y>
zgLgVu`r*5XMUUV<5BC#?dOD3BE<nN=jkpNsHo`CEKgjgrADIt4Oe}hY|50LTFTua|
z;(tr5dVcR<93MI>#G#%p=t1X6VF6@af9~vVTk*cCe?-p+vFe#1Ry}LI_)~~Yd!hU@
z94!5Ria6BMZS_chwntxQ|LW{+e}?JBKNw7xK1Zy2wmiqCS3NH#HtmV>UrH=`+B<tD
zU+!S>e}Y)`VC*sek^UT^r{i3vS3TzwtDcJ;EdKwPgQfnL5{n)w|I-eZ_Pv}q)Pt3_
zXfN5vFc~+!CtQylbar=c_kOp%hJzi5wf?&hOMh#}`Oj_+mh$%?HtmHpEgnu2hk9C#
z9_`Pz3qiLT4?DZNHZZ;T2lel|hFJ8-y!-|SOOhLjCBM}FCSuVec-q6a5K~V(o)(e0
ziJ4Ef;o_;xm!w?blk>f<bF2Q5Bl0|A;~%71;Ne4wO??o5II;RC>*1q_Lp>OOj2@Z4
zVTrNm(V0Ndg;w1E-QIkl#oSNP=cP7vec(Bkr~0@2fUTe67Za=gmk}HN@JqtM{XHvl
z+YpEPF<u${IDcu!<C^aLQ`+NPrkC~=d_J+1FZf~)e~ei4^!Kb9zm!<?h+hg0hJRYG
zAP)6le83&BuGa3L4a<YgfG*`|{kHpHwS7Su>_BY%f%@)3tmW_KU`f9RvG%tX52uN#
zC*6+m6K`lU{XgA>!=8-i)FZk!FunK(*OS*?Lo9lvzBf2n%D<6V>vI#a_P1#lcct2H
zArAf1Wqcz(f`8J+Khp1QeJ9s;omcChdBoaY3y5|6Jk*PS8?pN52oGn8RSzZsG8cB{
z=NPq}_PT@V&G>-&-%YG~?su^G=f}iSAMx))#Ku2p-?D>Sx>D&!h(kRsMvs|Oc4AkP
z=F<KncFBjT{t<j4vFhn3*7BD-@vCsYIZP~iFr2Q;Iav6{iB%8!vG^x?zTJ{aKhE^x
zA35GVNi2FWu5~|6EP7U9B>AfoFXjD%SlcT&-_}p@3y4EK*te{n`uVo-tz&x8BmL@h
zVkut+va^Y`|C~##{yUFY%fG<G7ZOvCdAd*@vcnkEjp0X{3r|sWnekli56t;+wY}u*
zYHMO`pKXa%&vwMBXM1ARvy+E+CJyzmt<bM=z0~Yp(#&@J!HV+rOfUYC@XLr*&y~cg
zX9Kb7xtdt?Nc&#vV5#qQ#Hyzq&;6N)5ao#}JHM^z&vP%R`bXMtPh!=RCf4%1oOo&9
z{fLbowC{We3*Q1_?LW9=YxJ0#7+tvQBnJ$nv;Cv9Ya`QZf4-U6#G}5qI#}v^2eIfA
zf8I^3?RB4r?<WrR;J9o0kNnfo(PdhUdPM)ieAPcv-$le$Z%eS)!IFLnvHI&oV%5{<
z;eO(<y)f6O9=YOfTTCvvclUHlePvGMP6#qd=?J(kWB$cHJ|<UjC8b>r{v7#tX1vZV
z^nSt%FSPa3czgG_$I>9rE4+9=e@n2dk)!$L8789^=T25bEx(`tB9EWtS7~_uto(ld
z%f0;e>du6DW>OW%BD%QzJ6u$&kL9~GW`BvxzqglPGj?^z)m7UFa&adv{{}C=S3YR$
z{qwl|H+uQ2?XODY>CyQ5<Eh|IbA5^Z!_6fX(2fvS{=pa5`merxsX~1D?ZJNgNe9w}
zsaih$r^N?dJkODPW3u^r^nC$2PtQEQhf@2y&0U{AE`GoL?R!k);{E>U^Y1sy^7o%*
z{{807PXGS1%>N(%|L^+$&9m%3nOWxFbC&rrI+*teZ|?rLtCxT7!?oj;zyGfId2Rfz
z-u}Pg7q#(z{_TEQ8^6Dof5~ID@qYgGkJrZUm#MA)gTJkf_wz4&vNk^Bm4EIZYUBO<
zcl@z7{tR#b@A^}1yq|yFUuxrb-L+Q##%F5d{ru1Ty*A!&{}s>G#{2m<JYO5{xBuf?
z*!7kE@k9I9k{8y-`|ZDetJ-+K{0HaM#{2ER-OFm@{ru-{T^sMW{~fQWjra4<*`_wW
zuKl^bG_L-4>{y%MZ~r+v*T(zhFWjXz-f#cuH`m5%{u1ApIA?BcJdQtR)AmX70J}5S
z$CMtn>61URHb-0hxy3=*+G&O-7~V^pWqg4){|lSGOq+Vd;wjn^ZSqkYo}*3t(&Ap)
zB5m+18=j>t&{k+OkJ<P<ZJ9RpYa2dATcVwQ+=eH9V{MW+M_YcvhNphV{IrSRTii=q
zqz(RH!?Uyn+6rx^V&n6)W!lsq$xoYl*5VRz@+pgRw9~YSr)_vIZIL$klMT<(7HBK9
znLpe3JZ+ga^%omHMO&gx{?&%(Xs2lt&)D!@+9GZ6HyfU%EznkIGk>@7dD=4Vl-8Rz
z`45&$J58H-j^)x8X@h^VT-pL{g*Nj%%cU*Trh>WVy8aYxi8i@~4bRa|(<WYE!+U9q
zw854(JWE@kt<Yv(Xyfy=W!lt>Z1@yyi8i^F4bRa|(<WYQ!+U9qw80!3o~13&R%kOX
zvGIA@GHvRmHhhY<M4Nn>4bRa|(<c7QhWFAIX@jk8c$T)Zjl~7Uw3(ON@H}mqHuVbf
zzS7!2^V4Rwwc%yjsaIK?e6_VX+G*OvYixKgZIL#3tqsr87HBK9neA+Rp0-SzdYuiQ
zqAe+=?h<V>VbkSkr)d+cSMv3Yr=6xvY;VJRX^XVM8*F%%wm@5<&Fo;~^R#8!)Q&cM
zinc_X+{uRLXs2ltNgLivTcizkw&7XY0&Rsh^F|w=r|o^S#YNg+SLUZJ&{k+OyD>j)
znKm`ohELI!Xp_6!@Eq+lZQ?C9yqC5}8@$zqXK9OUhb;S5p8YAs{#0PP5^;q%vxnu$
z)0Sydd)n|R+7fMYFB_htou*CfZNqzMi?l(?hG%IDv=!P+i;XYSH3o}w+$Cf{Vk
zbF|a63ARUGd1zCu<e@FmCe!4hou*B++3;T4B5lxa!?U!R4vX`&W!hAy4WFV-c3GUG
z4c3r{wm@5<O?2D%UfLpUu#XKd&{k+O`;w2i_e_iP3@_8B_Oszrv?bc)JR6>)ou*Cf
zZ^L_Oi?qQ3HatsPpsmnmGB!R>Tc%CTx8YN?CEDbHHatf=O`BL?!+U9qw823(JWE@k
zt<WY9B_HiHZDJw$Xp6MLVdSGN&{k+OZ?o}v+A?kGa2q~FTcS<A-G=9Ar)d+5Y&c%7
zV~)o;+G*OvY8&25TU5+^McUv9n=VURpsmnOjoSDUZE}p|(N5DQayC43q_uh4GHohr
z!>4FVw8?jnk9L|iag+`3r7hA1i*0z8wm@5<%^Ypx^R#8!)G;=Einc_XJl2NiXs2lt
z@3i5)v_-XjHatNaEU`FCTcEAbW{$J*dD=2<>UbMIMO&gxo?ydsw9~YSciHe>+9GXm
zq7Bc|7HBK9nRnaxJZ+ga)nmh_Xs3J0uk~6=UfKd}g*MYqUfMEkY8iQHOSH)W8=j+`
zrcDgm@XW#Fqb<{>4j~_HiME&dinPIUn=VURpsmnmR*;{zOq&`aKW&LNxzdK`Xs2lt
zt892LZILz@w&7XY0&RshGh*ZOw9_YBoH&Jiv_;zBRPxbQc%GQy`C&oBX)~vhpSDb!
zI^Bj((UxeFXV~x@?KExTOdDRL4W=y4(iUhdw3+wV_&jZyHg%Q_pQ0_%CeOCvIofI3
z#CvUcFKv-Fc%Kc=(iUhdw3%~ke4e&Un|i+upQ0_%CeO9uIofI3#0PA6FKv-F_@E8X
z(iUhdw3+j4e4e&UoBEIqpQ0_%CeOFwIofI3#D{HoFKv-FxWI;IX$!O!+DzWY=V{Bd
zsS9oR6m5w%d65mz(N5DQF1F#lv_;zBBQ`usTcEAbW<F};^R#8!)Fn22inc_X{Fn{T
z(N5DQ*4yx2+9GZ6aT}hcEznkIGoP^WdD=2<>QWm%MO&gxe$s~LXs2ltm)Y=M+9GZ6
zDI1=pEznkIGoQBcdD=2<s$j#XXiK!o%WZg$cA7SEg$?heEz$;8+VCuGfwn@M`HYRv
z)0SydpS9srv?bc)RW>|FJ58JToDJ`#Ez$-XY<QNoKwF{BeBQ?AY0I>!FWB%Y+WcCJ
z%e1L=jHfNpCQr8EIocBQC$DBW?KExTi#EK(>rKfgc^!mynzoneinPHsHeZ&uKwF_r
z++^c>X^XVMH<*vMK--(M;YHfuB#SGI&)jTrp0-Sz`lbz^qD|dlap87rGvBqgOq=<V
z#d+E?ZR%PZK1ExiO%@qW-V$x{P8*)1ou*BE&xZHX7HNaKY<QNoK%4Bf;T7V{_bo2d
zrtY>l$8@D0i+gz;u}E8JCl76@)8Z`sSWth`X2xxNp0-Szny}$hv?ba+^;GEBJpI{w
zolRe)4X$Uov<2D<ZHE0fa|6q#Ez_pH%<^eVw8^j7@Eq+lZQ`poyqC5}8{BBav$O@;
z3T<YijnC7TX{Wws!%MWuuUnj>omSgr!>4&2xy1gL{1(&GPSYl)ZFnzjkv90Y4bRdR
zXe+dtTWoxuwoIG4)rL>emS~gTvEez|Y1+hXHoTX%NE@&{v$Rv}ZzbCDh|QN8rKMkT
z+%KkS6Wad2<#kco0&Rsh!|U96+T=at?FxGE=Ebqgd7nUyyfxpvrqld`cQ?wLP0iaN
z<&TM6Uq8@hy|Wcu<^}xbovn3mdmS5HGcYc%fW%YSc+vgn;Lvg-={glgTdQx`P``QO
z{~k&GYc>jnqfVlwo*Wp-4fjo=?viL|bZBxcHwm@j%P}V>dL$-XZ!s~13<G#S3*X0F
z{&Ma8Hoc#t_iMD8dcUb}yl*L9P%(jm&GSEg{HF4UFEYk!H!at0hC=z5_D$dervDm#
z%T>#7Uf<g{Y74;gOS)<K&FCkWDM0ft>l+=KSkb)x`4r;%$8zCyzY}Oxe3ft6guJYj
zp?F$KUcRR5{KE2APxiC;&HR4v^5L)oP538D|G=o>@++Xh<~;}BxUshX4GxV~9XQMK
z$>o=?!Dd~UtW%VAf9gMcYh+X%D9^X)rtxw8BdY$EtFCxcf^-^uKqI($V{QHMRSkSg
zquPJ&@#5>s=LbKd%C}s8`3&W{Jb7KaTmR+b0|TpU@uvU0*o*hepWtFf^^e<s4Hqiv
z@~^<tSwjxLu9MaLK7VT`9>Z(t?$mgf-*AB+uX707Z^i1-<$dEr>cT+2$L*Hy^K1R0
z{1<ZhF04k>?SJy{8D88S3Ybm(s$9E;^&1)w_fFG3&wahN|M>NL`AxO)uKx_zZh`sY
zr}}pYZ#NIcqyPKki_b5w^%eg{`Hx(__cS=LUmBIKHMX>9`FIcdiq#lHwE^)ANGIMj
zoYHu|d=zgsjd$x;<*NE`bRdTcR`;)+zftQS<=6EL`Y@03kB*J69atX93zA;>%LWGf
zRu4}Gd42zv$KN+R*@I(pZhWGp=J@&TEiBOvFki;E7cao?9CXg?H?6+<_aj@sv-r@Q
zk9;4-PkAoF-|q)cE`mq)E4;PAd@(sB;2(M9yU^kJDE>YdJoyM7Y5%zQEMJU#A8~j-
zg}*-lo?--#Jf9{^mQVhwZl6TH%N(8?v5@qG;3-A$XnuT)wx`Nq|N08@eHB0T&o?r7
zmkoF-<k9zXOKRm&YZ<Elil1fwcE=~3U&YOzCQcq58$Ed>_Wg206S;xm;rRE@O|BT2
z=vzI$I_~@I6PU~{?E^#1_e<ITWZu{kIL9|#zk4b^ATT~KfmtIyqSRx)Rw17`n0L(H
z+%rxO%tr%sf6#KYzQcRqiwSFcM&si9$NKQmg#LlJc<w)Q!}zMZYyJx3zh(E2nn(bR
z*Znyz9=oBr@8ieMXF!}k`qnNxXx-%A(cU^eIC7Ku#~Tx<e&4^UQ{#2KDU|GYE{fC9
z<C(UPpMOD<>pPe4)z?3aLbd$tk&6#o0#fBa?C6E^U5UdqUj47}`u2T|U)?{^gLyjM
z>}U1c&z<!5X}q65x0jsc<i<6A!CuE4eaN0&VfmADcU!#M9)o*nyq~`Z@<oHj<9(I$
z4|Vhpj*XcUL5=tG<I5lB>m3?z^6zDy>M+M+d*U^BvBvxPd-(kjjhFn#IQdbvp*0hG
z#6E9;s!fcq8C&b*Kk%JSetf!3{1fg!e))@!UYwkpTzs&Tf5Chwzy0b+|Cq-6`KA8S
z0W==^4?5h*kMEx3FoliPj}OP0CUEi}dZ3g4?QfS)rwoPThhP5q8WfK&syO)<y5-9!
zTjYx>8t>=dExB9rw9}F&X*~2FcCb_a!0xsMHQvv^#-3kKXuRY<!s$QBgLm8GrO9Mc
z<Nf^0dj`kG2ZokwyvgsBFRh2!!H~xL`IoN51rU6LLgSJD2)F-&8fOm!<2s&e{%IZ$
zmad#wz0`c*BDH%m7#Q@Q$I$V?w_09sTjTLIdO)m>$KyQjh~qc)ar`mG^0a)dkB%qG
zqw$jecqhLfKZE?)g|o}Q_|VzqKW6dl@*nT+&zq{h9j%X;@A&WdMaZQ7wcndM;z*76
z{U={rks(dv;s4_oIr{BrJ+wyS{rn5&FFsg4$fEI*-yP2mJ|el>#BRwj-p}8^qHp|-
zdsN3G^Zs9_e1E+6^ZVmH%AYUOS*^eQ*i&D4yz%q*jM@Ri(SM-({$k%hetzFSD1X6x
zr+nW(etzFS$bZn`PJZ7%e*VLbKF$odj{ZXrbn+jaJb1}rIB00RpI<(OC7-I&c$B};
zE#E)h`S}MioZ}Nc;dqVwhaK#cFY_Boq5OV+nVjyv`|ja*i~L6%@8nM+H?Eq+9?ygw
z|L>vv$e&&4<UisVdqmQBzkGYT*W>p8#qM}4NAqNEDE4?L-yK49IA1~e$1Fzqdj8oz
zY(Fd%SAG&{W$W9*T1mGL9$q~(gHLwgye7x-v;Mp$dK}L({rFurKJ|TTbF?YOC!BFv
z%FCm?+fZJS>8u+07H<&9b$L8VY`&$Y;c81vKge&XRkXYZY`F#6X<Dt{g(&ZKlvifB
z@=1AvdkrpwNAUIFK72va+kR|$$N6EYB<IV$v`NMnh>x?UGrP|xP2Bmwc<iYF#H{V>
z-&Fp-8J-Uedh>O&b7Dp(evvKa_&ZnU0|Dm&)3kQ<mc1Bz0d_W>4_s*Jf+Xw*aFKL4
zgHPnZ-vRxCzszRZ7lRo5jfpj;6y8?m`Pr%r-XD&^V4Ao<Te{n}M{tj|N!ko;k?~qT
z)vx2T7N_-#Yai$Q6;tTJWt-xgZM^P}xqe%(jkLisix<-7X&<Cb4%qk=v_;xyXfuO0
z{#@D;?RLv;crWb++6ryg3L8H~dk1Y|$c8VWEzmwrn_6k(*U?VX&RJ!{v$X4J%d~Tc
zZG4V)BW*BZ!xz%#X&<Cbj@tMYv_;xy)ZT03wS9CvPVs(Lk@4CeyyuY>+J~y8^DM9z
zuzkL`{^b(L^VN#&Q_?{oyNElVQGGt2mREgH-hEoTI$z#Y{_6AERVH~KgYw7aSN~LV
zxwWafN3L(lC7qe_sXv547_0o^>>$8UG9&*q)~DVEnLM12_v|#QkI$$3@=W`=^O-pR
zX?X*y`+NFFrKTe_U+>fSP31=}$!uv*o$G!O-M_>BLqg@3j4N@W4&N5V*D$Sr-T91P
zzO<)I_%y%dQUNP-cwc5)yL@S|P2>Iifj6J2mT&8S{{4;hf69y3@+G(2Z^KLBFa=sL
zKdyZLeuU#ck#D{)i0%+OUzGh*Zb&FU^Bd<$`S#SIdOyOi|GFR6)=%oc%6VVM*<QS#
z|3WX`(Z33}1MN$r@FEn|=jWIG2cM<v(Re9;a;>zWd<}%xqthMj8t>=#+gEaF`{Q#H
z_()rA{rvnRUVOEDX~7^k!;AOx%PX5@im&>#e90~Tm-cP+z747WBtDg^`5pfm`O?1b
z{UpD9f4pFR<2)%}_~m-P^85K^|B?8*@d@u*wPS$YfAAjB03Z8ZuJb8Pf9kv?u`@;C
z{f0BoMN#;8Bfnovn-0eDDd>T){r&juIDb|B<KBGnjedOO<^F2@mA}aHlSWh!yhCoR
zVtWIA3ET(y3Vt7g^DA6WGSmB2iLcoHCilBA_~3q&-IRAD9C$u>Q%vXb+LSiG=2M$x
zdbv*|e$5dp@7EFjVesZ7c%{wduCnCwx1z^e@_4@i-V4MJ5xkZU_p{}_K;D-4s;!6W
zO%bc!Z-O@u-YmmiKd4T%E_Ke~m2&eQ?=*NX1aB#pH`>1yhgZtYeAIr|R`uQj-iyGS
zxY6c!_2M;8W{c9BVtVyMmRNbe1Kx|ln~&f{{jk@qoERHj+T;2_{w{jFC6D)Z@P0(}
za=nGCH`LjJ_c?l7is?nKj9Xb^)q4lRKPq}-d87SXba<uwlE?c!@LnQ%zgF*Gt2dfA
z#q^<niB<3S!TT}r<|23}2bK@WSBghR`qp`F2=X3p(c`@byz9YRir`iMPOKS_LrBe5
zad@TN1lK#L-V%5}4&E%o-FCA?wqMqe0_2!p{2*K2<NX1`KLOri1h49~IcxtdIlSw7
z#&Q*p_lMxU6ub$p%W(B7Z&xj)K`EvW+l^TL@Bnx}3Eq4JukLRVv_a9~4O-IOC6D((
z@LmSqN(3+Vw|0C*2+I#AasFSsCAjWE_2$dAKdRoJg7;J4O)*`RAJiW%bBR2r-P%$e
z6_5A7!TV|9<$GRc%9~|+)te($y*~qQ0ldWs-aa%mG@3IJSF8M8a(D$-Jl-;RF9&bp
zk^21@Z<0f7%zn)D#t%4PWr<bqBglCLc=Hjw*00gLMTb|?l|0^Gg7-?%`)HJ2yiGYy
zFVm}kv&5?RG35M==#AiAhI1=iUy|JkXE?Q6(c>+7ypMzTv*4{n@M{0!;i!f<@k?7j
z(JLjVh*j@5Gx)X{cykP|zrG5(h~q_`>CJq{*)iv{ba(xdU903TpJFKe?4J+$^Hr@>
zB)_g@sw*OlKn3M&$~hky8=qXgeD%Nt*XSz0&NrJWKNoWO0-PB0*6fwxe0+Qi%Ok7H
zueA6uKCXP#=lWN<b_0vXV`WWs@gB|xbkq1a{`pK;!*99#%CQKP+Lpq*`<CY99*y#2
zo%p|O{VTcr@-;XZ8NiMu2UYA?VfkwZmW3+}XIcLV%T-rCgR%0=mhGOeP4<lp4Lj>Y
zXX<~|@A}_xN#kPE_{r7dtNvB>BbQ%44L0WwAL0J4<#YX&tjVugf2Hwp{Y&F@eVMM`
zFL?Z$%5T@)SMP6H?^OM>sr+_L{pRQAn)&elYP03r)$f~M{xVq~k3z%iX%wLT-BkU#
zczv_;bMg7+=QpdzH($S9{XL8PvOatk{xfT~XOmyock{Xin=jnIH`RXCHP!aLc1`D(
z_1BL6J}i~&Up*1tzxK;lto@6NpTqXQ*?V80rq}-YBQL(XrrGN6?hXcr$NFN+_vZ(8
z_E2sAZBQ4P2dO^4d`xkIZu~SpT>nh}FB`YV5RLcq`{h@U>{kD_GbmrasHFUQ%+~8t
z%f`Em0FB>N`8@cB{(r>`%6I+4gJ+mu51?`7Z}8st$YWxd{{}DK*MFxMZ;oluto>uu
zOp2@i@$=7p!p>t=9}iAp{<IhG=U?c>^Y|rC!kOb2ix2yUpI@%0w&DW{TK=BEQCl@W
z`n*KrmEY&kcr05UF^kB(1RAgX#=oAa@n&(Ssg%9$8vL%-e_Fo!UyF&&&jcF3sr*Z^
z{7+7E;`le2e{5{n5b%1fuWyt2&1}>#Y}WH*b~ZbU{3lNi^qK7Rf0O=`x%6!6Kim9#
z9BaU6Xkz}X{X_E4M!z{2%))=O&5vg;@TB4@qd@Mb=%)4CRR8;T&u^cmv0ZR;>i@6E
zk2TVE4A>0+OZl_OKhh^(mEHXM|BLc>ZSMZZ@;5vG=<1Pyaop3FF?@5{f42FtI^2}K
zxyO%xL4I7tpB%G2K==OY{BcwJKgyp~{_%l+JSc5kuvy2ie^LISkv`m{b*@*=bo`r?
zoBNV)Gmqc1&Od~!LbECVUyy$Slib|oDX#m%`QxVg&m_uc*3D}F+2&u<H?C8-%`V^6
zzozyx-+xT|=lW!cggrIjE;!Tv$N5_w!%Xv!;OXa+jrPr(zb&2Z{B@S|$H`fapX<E*
zpM9qB`9sl*M}B+k;r`zp?6Qm8S&H-j+`rZG`}teFcz^%t^5XkF{RevSe*UFFcSkGw
znCkQWck?Fl-{!^F^<OXEFTd0CPelJU&v&XmKmSpGuk|0+&JE$=Skr^$_`Sr7_wz6H
z;+Kxe)%5E5oA?Kd5<>s@`5*T3=f;NZ?!)%~l^<_!$AssH6)(PjXuN;ej^Jke@%`hM
zzv9{2`sc<l19bM^aX;QGf1MZapFf}F#rx&U2eP|xfl1qE^=fuN*8jOp<bUBmYW4g1
zU*W}T`TBfCI(Un@ajWIyai-x3J8~NTC1b|VfY0|QJ^tCAKVtuz-TCV*^UwDDbGG@-
z`DdHyfSYyv{FmgnBk<;zKa2cB^3sA?%)kEq^G~+UuKu&mZ+qWn&c88w&NhF|@t?y_
zI6mt56?cAz@@JcWR_Fi!xqe`-f7mwIjO#D*HoMvM|5@k%&-II0U4Qt`^^5;pznJay
zi`nM?cfS6C>nGt9dULLy&NjbX|C!bGm)Yi*>wk4uU#jOvn>zk${e6n?{Kh+f^XC^4
z=g%g|O!NQe`p@jHzx&s3{Px*Y`_DGNfBmP~^8M>SP3QkN@BjYwqh{;xUq5O#zkmIv
z`TVulpPJ3@Uw>*gzkmIx+5G<XpZ~A(``3?}Eq}K82XF&%Hs=qs%|9s*@#yWp%{c%0
z-_MWNPmkM6u$wdgn3UI7&mzBlx>GIy+50CyPT2RT#~puv;l=y;gYMQ;qxS)v_{v&-
zKmULi?;n41Uc8^bHNY3x8{EHm+{^Eee@}Vwe))g(;!j%Lx2&f+2nJht-_!N;!#^#q
ze^$A<(SJ|hME>`A@&5hSych46-xlBt@2-Em>;H4Mt!*Da|2AH{efqK5e_rdw`}ueF
z;{E*aPulg5pZ~H=<o~P}@8>Ui@wR--URRh=Ea(XE1!31e-u!j$t7`qT)ZG8mRhih0
zym-I-PA}e^d^p+Xd+~mL_@`6+qYw0&>n~3JDSnT7!;7p=JWgab-JdaEGV4V0e1A!9
zQ{{R2b)7)m^}zq>c+5AA{<!k%{PllKKjypt*80QiO`EwM{(riEY>FTAoyn7P1L6Hl
zc8?if@7@&u|8zX`W3k2f(9+eD1Md5jb^rcX_y642YsWjke#z;^@ekfs8xQ@t@v;8U
z|5M-d;=S>9$-TAle))5M*f>7-lg9CxpEZsT_??~5zj&$GFdo8C9p~~p>S4U6e=fh9
zuJM|$!23Prry7^Pf!`aC%fFD{{f>)IY-zu9qw#XQlYXP+Z<tdX@Asb-FRzXF_0Qe5
zar}eZHI6@b$HwuQU25akjSSb0pS#z_`}#NRRU7Z?&!!v4>vwzAe}4YQ_iLO#dthxm
z`v2(IvVopSJl7=cFy~M&-s?Xb53h~a@^w8y*E_s;4rm4TuO#t&{*lq|1mx^G21bYG
zxi{GUDDRy~b#(+=?qIPzS0me24wmOsUPioyM7Fi35)K~g>FU~sxU~b%?zgnG;Uh9Q
zK}dIYbmOg0U7f*DFle3!(&u*0WqN%t&iTap9-NDbHT@+Hmhvwp7CkAvBB$VBd9L^h
zV%5{uC3@OYSQ8SdXS*G3{YB66o@E0&5Q`ppR(Tg<;~$j2tAnNfZzUEzQhv(8;=fkn
zP*021(=Bhsitx_{rdR)5Lu~Y*{2PcxPa5_As)MEcn}|h^lt1m^TZmN;ro!T%mhP5l
zJzYCh{UiRLM=W}z{tFx|<sU+fS9k{UvdqJXRnH<1A4yC->2x~42^1uCwzrwi!+7=o
z9mL|FHq`%aV$mbz-|t|l{{zILNAORGRnNm7{yA}|r_Ja|2kDly(G$qK*?T5MkJNuj
zvg#ix|3qTZv%F_$wBNxpb`27Xo}fRN93mDyvW<9nj9B$_TK}|#dd#chTUtbql>a!>
zi+^N$lGyYg;HN$O7h<Ca@qZ&$J<quq?f*Qn>gnh({^<zyw05NBoW!&j?7E$+{s9b5
zCpLQU9hS3+rF?l+{W%Usk4$}#*yus|A9ipD+CNXMdeZHprxP!_XiayR2D9ah9%=tM
zZ>;JOyfv}t5xgz2_LtW<82!KH^~9n_+J8p}!~H>$IMieO(}@=}wL+SVh8Pp@Q$6dM
zUibIQh*i&(#FAdhyUM|G9(y&h@y}#1SadP+Tu&V8=@kF8h=W=>yIPGRskgS*++C{v
zk^1gQEPAB;G_j<Y`gRbjzxE{-J;Hl{hvyTk9!QY=y*1Su_MlEFSL?fx=}mt|eQzc<
z{Rd^-N^JC_zuZnNdX^3h2X_&Rp4L=%>%9&}P;eh{sHdBHs(U=X7t~_>BlTVQrmBCW
zev62;y%rOT9*IAeSp9PXvHGXS!Q$UOV%5`)<CpPIb&ogi&j_&4u>*tU#>Mu1M9*WB
z`Y3MMP7vOs#d(0v7i?(t->u(l>!beMme}|c$zDS&`eoam*z`BF(@qW+{W}v=zr4}M
z>TeI*SA@5zf7Ua7*#C*upH~uV`m2aVkCb;cvC)Hey4Jzs|LcfVPm8v1b;L63d4<2D
zBNfcuwYGitBsTs*ebdB}9>Y&h2eIjI=pXwMYx^GH;rYa>r`3!P>CSYuea#z6w7wge
zUi>4+otue`9@O_%VoiTLvG%vSh*i(M9=?xQ_1NRDnZM}~Z)`>1(jMW*fiGCNTdjW<
z5o>!bCRRPi604pQh*eLIhx>?CPn&5kGk;S(vODI4AMRfIAk%CAd6-!BJW8y39wSyg
zPY|2_)7IYl2M43NEq^2q^_cOhErt18s7J>K>2E9M*7|3JSoKU0tDZH)qDR{IRASL1
zJNlUp7XJ4TtDY9CN9J#;C+yGSpJ$j}{qr2L=|8CNmb+K?e^j?+D`M665@KzymwPxt
zOg(8V-p2*r7WC@&cD#Dk9PeEJoXhmupU)>2Ju-i}*ue-2F7fckiA{SU{*xa56tViJ
z)9UF8I=hS-sTbaGJCt}!wVkA$Urns-w7rLSB-VbjGqL*R%^u#3SpAa5IF6|srrKRC
z=Adg{(Yta;^rYH?mIBj@Uoc<>pCwj3S9|zN#H!~yV%75%58p@}>cM_xrjqHFwxAEq
zBs&%6r=myxOue=0AL)1P#Hwds5ARQ`<<Ix<0tZV!KbTncbYh+jJ#9e>-dW1|NK0q?
zMy3<LNPoVWSaisTX>WC~=)av<bR<D^r-L!CNZ;k)6iy)SCRQCCm>)q$hpR*CpV_18
z5An~z#G<1Ezr%^OJ&y43QN*H4bR6s8)>Nu(39;&ElL@Su$EI7lIB%4!?+MejU|!PM
z(Gu*fO()Y6%nCYj0|9L)-#NsVOvt}j??bz(`M?f)R>uRW_b$X*uU$R-7GhJcmX7wl
z94sF?P7#Ool8G?tCBv1SpWFG1jyD^aUdO9zh_&C{K&*PcO04<6PAqyv|F=B+ZDQ4f
zL%!(28zs_RXjj=2!tqn`ckNYecfs?BRnG!qE&mV?A5N@#vK~H)n0ne<Fkiy>*$!PT
z9bIWNW=Xlqe+Sd6f9@t$J@*r9`UgDx5V7hhd-xGz^-r4}KkfMfI;DR-XUX2R{yC9Y
z_4E^~o<U+Qf0c(vJUm9Mdf*cH$DA+Nkzyx%olE&2XL|L|lf<g$X<|+P7h=`(tcRcT
z@bko~2OY=iNpr;5y;je<RIPtbCssXY6RVzch*i&d9zNg07Z8Vf?6}^FV~X`px>nDe
zmRdbq6RVzWiB->Qh*i(_9^S#jI}uY4-l$3cv{TO^wR+Ywy|&k7#H!~?V%2jMvFf?n
z!`FEDT4L3MBd~2R9pSUJdgiv)`e#pK)srSxJsrfVXI~G`^Y8)0p&oO*G4p@hUMJM*
z*~s+jpPPwQ&#lC&=XPS%bC-wj_VB&Lst4yG@Q<1QTRr?vi1IH?*ZOA>vFce&tm%&>
zRy`+p_(Tu)5UU=%0Tg=7{NL&stM$)=Ot1cVm{|2ZO00SwBUU|6c=$;V|AAQb;63a{
z59a??&ndNfR<zanXM|YwOc1M{HN>jtR1cr-;WLR<53Wjz9-04JJ@2)8wtJto8)&=E
zu{g!+0<WB7rUje4KCoxIt&i$Yd$@yG>(foF_1WLU8Di>Bcj9=5E{CICYgd;!{MPz&
zBhzd9-b}1|ZuRi(#9IEH9=^-NcN2$tFkYEmAO$f?tH<3;M{T<vOsqN%_wW(Ks^c9V
zUhLsxh(jHiCqakyvg+}l>C_*06RVE<iM2f*AXXg@dH7)u|C~6~f$<4C<ocZLUuW00
z&yvnsznn;{dip&)NUZa>A!5}x;^8r3>S^mhf5DBWuAsZkp0|9cR?p*1ul{+GSoJ*Z
z;lB{8p1%>Rp65LLJaMQ8=XKE2E$?16lOtY_R{yW-s`bz5#H#0PVlDq1V$J_SV%2kj
zhc6_ip7vJE*PutHZCx0n<OIBCK0BwoR?pVNs%KkbE&nydTK@JP-oe8=5vzajbpp|o
zf*{*}_`Yi8U(fW~UY8N8o-2tp{Z+)O=V}jM<Kb(GLp|usMvu<d_&zq}pSw@3fA%C+
zJ!xW1-{IkXiB(_5!v_+p9(<n$da!?qsLt-X`SnJoSO45hta@(s@a@E^?@nUXbB~8h
z#MG1SK!3&%=#0nKAIi6I-&+4HB33<%J$x*&mVcawPw?=G#GxL{XU!DS8;?s&r+)bn
zvFdop!#^Wd9l!AKqaOYhvFhl=Jj5ItY%dPS<0bpm`r|}m)zMF^{u?A#9jiP%;^8sk
zP>1zLbv)+v+jYE7IfvKx*3-`A^{|b!3wa&rLE22x&I7V<vi4;#d%qOdpk5gSf5gi=
z^LabI*1z}OhTB}#e~V17JZbPe1)fqYkM&=b!Twe8c-p}8G<Xsh#_5UXNin_ZX$Q}r
zz>{r~C+G2WfalNRpC);V9#1ED{v!TqlBeSFbb;ru;-8C}^bgakf4af*jQFQXo}9<C
z4|x72{%Mk@=<)0ep1+HKn&hc?Jo|y?S@F-sP5OuF)j#vV^AGV)lRVS>j-c}F51!}1
zQ)YPm{uSMy6D6x(c@6;2Kf#k?_)K|vJ)R7Bo(IoVlRO2FXFhlWoLfvc$y4@t4g}8@
za*lFelkLUy>YoMRc>#E`P4Y~6JO_biOYjt$<SBVP2ZQH@;Hfmpll(z#yB`9c7l9|k
z@ENze$8#unwgS&olRO2FXCZiA44&yGdCDHoVc?kqp5XrG+C9ni+U{=y&r8HVP4e`5
zJconlrQ)9^c?urS+rjfP@lTUHWsheOc>YWL!|(phwB3_Tul_j#JX?!@VtHhSQor?j
zJV%1(<>H?R9-BLQ94mM{S@66<{1d?=*T3b<&vJjqS5)?R-T|J3_~%FU{m15w){|s<
z_0Lh@k!vKG2p+4)S7LFm$Fmqb+lYT6cpB*`csxggXIt@41dsYBN>SP4IR-qh68}8V
zq<@%R{c|jMUM>EK;A!Nasr~jg=cCH=PVl@2JOzfk{aNc1#W(HoECJ7J!BdXlvH7FF
z+ZD{K)pHzpwgXR+;jSLlp;q;Y&3HV=gXeYN>5bqSm<ZP3tD(jZE{FV`_jpbK&+EZc
zjNrlbsrv7cld$aZybC<rgD2SE*0;W1cKr~uhbGDN+FmDu=MCV=MDW;5(e>)}c-{@3
z9l$dc!4u_Q6;$wedcd<Ic%~zGa4j2thL2>?owt=eo?h_mB>p*|N&hgt`lk;(N%2nv
zkChPPACG4lcy<>5MDR59kH<3ro;QkrB6#HDB38-4LG|6J?C}hOXBY8Lrb+)Wz4~W4
zc-|!biQsAIACG4Rc-}1jiQsAIACG4UJiCg2B6!l>4f>DAvl2YJiGSud=^v(7|EvPf
zT=7o?PecEBJj3AGUHlWl)6hR2&j@(lBL0csvClI{_vf<5GYX!!ihmBQ_m3=Ku&bTw
zznqq2diBp3c=izgMDYyDo*u2I*W<~7XHW4@6i)*^1&`+>@a!f2iR6)$Jkff}9?v*<
z_7?vvsP_-#<m~Dwxg$}#C7E9RGXb8I_$PwL=8V?U>+wv2r$zh|!P7`j!Q)vCo>uWs
z1P|`{w8Z+y<5>fqwD{+sCjG<o>YugXX%qiM@Yww##y=j<I`FiMe<FAq`p4rr89W{0
zp9r2tddeQpDd6c8{~X+;f0$nVb1Hbc#6J-{4f~JBa~gQM#XnIzjrK2(=XCJwBmRlv
zF@mDotL*Wd0iJ!uKZi8wAEsCToC%)J-{joPc%<Cy}_Jn>HiPs9G>@w^8-`-^`f
zcpB*`dpu`>=K%4~q4oZ0I6gDI`sZx$WW+yFJPq2b*W-CFc;<_LqIeqhACKpK;5ktI
z6UEa&Pub%+2RsYJKMU*q)2O|YOt1cVKX?uj|3vT@IWfnJUXSNo@Ek1uiQs9Zr{M8?
z06d3?e<FC?_mx;})ql$#&j-PCsQBlwdjHsdSk3GHm1KJL&w1ckDE^7y!SMpGzK-7C
zdp({Hf#)#sPXtdRJq3^FeDJ(Y{1d^`@c8BNd>A~3i+|o0@1H1LNv2o-TmYW8i+^Hy
zV*KOr<iWE@{1eM#*H5}_C4Uz@o(sWqg!m_xCt6S0<GBbtM~Z(AkN1zSMR1bo)jt=5
zCoBGm;Ia9m_4Il?9|6xh#6J-{QSGGK3Lei#!E==OCxWL@e=d7Gmw;!n_~-3S`iJS&
zKOY0n(c+&7o`(MMc-DjG81YX8PecEBJRb+ovErWyo<@4g9?vJh^G@;4q9*;r^y;5W
z!LvmC6T#C+Pp`-GN$?ye{)yme=pT>gGVmNP{)ymeq^Iohd<r}#h<}cV_fM3rB-5*Z
zJ`J9CiGN~wV)ieOrvRQ4#Xqq;G2@TNb2)h4E&hq+iPls0c&-3XkND@vc>h>`*Z!Mi
zdiBqh;OQ0rMDW<0(RzA4p3i`%Py7?X6V*<ts^IZ_7CcMEKM_2Q^priGtH9GQ{>e7!
zAEsCTd=5Oz#6Ph-G5+y*Hh^b9{1eL)(|<gk&x2=B{1eL)t*7kqd;vVm#Xs+e_m8|A
zMcYYjlIhhySA%DT_$QVpT2HUX^F{CsiGN~wV*KOrTmzn!;-6TaXgy_*=S$#OCH^@o
z-ak?OI?43vpKHN0EdGh*iPqEW@f5)`BL0cxiSduea~*g_#Xqq;(R#`r&-LIL6aOr3
z&Oa%pSO44qo*a0xu{_awraYdnXM*4)@Dv!nWT1NgMB7QN_GPitOs~%$>vv5;+sE>0
zQ4H?vJ^wtuvo5bX3v?Z1mhlrO4V&k9biLH23|D-ahd)is^-^tkN(zhXuyRXR(lTDw
zW67O??Uq_SvW`r)9f(bQOIOz}#Il}C9<Z8AtanG>>ft>-yf?Ax!E<iubcZ~F&-IXu
zH|3#>4aBPF8aEz#ZXgyt^1RnZV%2k#hi~@qw}?YMcrIJ?U=gCMY{Rp6)*qs$rK>&Y
z>aX?BJYx0F0%Fy(kXYO2a1Sr?@R7u!9z0juhQ-FAe;6<2Nq@V8Sl9jC?ZiWH*Zst*
z=RsoC^N@!h_VCY%Lp_~VPw1azRsW1m42>-z7CkHB!V{f%(bG>%J;N)Uc&X1SV%3-P
z@HnyRY3;(p8S;>ZvqDpN)IX0iz53@#VzyWJ)5M}j`tviyTK=;he$K(j5<E{F>glE)
zSx+k~<_D_&k@V|`MUPxhINggsn^^S7b&GR}MGqL0=efA0tNVNhi~b9ULp|8PjDK{0
zXS~#Z4N90Z==A5oU~+3><Dbc3bX#K4BinXPy!d&0VxtdFT_hb0Im2%xRy}y>hk2ML
zuD#YXy|&k7#J0U!TdpKF?S&6<Z6KEN@p$Ff)egoo^p<NJEPAdb4)tLFl9w~#N{Vf-
z<<<6*`pqR)J$ribX=2eM>AQ%v|Lp5v(KFA%qUQkOP!ApiHF~tY7%%k~{*A<{=VmYd
zR$|q2hZ8UF(z}aT+pFYY(endh>S=F7dv(e913$I5+XeKZL-j0NQSCnxzKB@k7Za<V
zC0_gq#MIN;>tNBdlsMFb{Y&(q|D^G_e$Dfs(w`q>dhrkLeoa11Z2AxS^P|L~N4Ce^
zc;F|9RZqpiqUR~%P!GmKqX+GV*YSjUwEd47vi{NfA5FY7pFf`dfqf6L+63d(UPf&6
zw{*8(>F`0%(hW|$`2T8REx+jD>xo1C7(b1EJmiMgd4&44{pPN$ZQnhKRZp5&<|#7%
zbP-GW6KIe99E^9jr1p0(k^~vzP!Gl{)8FtIzU+`gyuPV=HZpzK|A|%4t;C{7{BwsB
zFaG&HvHIse58qF$deEdsk38@T2iXUZ-TuF-)<27gRnKB#_0JL~US19PE@II$GB7&X
z=U_OXrJp#|gMKFdF>g?6@3I}v(eohFtA8FQRy~h8Sp4(2gT+5j603in^6;OCRZknn
z1=C*EL3llheaD5i*NWj<|BMh*PsaqY>>tv9);aN_=Tu_RBj<}#4jzER&LXBB>B8pW
z$f)BBD1v90-i!~x&k-9vt*MqRN2>d)=$S(-<;(Um4{z<^R}hDKTB%2mFN`<kf$v;m
z(IfML^S$_siFLeMPpo<_busi`=3se$$ft=_58i@c{iDYh#)}?VPq^J^wY_Bdzz)O`
zpTd8;5UZZKPCSwXZzVSRpeN;E@n<V>sHY8j@L*PCd$qK-ZeV(CuWN`!kMxflh;@A3
zNG#<`{7nu<Assh6So;6Bh(kT?)T8Y+X8ob%brI|SHP4G*K&*Ne604rW9V~hlIavI2
zByp%mUe+l7(e`4zw$~lRq9=*z=G{&_f`a>rwY?tn;vXVrdv%l@EdF_fSoL67HV@ZE
zwin7<lB@O~DgQ)b(JNa&vFcetY|2OcDi^~)BMuh-j1h-=&|dPeZ#aKryp*>b2cXA^
z#XsmyEl)b}Ih^o5O|0$p46(M?vksQ_dd|V(pXZ4~J$!j&*j^`9#}ny~OFcY5ENg3J
z)Ay38wV~C2uVH%QPw2V9;S>LCBv$=5dH7}rV?3Ss7P0EbYk}HR=2a@#+2zeMjMw(-
z8n12NdBmz`0kQZ~{Iif)+xKt}FY@q_#G#&Ut0z3)WW4IRgILFhyWM#B=YC?<^Pn3K
zjSms4zOswaVICo-o^(eW^vL;w_^GoxUo!cz+G?kX+VNs%;?@qi@{+>ignOFvIt;ba
zTuTv#SBzEX)9bzbml5lApeu;Azg|U*M^(&t_jzJ%|LcgwUuL`{wtD1cM`!GJhNq|n
zFI&;{b0=%-v!{n!iA4|k(_kmD=n>j|h(!;kf5CxHeo1l=vFd4uo_Y0xS|qJ(s%zTe
zk$T<c;qMWvpYA1AKmCAM{q$2W|9=y!uFl|${UdaxgVokQrv0I>kNCb7M(!r}=hK;9
z+E=!-h())|2hJfj{zTXZh)sEGg8mDMb$_^+Sao+ncfC8?j34EG>zp;!_8lHy9c)c(
z^rV8pZHPq=nlJYnVxtHB{dL6Z$DN6_opvFn9t?{S{S6lZHT`-o|79M&g4pQ6{`om#
z(KCeW0bd{%J+g;g@8p+-y<a9)J!sd6{)V?OV3U5Ox>9Saev<HZ5AQ>){p|o^?QaJX
ztDoNH<$pV|>gtZ{Z=Icim+uY_-%YIJ#Se(pUk?zAuF-*>zMl|_E*X-3K`iyg>v)4l
ziKz?kvW@6ZIB2M!maMDw(}~1d@1?}*r)9+Ir{!MyF=FLEiI{qDd160r^h)E^a<<*p
z42%blJNeOn2cIN1{y=|z%Ejo<e<l_^GGTazSoDv<VZq7O`pFpn0%FzE!SBt;T$%%X
z8tbO*{sxY9PJSukbYkI`P2We-dx*uah1nC&ANTKp>svBMSW~_4U{lzBs(<@dE;9(n
zKVO3TgM-0R<-=hpm=M0X&GWqu*)GKmmOl7P`CbaXN$^b{6UQgVKU}EwHqo~Y8rSE0
z5%^YvuQVCQ7hP}R+ctvlUifbf__D{<^T~DAxb?)_fKTeU2tW8QtnaPhTMNFzw`_kv
zbhyd*Ae&l??7zY%{<|JK=+I!#r26kW;9Cd2;&eS9-mz2vKmtjHPyF{T@VV`A8~9EJ
z-}JZZ`Bax$QW+zB;y=;XH<)f$eYb<}6!4X9X^wAO@JW3yK?fgl{P$h(oeIA4t<CdE
zeb<ApZ*XPU9(RE6H1Jiv6VGQV==~%5q`oI(0_plM1HRM27u+7lXZo|e>ab>e70|2k
z`yK`umv27!&H!KHyY+lE1RfARX@jz<Z?*pogYQi6CGV)`i)vS;6+U_8(Th>vA;*6M
z;F|(p>dppy7}S(Ut?<1*<O@>TUk(J{d%&0ZUL0S%|Goj=eF(py!PZvgTL8YZz?Z!%
zj!zZUTG4kixE}HPUmy6+24C;@>-pq>VvmOP&tC}NYp`$p^5wPd`*-lY7ks(9>-n@j
z)Vc~ydu(N&MRUiOgTVJb@J-!Q&(}tRIKIz-@6Y%R4Nipn$7=AM1HSyd^?b^s*4Ot5
z<g@)H-Kp(y8u;Fi@ItA1zDq;CYJYwo_|8Rm@xJEyK3U}pk8d9W-v<yr{eyTuIsbL*
zDt}9RTo&@RrnSDG0N)1@Ub;Vy&-CZcD85gHeAW714!-jcUjAVmU-a=q^j(6!XWOI7
zw*h<~0$=4v^?Z$vZ^HMr2)>QrJ0E<(kL&rgpY-(O+dAGRe1CgIsBcjHcOCdX48FvJ
z^?W#;j@^VWz(L)w@5SJ|0DQ@x)bpvnC{p3;3Hj1ps_$U%<-wQwX>)us)(huTX1uDl
z$06Xm5PX@3;`t_VeahD;e4?q_j91m;+X(nB0$=vO<N2cMEqpTGTfVBkLGWD+zN!6F
zb<f>e9@C#?DqXV)pN#iD-=X092>9{`#PLP<XW^6a{&%qM{9qyYJ_^1<rk=0S_$+)f
z-ap~-9R|Kjz*n4K&llx4WH+1e$$0t~GoDucw+4J41K;$4^?WIlGy3^q;gj*y_uuK@
zTaWP4f_lD9>67u)=Q{^{A4hokpg6wx_LcF}@>R#b^T8*&D+kB%p<;4_z}uw0GM@VS
zE(M?H4i2s7YcyU7pNyxLZ@5SI-z&f;x)TfQ`8MT08BcwEp9i1lP99dzCr|0cZK6-c
zQ=ji^;1k`cx7G7)N}r6UmajcL9$gQ<PlGRWcmqD1zxp~wpNyxLFE_6K`v~|7;LE<f
z0iRC>EPOJaTE6OdwG@1pgRgf{b9|x)bis4>d^McEl)!fd_;N?Y@kxKi{aJ4lJ~^j3
z0>6R5YJYwk_^t%s)RA#~I3MflpY-UA9vSa{Vds-c_21#(`waN<*?PW4<CWwWAMWGL
z??=J+S@0F!QP0<?zX+d<_p%Kwt&UgA!FLt-ibvJ+b=u^S{|cXs_p-TsZwKG!z&E|P
zo=^3e3m`UY_^<HEc)!2#UwfNixxTRod>g=5Iy#;&Bys+b{4(C}XWJv}FTLRVJow7T
zG|wmV`$w@4<NPh0Pd*F2FMzLdZ1a5E;n;C8e&~PU`OFdEyBd7Kk~luopWB?Onm?t!
zay<IJcRX4PzAu6=aa<gq&t$Ri$+#|?+rDRj?;3<BkFV!zG(HQT8P`3&_k-_C;7grQ
z&!_!GH29nF$+#}t(9+3py#Fxxu0?p}UG;qU=3eY3d@`=f=GOO<;430LdtyD`ru50U
zZu#27>m^r$?>dC{zB`UDdVV1K<alKHs{8L3z;`|Pay@Z;_I`oQS^Zb|<ai{TTi>sP
z?*{Nq_15!ks(s~nWcm8S{o@AkeHncDzIwjUf98DzHn09G`pke2<Ldh;_`U+Z!qR%a
za8ep^d=ox79?1s2oM0JW#=!Sg@D=;(`9ggrSM5JCmdbcP$6Oz(wr>u6H-c|^S@V3R
z&fx5G#^+Vw+X%kWK=XXg_#C#!N#Oe$_{xLvd=mrXJ+-y9$BycFFd2@gcY^Qh;HxZ;
z<1^z`r_U$X{^a;u^sY~SAAC1~FBppB^O-CbKDlQ1F>kzo0j?K+1AK{<^?aw9128Yh
zd$hv$Vf?bVo;@&F?awzu-_77luBzwL{vwPXow?R2*C%Cj`|~%!_f7DnhU@v-aZ@>N
zGxd$&y9RvU0$*mNo=^2faSES|f1ThQ8f*`*=d1wVG{Uo^&GAX!tL`7w<J%DUz74+K
zv3Ne*443y2)oj97-9JuVsr&Cr@ZAEwTrQrkhKNA;qUJ*r;JX!kQzymoMfYdntL`7w
z@%a?+eFuE`@i@LHb#`u5Jzn;Q_XExb-)-P4Ow{x3-=IH>zUuL^s_%o~yB&PR$$Gv<
z{YChqj+g7f_g(N!ude57Z*W0J_^QXts=iNy?+);l*3|Qb`Xa`A;j11m+r#TQSAp+N
z@Rip#;6q-26TYb9<yXP?J@8f5HP2T)URL|_wcxu8e8DNr^F<vmF9hHB!IwC-c|PZO
z8IJd_#dvx*_>!m9^Vu&cLA!bXK&=PI0e{N*M&D5N_`3!8?g3xw^m@LqeQT)<lCfFF
zEZO>&c82q-mx1qI@MX@3=Ns$k8@4&C{|cXu&wWdmhV|VFd?oN@&y44b(kXm$|46pJ
zrPcG_`_LZufv<O}Ilf!KqxE%<AHM?M55SjuPaL0&S6Cn5ZA)-&bOL_QV$SB?k9rJz
z_k(ZhtT;a0Z;m*=9S1Jq{0HVk?){2igYSpn%b#7(*XaEBc=A2x@jVW{AAzs%-g-W5
z2eodMh3{SXecsI1s^iOV!1n<7itnrE)0=oveBwK)?>~*c>hbNj;QKN7rq8M8Qyy85
z>2Ja(K75fmAM6f~zfXYgLGYE{AJ6A=2^K!7?*^~FzXRV-z*j!EdA@tV_g&i_;e7H*
z@ck5gl@Bz>C)&h+FEsU?7|`SS@4@#F_=5A|_@qB~*Y#)NlQ#aGSKmK?@4vy9_)r{Q
zl=rOf#D`be`iA2{1$+;KFL{1FU!(pk`ow>?czpi^-_O98`fxp;_7_=i;;Z~1xMV!|
zDQw^1_{r+Ozku)O;LBW4&xa39#%{tlfVsGA?)WkX<6jwk*?c`;sLy7r`B(U4Jhl3&
z`^T2x`vv%VFKnJq##75z-G7fneIEf|?xJ|U;o+X48gT&MfIgX@o^SkD9iOw{dlY<A
z7svDYT!Mv9uGjq@KX*KR2l##ozWhhx_|Tuz_3bNs(xwM^*Z+<J-><+|_-H+!eGid+
zZU|XxHsO=&VV&Oj(PHpD2EO7Y^?Z%mSNP<5(>?h0IoGd_2H&s2H~q1AzVZGQb?qVA
zq&*(B^-XDiJ_dY`gRitcp0B2&1i~ln@g#okdXQtm_Z#q)KOWB)b$k;((e(e=I}i9q
z%H!`Zy@TjS7wyiJ1D&tK0RqvhXn`cat$pV^3_e@fzN5oZLLdZUdIwQM5+J6NLRd&h
zNC8CeuyjH-y@&TV+8JrQdLFSGz5fpgqWkR4jGo=EXMgiN+q7D35~;^t;Jpss(D^o=
zxxU0B4JSo-dxQ4|c+Q139@-2q;!!>BFV~|wKHdfIP4H3|)$;0H&!KG<ZJ$ppZ?CH3
z_dW350xx}WEl>Ij-5(fz5s$W~_e8dT?}PU?c$rITc{Lk71ts1-m=?tP>f?<M!Fvb1
zY_XOn`6*1rogdr2O)#yDYebzhGL`GG`++wEUj9-$Z|1?hHT{{c6;nOlS)BS`U*-Db
z2jIO6o_m>{S4m1alpE!^QaQis_2;j^dk?(A<;i%ot*`Dss{Q3-@ZJZ{yTZn!{)}%^
zMP9_CZT^js{<jde_aA^aaHWk0kYWEuyz2hru*&hqyx@HZUh%40UcLP<@l5-Ve_(w-
z0&nn_wY+-$g>o}pzq%K^kHPb=uH|JSn(2^ubp7hQ$oBaT@IC>r^s8E)<To1^>@<sf
z;!%B1EAz&Wk?rZC(C<_5f?wNtQI_!NoF=>;R_!l~f%h4BL%&JRqZ}_ppV9Z5`B>l2
z!E>%n&ZGK<>s!t5pWuA~Ug|m<uiT$&HZ{aD)%QN|z63A*TN}@$UfB1jzK3A@r|)OD
z6TJU`ml>?()!RN(ex~!m1yGN#z{_4=%QI&dzDqn)eeXhk4nA|u-%!hw{%7Kr^#hY$
zo$CB|apX4_c<zmM9&X1D7gYZfkEV;t{iRyp>yY2v;1zDN@u<G|we`rW%z{qg_0H=4
z?Q!tt0nfYH#skQ30=E%z`w^zsaD%w6?-St73*NvjwY+-UE8-O~osCKJmVkcqfmgh>
zmM8TPt5<nraA+Eg<o6`<n;*Qv+iH3EE?@a&<6Q>ckIMb|&_3C|JO$nY;Q7C+<q1b@
zG{5)2qu<u)oH3ztKf}}DEeKxe_sMuv3)1i{@bvNcGvF-*Uhs$HJkoD1tgk+vOrahN
zgE#a?JI{1IjPj%DSLOAsp1(bd{1ySv`IC)T?$0&%GZ2gV%bSt<J_p{S;HB=c@pKic
zKCwu@^D$}t0`L|CFYVXz>h)*hk$x{m>hU~yi-VWBvz90QWGZ%W)yzVVc(l%MguH3e
z|CU5OmH;n%S1oTs?bi${SmM$2+lYQAWuYB-`9BZmz4Mj%zBK_7i^j(R<^D2Zg5>uC
z)^|zp+`kOx(Q>lB!qAp@G(Nh@yz2JvMevpauW+}Whp#cz93N0_RF9KGUgh{|Dd@K}
zc-~*_Je`;L#G`2tlipsvg#4BPZ{Tk>Ub#O{6pmP0SIUomvrG5CrIFvV;1%z&@q{qk
z5|5_q%k{0^@AC!nTMoRzzt{5W?SJXs4LV<48T$}@{_!Pv%Y*0tqn1~%zQm(Gawe?i
zErazP0bc2!wLJXBD_-XN1&Bw}S(tP^K0$sffEWC$miKk}tq2yq^!1@n!CMi$p?hsS
zbA5?L`GvgJ%KbjeVtrQv&$-{mD?h#^Tu}W_JetnIq}TU{$ZuuvQl(m6z53F*8`bxH
z*tQvMT{3=u1l}s(r5~u}iGE^LdgXILx*qXynO7aZKL&4A@G=k9^5o+(8;^1-MR@yw
zw;FiahiZAk5gW~qu17oo9`?VL&&&1&FAZM)VLQ*Xe<S_qn#6-;UZHY7;Z*Qe2hV-P
z&a>r5=hQ>x^Xcm6Wy_(ztN~u((d0bJ@t5WNs`WSo`K<|__n3`W?$4E5O+-Wb?3emW
zct1#$cPMykfj98DjfZ<sruH2w9I?cs`qB&cgQTVZ^@6uHc*Q4bdG+?c#4~-~-4EVK
z@CKi(<(cXbW)yzbO`nO<OK%U325%I2{!_KQ$u&pe;qvr9eb#N#&jYUoywcOPJjpK{
z$5oc_s`^c=?0*jjZyoT0XY9P;q%flVOzV3Lc<X{U^lWlo_4D}Z_2(18TMs-ZNX8>p
zRln-~?IiHl2QT%!jfei+iYG8dUc{^FSMATogEtzy^b0m#1jHfnO!^%Q-Ui@hUaaNS
z>(9ian$S!4zh8j2A$ZxBYI)LM!lj1)PxmrZ_3NA?<81)E40!pMYk4&X;R;DSlYS?I
zHwHZSm0I4{<yX~j@?^>HBJf(lE4*ssnU4?XUW@AbR_l8)cw@oyUbFGc_Y)G&w7%zo
zHx9gkf7kNr)t7kH^_^MSe>@1@c<_p^*YZR^!`BU*)fdBYH`hn^mxsWc0N&smwY*94
z0kGkI2I5t&zjRi|@5A6t1kZo7mM8g%Ro%teepr5ED*N9@z?%eK>8<3vYJS!FUIE@@
z@PfCK@#tQfYJaKjf3E~@Bk+davGee4u$t?|#547mB6u5v=e%p<(f)VLWUU?f#H;p~
z>iYf$yeZ(N-m~${=UsHKPj!8(ylcSQ1ibY7wY+-$nRur4{Uvytf|vQAmM8tBb?n$0
z{fK8;-xY8?v>ABW4{Le!oLYJbw;b?)bZ-tmT5w)MKh)<RtALjUFaJ?3PdH+uJksxo
zGH*;}d$kgHn}g?moSa9;&xe(HeU<y+Rt9ei@Cu*Ud8QvWr1hnG9A4&CKTjC}UJg9(
zQ#;SLzH*X{`<u#ay1#f;@U{eR;4>SK`tzik{V(N5$Kywa`BiwUfwvWS#m{X#dwuD6
zd>Q1_F{8WkdDM#FZ4KVw7qz^4=SP$u9X}r$$?p&7&)b0Ke_6{D-NfqtLifU%#@ipk
z+ZMdie`<LX`FPUCGmW=f!P^eJ;Hz3*g=5&>Q-0O)HmS0Gz6reT!5ebMSiUc5t_KY^
z)B4^F-VWe7b0_Di^{rgb83b=f@KW>GdAbf2ANf_+x7wf225%?u((@+c(Y=JG^*sl?
zox#h@SId+8RVsC<Bp2SxuhzFRK28U37x1$4*YboTR^iIu4XCbfwZ4ynmj^GufSuPz
z4@io<&cla2rsIvrz}po(cR@SPlppb``}6AY#&qy@1Fx`9GTy!5t&E)Sho$ZPJ^Z|q
zCGmM7y{7Ho)|>9{IHcbR?<X1HcClleDOuXzRi9@wb`l;B=HL=rdH=F4Kd*#he4W03
zCyucRa_IRxyQ!G2AMD97F3**h-<xAfhxBkc7JfU&<>%{+orF_MLq6DR%KVQ<mVbt0
z%7^$bMB<k@9;Qcofn+H^e+!EZd%ef=OFm<rDUk;dL7%bJeD>$(uSkBgIHr8)`2)Qw
z?u1~oIVOD|*SJGC7JWR9g?B8+lFvA2)AH@%!|FAzn$IVZ<-g=u^qHHV?=nn}Ic>O=
z2ZG>vWYJfO=a+nz;#l$-?~JU~XM8oE6C=x?!m;FYx{9fPoy9Tf(}@Glb2%<|X9!m0
zSn|1yW65WNGqP469RJJwBlvkV60gFs=s`cqpvOy8k9Bx{sn?hYZ!E`>&qQbR5s@bX
zq64*7^0_*){IwDO4IGPJH*rk*Q2)AB&EF5f?%<f#kNVf097{fvob_sk`$WtudQ9Ty
z>kQMQ&Uo39=a=!m4acI-E*uMgSB@p0$<gN}LLE9OkooV4EPro={~*Vr&%+!`K98yS
zsee7kG4bgMUe9wJ<}(>jfnK*pA3CTTrbnIe(tdaN{07R0UL71$KJ?n3<8psST;Q1U
zp;s@*oDX`+=xBeQO!w#0Z-~(Rx9uK2?<yM0?~%xJM#a)H`n3u_{|J|4%da&!ru^wX
z#E~3J{_AiodUSU8PfFvzR^Gobreek-|JoG@pC^Y7I__%b(fYR)3Z2)gzfa#UzCp#f
zogi~F$K`zReZ$*07Cn8AMbEo97CpzA^hAd%&yN8!iJ#v@abI7TvkAwfPwUw6W4GW~
z>bo_^Q9Z_4^w^!}mwd*X^ua^P>F?A2>>i$9;(OKKr}hUU@gp2d|9_lg8IJ+S!h3;Z
z$!CJIW=*;1E-{&ZZ+;#X#WbGw<+$8`uv2R1c$gl&9Wx8PJ=0A4mxB}!NT-iu$!8++
zsZD5XSTD@~T4edRIF|an7l}XQSo-rP983Rk`1x40e)O7~<1n9A94Xa|$8mT(qV%6}
zhv?<epN`~sm>xZy)17~r@9#Z{=a+h&rg&8EGdK?O8HX=~)Q+!KJoQ!TwIDyAOX9^i
zCVhyvR3u)OW9dI5IF@|U91Cv^jzu5C_NsNv_-g+-m**d*$BeF?w*GO(<FTuF{&Kyb
z=dU=HdR@=4<Z~m(tj{E>SG51M;>Rvz{uX}Tn8c$w7JXVH@pz6!pGh2xK3R^1w*|+d
z&)C}W)ru<zmHkV3ds^0Gl2MPlm`C|x|KIypjwzq6p8meSb4>X_t_crtO!?63A&xm8
z{8E%*e2uGaf41f4r^)*4#IdaJZXAnVdvGlJ>=l`RD#yZW<CycoU*4p={~gvpX5q=4
zhdS?t_r-_nA^$(a%gf(?ArimLG37(stJgR#>x0vtA&w~@d`)-GdmKwXI6sPXpK%#_
z0IBG?KR*vn;#nLMpI*HjQ-1X7=a}-L*TIqbkLFl-$8bDMkM_=f+xBB=ex98C{be~`
z9p~-zlH≤k(uUM@N=#<(Tp({l{@E`A<~yqq|MmjAO}vbB;yNu}+PgsCjk!aGP45
zwokWnEcLyM<6(Mq&z{xM+iBdtlp?$bIhK6J)jZF2TxOiw-)+m!)1&pFe0Jhk^4TpC
z@5!<Bx4k*0df;(tGhB{|N5?nq97{gqolV05z+Y!?$Ik<d!}T%I^KrF25Y96k57UG9
zw;j`G_eO8HdX49o{xKBcy~nZSGr`H$7L3oQ=<n~(&j*zJW^pY1UXCmMpJVAi2kZH<
z!8)2_;!SVsZ##x#$!DUo+3@3Y$TJb&Q;@&^3C|zZqo;t26^8TM`S^K+ln>Q+L5?YZ
zic=gDpW>xBmV72To0T6wP(2?<LcD#T`A>{2e+tLae@>6YXK^g~oXfG~Q{-6kxr}4U
zXR@=|$QpqrJBOO|XzTBhOWjzz@U;d%Pf_w)n`6nRg=5Jl!?EPk$}#7IBiZQnoN*bv
zS=s+y9a;Wbj^*#)5Q%T*Sn|1zW64LJ2Pyg7#qFHWq;bv><$6W+nAX+S*C!$<y9qxJ
zlKB0Yw%}Ou+md6+XB#y?)qfX`C7)e6mV9vhKfGLH$Kuz2s`}g$S^nM#|3Qu=pNBb?
zd>)I;{~X7X&+{Bd_2@se&|w-cd-L-wY5A!$W_Rz)G3i6+)9oBnKJ@*e=^Rr&^qQ&W
zr?`h>;?e6Mjzymd27Pe(tuo$Ui!A>Z$MpB<^<E_YkYmZ`6OJkW@{6CZ8P$Wf-JLUK
zrmAhn&q-$Z`WwA_2rpk=A3FZ$nDU|LjUB7vp6)*9Cmd5gIQz_;$g$)zz_I9aI>%*w
z#%0E|I@y{H;uwst==FeTyd*xCr<WWDhz;MZ{=dni;q{ED9@9Ig(dDML=o5pt<@qK5
zJjas%ZX8SgW1Vc|`ZHFF_rExR9e)qckK0>u|5R7Uy&RLCorkscKcw1efBGoLq$l;q
zr#Ys4=<!4^spaYW39oQm&S%Vo@#FA=8AsvQ@WxJ>sIH0Q4$W5VW>8Oj^h%fA`lEW#
zb;G_w+qCHZme#wxAK~Ys(t1$5EXPZ7yb{N>UerFCW2zg)`Z`aNm~^1|Gm*bPiDS}*
z+Bc2F*+{%aB;GO-NA>9K=`wCVWIq{7i66Ygcj)o64VCdciJ$jMI?-zrj`{G_$#G0N
z(Q6Nmqy0N+OghnPFXl=AoyxJ~(-vu;8HqbNj_NV3r>m=tMojo;D*sQK9QLQS{;BQs
z{lk77rl^l&=vPmvE8>?xAkqwHJoP6!G<5knw&lFgu{$`X{zSina3IH`(~%q})g?N=
z=;Z0Yk3V@L$D-S*kyz=`O4kb~@%1s8|IEnzXLF28gyFcG<V-COJd02y(`2l8v_GB5
z&*zo;ox(952UUKZ!LjIc3CGd(P8y3&moZP)^Vb|l^^og@H$>*YDH7ijiErarbecT2
z(w`2-(@~;b3BRNzT`63D*)9+AbB9Hzn>k(-v(fh94;+h5_i#+QB-tKC=a=o%Fg?`u
zLeh=+Pw?N9_^C+zOeB6T5<kyzMW@zke~Ji&t^^gUU8#IOfavu2GvWSFbb5y45uu#U
zg3k`W{=YiMN&C<0%<J#y>z~>&2frLSwZFG>=1goJ%K2ayS$?gN#pl16?!)wVOpjpt
z3#Ml<J&h^A^eiUfO`qO{$9m72)z&>7d$G1&I;N%9^+Tb>tsu^1+*?AA06Y#CbyGKl
zpD(1RSASQ7>S23jc>Z?Gum_eG-dc!{N1V6vLfxX=g7BUjmD^MC*2auKK|EmK;VBFS
z++AIB9|y4qg@?~gexP_eqU{94B?B*9Uvqv#!b5(0DPA7DpCV4(SgRjx)6rbsp1}@#
zWqAH}Y~=S=yxlO@iHP$CUcLMZ!V5;_eyDgGfp-$(0V~g<pW@*(w^HAY;r|T0l9gxC
zPx0t;Kj|;j#ZCq<byKZ=y*)E=-r6^HPTREEm~GfA!}HTo#U5x8-jCt`9K5`NS64sb
z?Y?$yisD@i@F}F9m1ogUc+Zc@T>ysUcL|(RNk0Rxw`VqnsYO4<yHN4=L9V|5FFR1H
zAMF&vt53D}bGkf#`wOFTo2d2O7ybZvo`HuZyv+F(g$MmM#ayz!`@ugAJU<~X5FYZ|
zO!20|KOH>hwA%H>yLg%NOY{8gINZ--F3Hb@e+KDi;AO^_pKEBk9TCsN3IBfki=%Rz
zD_$G?Gf6)K4^4QP`%6)H&~FROCHc|z^(@lQ$}_KTAiS4G<#LL*Ha7TYgXf%{P(Pl(
zy`!(MfO`ogzyAPr4tQAuuikibh4=EP9PN^X_Z9qe!Sk#<i+;j`e(x%tGZ&uG0G@B<
zS@aVg@_SG5<^u10((ep|emLkrOZo5>cGxS;^S5JAzK<4J-?_oNfb=u)Lj8teDuF9J
z9B_P~c=LdFA?au3nZI99RJ;!rZ(i^&BK@p9^XHp^@WA?5@#X{XV$$!-TK&THHRqS+
z`P;F+pD5n^;9WxcS$X0004y@><q8k)eu_!zu>g2Q($C5(pQlirhrP0Qh3nJa-!b(w
zv<PoO@Gb?<<@Va+D|7$zg!kI0+~<n75O|k?SG4lX<HHx;?rZ11P`rJ?yBxg0$}{KZ
z{F&Dm`5gv^tnVE7SAdu1cD=qh9<q*)tnhF(=5WP382*){pMh6ze0UWeeqT%RT=-X!
zepa5j|M`lyrP8krykC-jRvu<Hz5c@bp=}gSGDN@kaUym#>BsF_KiaV1>NI|BzGr3>
znHAmv&X_mB5Z(vye?|Hkcp<}FUr%_ikIKEJc>h`&&(A0QtUOEqQ@p3ako-#Ue?$6N
zdFK3_yIDWv_l)8_0RI}&kK46=bl(w<C(9?xHTnq;4Zl{r2jO2!`Wbk2`=9Wzz50#f
zJp}(c(k~&;7asDvM)4ko|69^8A<y|M>xb(u*DBsI=+E?w?=-h-{aSId?C^EG8vTU#
z)~MW8Xq5UM2j2ChpMe+ZXYMba@UXpFRE>|*!K2?aELwTy{^AP{`Mryie97+}oCDuT
z`X%H!e`Eco_Vm6AhVX{q-$eRxyVkGX{w*s!X!nZZy$k<l($C6^Rw9f&#rwD7y$Amm
z($C5>Z~weAMwZXBY5)70;+>Cr+)6xd*XwJ}EPU4&9>Ryf5dAKIe;as#fmhKFpBL1y
z&*c2j@UY@t2>*BBrMX?_C*#+2fS*;oM-=ZO_`e6wHSn-~#`$Vj=QLaZj^^hH4?h<E
zsN!7={|}^}ffwdy>CeI&8kKuY@h*Y?N7B#AGmkgtEY=Ua#}%&#|8~-k+iUeR=a*Hy
zCl&8f_<th(3_RT4L8m*YZte3mJ>g-7rxfop_;-+g2A<S6S__JO;h{f2t$3Hi_esBm
zJm>6c|9eL9u7H0h>BsH0`kAuB@vQK$eSTK)u7iIU>1W`LA4|g`yfPl8dcs4)bBcE>
z{6CX^23}>nMPZ12;XyyzzRUQ10=&N<4y-%_jNUpY)UONfeQNvs5_ormm*)1^`Uwvo
zwA`oG_htBh1<y6`!pzM5+2i@!-^cYy#d`(*-@q$cdFJuv3lICZA1L0d@b4l0tUPmm
z&bi$GKEU-!#d{6@-$_4i*ZR>&!#Ynbe<vhd`O5!U;bES=74P5h|3Uhhc;%(d`FX<o
z5Z5Oa?{)b9B>fD$u>V=~Q@lMC?+y6>BK@p9bAHZwtRM2*UGd(8e=q6B?OMOi8J!)L
z&6SfC9yIxp;=Kj`KGM&?t2f>}#rq?eGTz<>?|#xRA<q{c#_#Ql_YQa^(k~&;IiL00
zAKQ1u8v^eE(vRD<e)YD`S>YkSI~4C-@E#=nth}$jn$^}B{rJ`Kga?+dc<+Jt5b0;-
zVP?b2SG+%<UFz{Zcn_0)241*d;~V~>%nMjQy#IT}`vAO0NI!0mt)K9q-;s*<A$X6H
zehGQreJ$bfA6Q2z-bdg)M*4Ak-28muVgGit;(ZL><D{Q~S69FLtND3~_X&7Ukbc}A
zH@~d#px-fy_bGT!l70z!p76jrR`EUq?<vx+9?x*R>kAM19jADogZDIefq_Q{R=7M_
z?toZi_^VXa?|8IGfBpizXTVEydu;uL2mOAccwd6|EO>50o+mu8PEfr6fcG4D#e_Ux
zc+l^siuV<G0qJMpna&fiX!v@ds^5utNA$ZMyyr<jZm-qPe7=<x9`w6Y@tnEw-B;4j
z%Hw<@#uFaiKS}ZC0`EoA&&n%b@2olB@_GI)+>7-y#hV+vmq<S=&$9o0koEg=RPGeT
zn-9E~Nk49{)z5r<kX1bDA5xF`!FvU<Yvq~q^AvAD<+lKMuYy-J@c8(uJm9d%u$M19
z)b}*BNPY`~_ZoPCfmd&T={!{Jf2S+nLg4)yyfnAh>NhN{N{|&E@;gKE76$Ki($C5>
z_h(Od_&(H`inj=OZ;*agUL~brjxRjqcb4KU3f`NfUmYI0EbS79zaFmYcMe*l9*cqZ
z7U{?BvGo%kwl8NZ-s0fBP5K#lbmkV`eLOrhiag;tExB_MOMXj$_YUc2;MF~T79P$&
z&QrV;ctfP0l^5znISqR~BKo!D&PRjfw<LJ)l78G?tDl*d6<$Z5bAjS51>Sq4UqYTI
zyt!I(7b@P;;Jr`!S$XE|i!VIrcah>P1KtOupOt4`U*}QQ5BXiJc*}zKA?e5MT0c2|
z!*y*sZ65Z@3J;$LT!I#<$8z9(MEaR{Wr8_BPk3{;<cf;7Ja`|Iepa4EKgGLL@kW66
z3F&9$ne%fVWBul7$z7&+D}eVY>BsF_Kf2EtccyoBOl_NuMTWhy!aKmpT#gpeZ$<Dv
z1J5<^LV!6xPk8gT<gQS>mB9NPyrPw7(NFQNRJ@hJ`vSbc$}{KZJkI(dzpE5)74W_U
zFU{>*zg`H4yHv{phU;GaKPx<3@BF3WtqR_MNIwHF++LaU^Mp5FOYUmLTMfLgNIxsj
zqMzdZO7YU*Ik^5bu;2#e^Buju_3|6!`P-4-uN7}~@a6)qWZ>bGC>)RDCN><AW07I6
zA>m>F_8YWF|62pRxxq^<WXo?DrV?a${&u``jpD5d-aO#ttvvJoydXT}cdg=$1aDr_
z&&sQ$G|U+k9^M^PyiwrINBUWL=KO|)hy1QrycY20C;b+->u0$9>|}WUcCc<xymi1^
zfb^@wGaa`Tgx5a1|3<}I7rX^YKLZc5;bk5lgTh;|C3h3%lK!$Dcngt!R-So$3<(eW
zx0@Ajeef10{T8X!kFvAnm*M%_k>4$tOY$2H-Xf%*i5H&t(&fpvGFtst5FW<Rt%|n+
zc#D#L2A<SGtim~iJbxGZ%fpJdA$W_CepX(1U$kbCKO{W(k6@DZ-4?vXNxwyF^^?5B
z5=UMcp1*v39B&G5JMfku{S3T%*GmeD_q5_|4_=D&Gw@7#Nq&PoKYc#<FU8vdyd_D$
zguEf)fpxFq?Fimdq~Bt8{Uj%CGdzDgj>rG0csqedzb%qC@WSl?eY@xYjVk*E;Vs&d
z`-kG~4Bj%NpMe)bEaR8w@4|SyU-5PUZ&}hWA#X_K_oU+G!CQ{>TfA1kdiiB|{`QPB
z^(n>M6};t1KLalmu66vxG36TvLOlOp5Z+=fxhE8FH}FP~eg<CH|1A0m5Bfc>c)NqQ
z0_m5KH^lRIA-`u7Zx8TRB>k4C)vsQD8J@puf-~(E#oH6Sl}JAW&k28+jc$poc@>0*
z@%AcOWqU9cyp>5m1F!D*<@ss+-lljic&m_pR-R@5t9ZXtyf*MwCH+!1{ml0hW_bQ~
zu>J*xtZzH~)kr@p&zxUDcv#;VfC+CJcxlql%0mM51;fZ|P<VL%UX|Z;_^XqCR-XC#
z+K}+@QTR-i-|^tBLHaFe(=U=3hZ&x~9r1xGzn{Qglk~IlEcz+lm5O%){Iy6wE6<$Y
zpyK^X@qP+_ZPL%cqp#ZG{0-l(rQ_(~ub~QWhI5tTod|y<>9>?kKlAn_!}GUe`*OA7
zodkar>1XAckFN^CTe2nhOU3Je-$MFXdFK2E74H$n`vm?vq@RJu*AK#PKn<7A84@1e
z|C8d)fWI#3x3o<^bAQS3{Iq>JaIWfj`xLzONIxqtk{E{t;UPXk@n(XzKIxZ`Hz+)O
zb+b$H_5*J;>1W{KumCs2wT{Qb2IfSw91<QHIuvhz@HQa*mPx1|&rkY228L{}K7+p@
z>1X9dGYjK_@DTcn*9kvE`X%HI3J>3Xou$_IbMVHHeg<B6Ts0n#ZKwMRhrfn|hqsSZ
zyaT{%CH<C7s2|Ty`t>PZ3fB(Cl73d+@Vq1{2oG;usd!z;Zyf2DkT)nid~0tt#hV2l
zeHpl9;5mCzHs<j*Bs|QKR=jTTCXjy1+4Li3?JL9cx1(uw#p?l&ev>b6<(cy<2oLpG
zQ}GJmO(OjayxOd)?+gkLZ>^zt2Z1-4^fU0n<00fXrWH@R8U7klytNc>4tVrCYpLZE
z>c{i9<Iil3NyhKN;L-19<qbT1mmJq6(IL8K&l;W`KE)LtzUlicmKWY3;7vh1VBpE|
zpIFTt6dvlat;+9E@HPRjWaU}Te}xCuc8Yfxc$<Qk8e!MZlwXGDZwG68#XB0j&A`hW
zcr!cbdvEkD(dg4Fpr7LHqIe#7S<=tIlf1-YvC3;uc+l@9EHCvq2E5G?mlE=ZgonTI
zisBs$-WG^cD<ss9=cn`E*A(wK@N$Ur1|B{-z;P9Aozs=|6sI6Oym1B?vc9Ln-x9n5
z125Fgyge8c9`t)%<##4{TOlr4d6Lm^J0v{3{R@@fS@5?8FSVjwzu}}X%JBT{`1Zws
z;++kD8}RZ5o{1yx6@-U3PE)*d;BO1wfPq(6KjA^Y(-rSr_}h_wR-R-u+zts3Z@;8?
z=fU5e^jpcMU-^8dW_z09`P(rZUsk;H;qO5DS$V_rlBgg&yzz?ST>yVa($C7X^gqRW
zRq-x{zZ2<a<w-`v?U3TFt@a<6fVVU0w{oq1;dXxZ^nUz+=5QL0$$0*D9BPhKydrqJ
zkbVYUz5Y_D@NoX3c$b2gC;bxg28D<I*P?irfwwE^mykCkye!UN6z_8Ib|d{(snyT4
zT^1cPJU<=(t*dyy25)!L&&vDit8PO*3Oqkuk9`@-%l7XY@b*AFVC6|h!|kBthqv!n
z`CSX%p5T=XJU)Lj-d{4LcqPTV4!j?Lms&NUemp<v_kiO47QDT{%O~U&gopKgQ1J%A
z+ne-D$Qu+M_HUnId8x<s;QbJB$-pZ=X=ivDD$$VeR&U9Du6Q?qN55m6TFs`P`Fuaa
z^V9W%FBI=a@P3RqZ{?ZK_Y1;9eqSoyP2lZ=cpxEfP<S|4`j6t>4BozoOIBV*vSH3p
zg@-c~#k&Q({Sc?pHvP=o=M2x^PJd;-YJG17Zz_0sD{oj@m7t(_b17aceKrl=fPsfA
zP5pQp86MFxt)~KFXHa-}lG9dTNdFrPzYV;Sffp`nuJ4fW)@sRZt$5?$x08OW*XkGY
zEd7t?ZwGH1#TyTQ8tG@{ne!_s-nNQ20seH-&%h($9B1|{)An>wcuv~cPVpwf?;!mQ
zyn5UFA>m<rMc=oSdQ5^pgY;XYR=;}t%M8y?{g1wHDZI(>XOeza9(Rz4DJUL&Ur=~=
zp+E0W`Wbk$``YlmU%HF8=5nVqC_Jn$-5)Q!KZDmv`Wbk2^%EZQ?p538zkqiD>9?j$
zzsQPmnBn<p|M3pqll<-m??BSe$}?{d3c|zvgq<qCd%)`={j5B5euKh8eoufQ`TZUK
zEYhzIZ<F2)3sLHj@X$c_V+-#o@VZIAwe0$t>XPC4>HO_dwY_>8ydKiez~j%~CbXJ9
ze-mB{?w3-$XTU3veg<B+j^^#bpzyGNyIk>}1@9ozFClM8c*ySx#d{9CUea&vTK($f
zm*M%_CpgnDP`m)VKGM&?qbI1*hdrix6oj`9?k7~d=fUeI{S3S?KlAo^P<YVqBE@?F
zyxF9mm1nNUknq;U{e+75B6xF1zmW$0%GbfnybRCZj_=l7qImCvcQEN^;?aXQE%&Pw
zgtuNx?sUcb0K7v;KNGJkphZ97LBF#U??dnoCH<^C^ZE`65BZ&`cprgx80k04rk^vt
z2VX!6YhC?6!}GV}YRX>~?_=-|C;d#k^5W+F3c_2zC3m;t&4+Ex5u~4$XVH)6?*i{d
zEHB%u5#Sw(xMby-^BYq6(f4_Tw*vg5z)Q8%>PIKO*qh?2UG4X~GdzELzw;h_e@S>N
z!ao|kynz>vTXTK|;i13K_m_mX5_}K5AmROl&J*F^X&;UI85D12_{V^k=JvSv6J`~U
zzHcP?tpfj8@LU70b;8(j=rZ9GM5|q|?0Gyt-QRSs>MyH;cN};{1JAVG6Ra=!;SKtJ
zq~y06{NqVKE6=<?cb*LOqx|UmP{K>Y{|V{G?X~(zM#F8E=WoZ`bTLJEtAlp}>1XAc
z^Yes<{BROb;jIDQPf0&3FMN*c@a!DN7akh)_w&{S??lqC4ll|o$4_zp!>8f&{X<#b
zwcwvb`f+=$e&+F!6<($#cdF_yYlHVQ($C5>Z(ls&VZ8l9@kWAoGU;dKne+37hx`T<
zZxnbxC;bdOY**;IWJh=TzU7FDPqTiQ=QPD@0q+#jkK1eYi*Pv13J?8-zV9aWSO@;8
zq@R^%>3_llpT3VLydB{Gg7mZU%=!6>x3l8y2!DX|Gw|x&59d6?`i*VL?V@-)!9R`k
z<94lIn3=i0S>d6-<Q4CD_@|S833;CIkl!JS_Y?SMkbVhyzVOC72Og?;C%`|G^fU0n
z+qum3aGquT#<k=QQ@o$TKa2F^_FDbS<1H&Z=y!zToe2MI($C5>*TWMY@;hAdPJ(|9
z>6eh_3$JZv$B~Lh&lWwG^h?Ndo@4#Sx8#meyp!RdNBVKQ)-T+@S@aVg^gCMdeh&Y9
z($C7X=qEhn=PBMP@Gl_!3_RXm6>$4=&F8<q@bDuf$0*)Xi{bfyq@RITZ-4FttRJ>l
zDKKRFvNZgQNI!0`)z7@W$_fwNeM!X|K&}^)epa4&e0YlIs`Wh${w1WJm1n-6&=+3o
zgz;^PcRKtc>1XAc^K+iB>esG#XTZOd^y7A;e*I3(^9r)U!=Ia`cxS@DjP$ee!slp5
zv-A}29K~A}wYi-1Gw^!D=O`F2IQhcc2-g!;ea`~>3ewNO3v)7$59bBe594h=mEYO$
zuO$7rUF#R}EaO*rlUs6A74Ka<yYwp3&&o6B=PBOX>iMU4gZ)d=&%h($Y~Q!{g@=`R
z4U>$wzk+u)>1W`{_!S$jKmY$?wf`-G<%Rb*@P36j&Fxx0;fU3|tnl#WqKbD9c)v#M
z*5To-zvhN6Pk6X~wV2}3vrvA6xM<+fk0Ri1AbjFEO{Z4&ec>S*sd#?}?;6B`ffuf)
zd3)u&6zWIkAJ2m+`q8s)t_3g6?I!(dlEWKW;lY1F@#tAM*Ma9I<axrw-9RrY-l|~z
z7E{r{!)(@<&-2sy(FiOr>$@8KK}>;xS69E6tNrgVz=W3u>v~LSZjY^>@Q~l(inlsg
zH(+wDJoEhrp73z(d=)G&`K<x}ModL3&n%%YJT$MMcx%GH2~%L@S^D2AtRHwQD&AV~
zZ^o47_SpI<-b#wMHvC&KxdvYN{P7Ol;nG=iKdUFaakv#h@iu^eE2g4>XWFian!fPx
z#%s6+CiU15tlKaJR-Q$_SF8Q+1oe492CUy<N^`r`PjWHYtnkqIQ^gwt*6%U723}=+
z815(Z6z@dEYX$2Mn2J`ODLcpU74Ia)8w=JSF$D%5eNh)zllwcn$wpqUv3?UKO#Hdx
zjRWg;OlfY{`bmAoYGPJ+*q$!GV70#E!TJ*>*UB^BPv{8`e`JK>Z3WgHn2J`O`FO}z
zyiKvZ^uMj)`<Ma)ue+bFB+*%Lp;xbK+5dNS{N@yo?hU&WQ<~dj>nA+ux0&M6y<vA@
zauf1A;UU?q;?ccff5ucy$n%AV{rTpKNB4&P1yf+)h3CJm6DCb^#@FcgI_sBf$!(!{
zJHWpiQ<~egej&qh{3kpdPtxZxQr{in{}q#K<%M}f^Yes<{OI!-;q3(fZ<vY+dA{(_
zndtKv;q3wc9!!COXBuafJm0AHKl;2wczeSCJEk<ZYyB$Qs|uv*WEF2W#rpyLe_(R0
zyedO`p5pDScy#a3KQR>zylMM)%);Fh^l?DV2j0Ho?V@;l!~Ykiz`zUhG1u35ll4P?
z$t&Iu;oplX&F!`NS^A&wwra`ks(6c|F!v#Ltvqvnp5kq*u1_uj-u>Ve4ZPL~^o!+=
z@qwVe@G!%6ikAYf1YTg^)$1?LTdW_pSKBMzlHff6UYgr$_0zqi>Su+Aw{}pxrNDcT
z^h?O|gf{~Bb1L4_;5|h88F=Nd_c#+KWop0P<M_fuemg4OK43kJDKPLvzghT|r|O!r
z_jXmk71aKBU$7p*l;(Dme$#7z-o?oZ5BL=oZ$Gde#pKrEnYJ^Y@VcjUj8MF(U_FMZ
zXyAowX710v@V0BoEw6YkSdU{0tUU90bKYV7Fn-rlyf(0&z?9~8tzW(Tvcf}t>nmP6
zSWjYdO}y~9ukWC)sWZ5$72^pHyT8$jHw~<(Fcl3vse{-G42~~6jJFLHZ#q~{V+yQ1
z^YN84#QK4~f#P+5^$eyow`=_*FR?nmtm0)9Zw6S;VsZ^UI<T_+UXdrf9a?g$DBeu4
zp2Ji$@WOm7{ZDx4FRLov{$K?(CFD8pvVO>KWyR|R>v>FRZrA$N%P%XuR;O<z#XA73
z7cjX7Uh9O(blt`L`LFOW!x+Uo5Udw56%D+wuIBSCUwG($t%}zL)=QWID=%E@n*ZOc
z_P?=;Hw&znF{Qa(>u2hB!p#bAR-tRW;&p@d3MSXUqu&y<y*%OV)RG&gcs*dfim7Pe
zh54DcSHAGj|0XCNolC!lDX{X){n>e+_1n26_aj_m!fvhn`+jsT{clWZZm-qP%*zT7
zWAev}N9WS7V{)xLvx1)R(7c=C(Yf>+n2HH`zVOihc2_()mwppdVC9+nv-1J#hx-Yq
zLOjur&ZXbNl;-wY{mko|6&~_)6_1Xc-^S$D;~DOM@q~vzNBc*~kB*(+!BninGmR5p
zcsPHf{iE>c*m($3VBpE~2~9lbL)LHCmK+^l2#=1P-^G;XcCBBioOyg?g@^G$#}~q*
zW54$>xmKRJKYPMMeun@iJUaG!A5$?Q&letcT8AoL3yuXp0534`L^rX+dd^3zAKvt^
zyyUkIcpoB8bGz0r<e2y8S;ad>@zw?JBgC$iXTDzIDc-S)w;p&OBQ6?vbRdNNd1v>`
z+5=Qyc=%E1;}maw@IFBt7<lE|2_18NosU^R^p~F~-e~YXMV#h#tzX^#SK+nfj#s=5
z!TSucYvq~q^Azt&#mj*AIpU&$M-OU7j`T6e#9@5}fiJv0aDS}gjREfq#DReqGR*lo
zpH%g`O7U93`x0@Q+iUeR^RmK2em_&ZvEcm&v73<R32%Zk^<>2x2i{kRiw2%-ucF#f
z><e$tmfX)3Z#;Morog}p$D4V3<$PMz?-a$G02Y0}B+czwzmR9qPk6}h7m7C#thq6{
z33;CI@I2L16>kz)^I$3_<oP^**Iu~)O7SLxH7};X%CqSASyjK&6mKK2=EIcc_SpIf
z5BZ(0czLkq$K+ahrrv^WvhdJM+b7vR?+Vren2HH`zTz!OIe|;R$GkiI1u+Fy-dA54
zA4v8&>xbiwl;ZsW{z912++M4nd403OL-SIKcQE{gF}YS=`SZ7$?Ug4y98aFF_Ln~g
zYY|LEE6<#t&-2swFMTX8`aKPwelIMr@+`-HU$B1Q-J*EUz+Vhgn%iURr+Bw2-m~x*
z$K)D#^dwbW@5EDSr{a4)!(U#7mvL@WyyxIAfvITV8ILm<>kAM4<#&n~z^DEbSa}xx
zzO3r^d&PSmKK1W3x101c_2;bO{Xy|wfKUC=HSogc$zW$WX7X_H5P6Dso#MR+e`!ob
z15b{xYG8Q77asKct>V1|e;G`Hm1m;iz5i788&tfP;V+9R&Fv=rOvJ44@Nlf_74H@J
z%VBa2ypR*-5oLP9+ovUWgW|mke|bzrE6=>W_l1Y!wQCjcHTWYi1y-IpKj$mf5BXiA
zc>jjK0;V*#YyJ4Rs=v1lH%&)gS>fR)OK()X*Ws^-$u;omjW<tt=r1=b-W%{&!c?^K
z%;U`$9`e3P@!o{LGN!=Fi>zoRT&ME;E{Cc8+go6*f+_8U?ZfmdKUW1$!7?15W_f=4
zzTe@B_cmCoVsZ_<y80>J5sLQ?SgT<wT6q@z6z@pI8v-khDX{X){n?p|^}BHG+#Oh6
z_DAo+U!9oTuJsGo*UZcE{9RYCox4);-h;meCO09^ll*R6JNFmGdumZU{||97A<q}y
z7i;I91Vh&MU+~ugFR=2$-#MyuJ8|Y_{T3LR`={bP0e@}q(%fFFpP82x-qIs;Pb=R0
zV2#A&Cggd-TWw_Sadp4SOt9$pdy5HqzVOx^nOjm_zxn_yx`q%W<T>-OeiKIK9##3x
z0E@06q`5t|e!|;)WbO{d`w%R;hTtaTdBWRiWR8BnM*7P~V9_;%VnUuTyuC-}UR1n~
z!CD_vkdWuh%lb_pnWNvYk^DXZi>@K0xjnXiJb%~0BXh4P-lt$~fXPkB^MrTY$lS|{
z_Z3(hVk#!&`8<F7sUvf*DjuzP2D~64&zX<)!|$FQpsr`n1>P9&(%c?fKjB?DGPkVa
z%?(~Fcy2<TC%o%N=4LD2Jm8Hb{Sxwg;oUwmx18cF3?5yR2omy~`B}evFy2%>76Fg0
zNu;?wwtm8UWMu9jmEWS^O(6Xe@;u=^KQia5{1yXmBI%cq=L_$xk-2`A-?HG*HHjc0
z&sl)=`*dXPPL<zs;L$aSG`H94Xa2pXtnlU=mFrgdEe{@DlW?s(^Zv*aUSVXeNAXq#
zZ)4KW$}{KZ3vcA8++AvYR|9Vf>6eh_EXew`j>;8OerfPFA^o^Lwtm9fY*g;gD!+BW
z+m!T6$n%7^{is|C%L7xsUa~Hn%@7w8@_ga#H7fUj;%x|C7IBb}=PbnfwU5f}28QI9
zfwMVyX>O0LpYVD{<@Q#*G4Qtl&rQhlgm?I;9PNK3zg9Ro#KnX>Uw9{u%Kby}#)7vc
z;vgZ<S(x=ZdsOb9iZ>3ttq`ZVJ+^+ryL?pcUy3&#ysZ(t33;CIt{;`VSMer*w+-TA
zLY^<YKaI-Wr+5>=+ZJ(<kmoEC>euzpQMvmSZwK(UL!9RJ*!uDO?T_R4#T9Qy@U}<n
zCggd-dkMcUu6R3vw*%s0LY^<YcSq%(P`sVN+Yxb)kmoGQ`h7Jj_mtx80^UxD)7)OG
zpZR)3R(OlH<epT#Ja{`Jb`$bEp1*y?mfXXNw<~zNATB24`NA88&ubNL^Aw(?fH+9V
za~9+AJFX>n9Jb9!tGs>Q0=!+pOLKc{{e-u9OYQ{4`wPz1cLUE&$n%7^V@qyRwLiKW
zyxqYoCgl0T`(aBir+9R4@gAg~m1o|cJBzb^N4MlQQ#`u2cu&%g+hgk|ypvmUS;eD!
zi+@1+CFFU+JGUjbx#H2i#e0!{33<NouEhP`injyK)%Pa-67rlSc>LbjlG|3D@9zlz
zhom33$JUSMZ@;4@x1Hkc1ph~*UqYTIynnUiwpYA0@ZHHDlYR+#zVM!G$t{U7CgW{Q
z@b)4767rlB_dk4ZWI4rK3%q?vKW>k$pYU)W&QglEHhBAyehGP=@NiveX~i1}-c-`h
z$_v*En~-5IUwG(e%P8I`@LbZ*z#9gL21~Mjcz;>NYXPr~^y7A|UlgI^EYIJCH%BPm
zI^eaFehGP=%5Me5TNk`(q+ddwuXrme-u2iXOeg&k@|>lr`kkw`2RDG%LHcofZ2c7P
zBE`EAJh~?4Cggd-!;cl6r+7DkN7n$02421E*}m{_ef)gIyBWOwNxy_VXX&bb7bxB>
z;B}II+^+SjmtR)#E>yhB;U7TyCFFU+!}?yLcvrwbkn~H)^M!};QB=Gu;dhaK33<*k
ztRM2bRPnBYKa2F^_FDbSpD$z;?=r;;Fy6XJzl1zb@ty-iw)flPo`D|H&%iqz4<^LZ
z&*<aTn%_J2dHyc^mQfB2;q3swK>8VY_4<pmY_<PwsdziWKZx|>_SpI<-d2jY6Z~G%
z&&tzV(W>t$-qwn@GyFc%FCouYyloV37x?|8pOt6sFV1pR{kB!SJp9?DAGgQWPk89h
z+bP};QRq3OUqYUzc+<d;{_-O@2a|qQUbP0=^M!{urz_r%;U7Z!8F=CMSMk7@sWWT8
zf4RKqSLkh1ynWywO8RlT)=%dX^|KWo_P2_+FZ{zuzl1zb@s3cu{oo%?`X%J~ig%>q
zJqG^>(k~&;84>D7#~W`b-sA9(B>lKOu6_eYY*_g``4+`{0{&6p4RU*2UP<w8RlFzR
z9}V76LSE{~YJR_0yr<xM;AOZyZhm>;;r{VIDBjcXj{&b>;MLon4hRq9_cq0Q2L7?6
zUqW6<c&P916z^I1$B}*sd8wnS`u$Pyo`Zip>BsF_zk2JN7arF4cEt<e|Ah2Q$QuwI
z*7r|}_dNU)NIxsjynQYS?>v0pT=8Cj|5MV>%8PDU!Z>v_>xcEdQ}GtZ@!E-`AGg=)
zXU;D#JmCMScrU^~iS)Db%+Dtr;Q48LaJAxn0{>^EpMh8J_hw5fztM{KDg2X3KLf8`
ze@S^&{Weg%&*1-@^yBtg{mk{q3lGg3E8geuPa*vh@&<&5?^jG#yf5INO8Oai6UWix
z3F)ro+V3HigoimcQoJwW|AO>0@Iw8}{UvowRliA!_Z9pB(vRC?>nA+qcaq{c3*h;G
zq+debfbj4zlAkHwT;QEf`Wbi*4tVkB9n<}UCE=m{WW}2syfa8Y1Fx=r$A<dR^MZb^
zc=LdFCh5oRvGwEm>4yJP6mMSe&LaJ+JoEjw1H!}owx=rIeBhl;`dN9TVC}0UJT(16
z@fHN{9MaFqv-H2?xc`Cokm4-_-npb7x5w5`@g7#Zg~2<I^fU149ZwDjZ>%%t5ye{s
zyz@yv1Fzopsw6!0mq!(EQSdGx{j5BTe#ck!J67=)1Mfo8kK1GGr+CLH-s0e0MEV(c
z;qR`ZgW-!R(H=9Pct2IVCBVCw^fU14jkl8Upx;jvF9qHuq@R@+%`A*lKdEm2PEfoh
z!7Gw}++M4nxgL4p;rQ=(#hVP?rKF#gXU=axcv#;_U;tVEJY{S6myv!6c_rb2vytL$
z1OIZ;uO83vd&j90SU>FFHdefC;a@@eal6(pe4g7x<MZ6|!b5&j6mL8DSCW1PUcLS@
zAUx=|iQ;V!|0>eY$}`uaBs?^4s(3rV|0U^{keB)?>o>h)#%7AQBmApLKW^9hg}?V{
z&Mz-KG-nlWC-}c2{S3Ui`UwyEZLWAb!~ZquXXTmmD+v$HTPWTx@P9-4S$XFDTk1sV
ze@-E%czO8Okbc~*^^@(h>G&WoJmj~f;_V9mTGG$TGj9(D6mKiV+YSD8q@R^%(NFQF
zE8gz#e@prqcyl^C4#w3%{Ca42&GWueC$WC`dSQp+?E!y~^y7Auex?lb!b5+Vp?G`3
zzn=87@+|uC{B-^?Q}KQP{|3^}$}?q$XWB@9Xx?A(_JV&S>1W{44-w#QL*xD}^)t~A
z*Vh$qZ}>Nne%x--@2^MK=9m{A^gBTDehB|&(ytD$mWE?^;i2(B#rqNbTS&izypr&+
zz3)=IAH%<u^fU1Idja(Gp>$X~?3Fq>)Q{>rOY!!Be;et??X~)u`%9kZr$5xKc>BWt
z9qDJ~4NI#M3@BcY;_V0j_oQD!UP<u^iZ>PhA4oqd&%C~=pL731e>q6;T=;(^{kXkW
zKdhqR`x<%Sp}AM_+Th<#`qkrARI54zJbxFy&eo@R?ePCZ`X%I*RDS)6Hx2$Bq@R^%
z(eIS1ezO(t2>3qf$L&V_s;gP?@;pEF=Q)aZB>X!`KP%6?e;W`UHa`a|-cj)HBK@qq
zN=n0=lJM~EA&Pf2{6CX^R-Q$_Q>)|mP{s4$|AqA9_SpI<-eHP&4E(!EKLZcDH+*la
zeATHiY-JG)DBj_UcRBpOl70rB93Nnpn2GXAJU@M(=|aW30{-7fzk0kV3~}lg-2dQT
zs(4qzzlZeW_SpIf58LO96z?kde<%G6JUn{AkXiWFfbbArta!hK{}0m7z^ivWSrQ)f
zyF~G>hW}5}&&o5Wm>Q_+S5&+J{=Y~+Zm-qPJl^ucLw9;a@t%i&FX?CHne!VE9-1Fj
zycgi#NBUWL=J8fiyr&dzQ9KLve$vm%Gv}8&t*YO=>UpY*!7q`1+^+ST-qzoSPtywY
zgy1MOFT9D9#?7aAi^G3_^fT}(<1Gq9JRm&um-!WM3HT3^epa4EKjDGBfa0a#KScUj
zd4}ZhR_gSsehVqylJFlU{kUE07asSSe~%zv;boi!6>ll{kC1)_UZ|h>{?h^B;eNd#
z)nAqd?@`jv!0YOm)q!u@l|Q$Ou60TA-ch_|z<Uh5Ap<XDnDa}W!TKS;cNK5ph46eM
z@G{)4_3KC1v;F>MUU>MU1*P92;5`9e!N8OLCswcXfbi(}O7Rv0?@90m6Y@&J!yoBW
zyv4zL3cR6&ywsVjAGY`XinkniPlK1?cCBB%=gH)ScRqeUMe&vg?-|n1z^hl^0pa2M
z-A60l3gA6U`X%I*gm*zp?ij^e5xnO}KNGKfoNMWSXR&@`ovFtv-a6m~q#w6y{eZ)R
z1&pUNPF{ErWeqUodfmG4pC|ncys+Pyw^swgL;qV-@z#U?0_kVv>0Vs*OTxojYboCP
z@LweT67o`Kvwl;13L_P7H2jxHKW^9hh1r<vkr&=YExEN7Zv*%*lYUm7d3!Y=JoLX&
ziZ=oNE2N*5XU?xAJmj~b;!TAAD(Pq9h38rLwdgMVuHo=k>KxV&^JEln68zUlKW^9h
z)!W|Zg?DjFZj9nhhW~HU&&sptr+8x(ZzK4xlYUm7ML)%BRlJSizd`yLcr&}FPRHG|
zQxEQKt2w?;oh$k|bH*uN4*r{@AGg=)H@prK<%NegrYPQ)@ZTc+67mLwhw;0K;%x>0
zZPL%clUz);Bs?^3s(4$&e~0ul@ak<}Qs=RL;AItW8~8({AGe$I!`*~6zYmdDyv-DE
zTlnvieg@vu!c1J3?Ct18p`+_OAUr%?VspjI!+($TGw|w-kCN~%#r^z>w=4YjNk1#k
zJU&wAvwq-huXww`|A6%4_FDbS<0G$lJ1E}n@INH|3_Ki;L$|g<p`*7}KjF1b%Iu_g
zd%*vQ^fT~6{i2l!<C5^upLbNeJ>h>$`dNAA{afmSYX93=@m_)d3F*h}wfdR!%PZdF
ziuWq~Pf0%mkAy=@e`ina^Iryphnq2;P`uaRe@6NlcwuJd`j&);{n3+(_iy;0lYUm7
zrT<;X`d!hIds^{chyMlX$L(6bdimuQ?<vK51OAt!pMj^I*IaYGWI%ZMe$O+C_a^-R
zkbVYUz5GhTgMQB`-dpg$BK;EbQWvp)$S+X5x8c*lTZY?X>nFU7Gxd4Jdj~!}1EFBx
zg+I$qcl*>{hZzvwRroz-#T$Z8&p;S7@WSzH?k^?bLBAIi?_K!x41^&o&)lC=7qfoI
z?<K|i82-GZAGd4$>TRF%!t0&W@t)#+0)IZz&&msXP4xc(;bD9AzT$mKKIxZ`R}von
z&<Be58T<uEKP%6&{kw$q!;#I0iuXDE1xY_{uhlQQreT~H9@h6G#akHnelJA&S$XFC
z280Lv-0J@EMc~sjE(WbUEHJ0`d9NkKn@8~$g}(^tXW(`8+JA2+Rb>6}=jK(s#o*I3
zE;8J1($BP?%L@<s&8K*a!>4Cl6cX|Vgopg*SG*<Q({F$bCghcbhx7df6fXsT3DVEN
zljq@08bdz=GW?agw5s2Nink<u`VEi_x7X@te%^3ic+hVl#ajhF{RT+E$}>N2ctCha
zdnLtN6+Zn2$Y4TVNq9KcTv_q{1fPBbWXQk^H_YYZtD5J-r7mOre$$fsgW}x*e;Lw`
z+qHgS1<c#$yzsDn{-fgg@adT-1uM_IeI5|rH7&W@74Iqd^h}h&guIgQaMRaA>U!PN
z;L$Tth7$5pm$QD?;`{81ci|%Vz9Z?!?XmR}9`yS;7(kTom%0f43J3~To<%?5A-`WJ
z-o@}&1aHv98`DYyZR(7k-j1-=)&EPv!-uawQ@l&yuLRzZiC13SoL}k+*6%v}KD*)-
z;jau{hTFA%^^Onnig&W&T?&5{($C5>=Qp5urzqapi{klzq@RI@p@{>bE<B-_?z<TN
zDhbb-;amy^5as%w1D~FGGi2a}>ucV>rLJWCu)VrM@y>;x1~0?ywfYUuOQO8uU7~pB
z!CxJ`f|X~^Z$NmspYU?UJ0JcU;0;=NlF@Km5+2sKsCXB^Uz7Ap$V*+t`r*%9rg#^^
zUyJnPcB6jXwV$u%g@^sym5O)i()dJz^fU0#O=!FsFUSrE4>Md01`y@`avA)Q2nG#2
z=`W_^gOc!|-zAE7Is8%J4Ow~S{afmnRsD*J_eXrUvjw~ix101c<(C&8#?Vdb`=PhP
zr)Mq|3_J{_^2@{<5FX~ZS@He^pPsoiXyDb=Pk7Mp7R9>*{(9gIS$P)yuCD5LtK#|a
z*9R}d?X~)qKmV;cAj(&GV_I)hygT93_f-m39zGEbe<&;zlm8#6@NoZ;;;o0@j;8Oc
z3|e{S?LkTL((3o8*N49$>1X8)OREy3epS_Pb;TPEpPpHj;dZSbc59v3z4g-*sw*IN
z@)h1RXAQ;M06smls$k%Si(2*{!o&W2O~u;~K0UK)(8@C(UzLP+b4zY5#mm5_XI2ea
zdFK35zh?cwTU+tQz^7+cWw>4I7v6W;I%(`g!}d>jx3uKeQM_^R>6ujp11}tJ=KKbP
z2mM+VZ#;Z@X4POqUP*Y!Z>-|&2%nx=HI$H-`VH$h#+j<~+X+1SK2nC;wSG?c1ESHc
zSB~?-L!|Tj5qR|dtAc?i+ba`rK=F1~>-%H)^!=+rE6+s3Jr#<#i{kAApT2K4l#rLY
zraFG}inlNPDWo5_YyIl2Z(exVKJTh{`@!FY^t1A?zTtj&*oqGb5AW`#cvInTO8V8|
z8NP1dl!VvU(Yw3ix$rk5{p#?BCB{MO+NyqgDqb7>Ea}JXvGo%k^!tJ0oq~P)=A@sM
zXUY!uzX%V_C#&OwcCfb~{j5Cm{;kCGcj5k~y;Od);O9s`E6<!?>bk0a2P$4S{4Gg8
zZjY^>;&mzB4e+-j{j5B5egld(sCYNR-<tF@@MfcLl|N6RUo48eO2V5oZo>77cMZ;M
zwjuotyl}kXe&pKCy_5Pa>xcewr8+;l7XG%RAGgQWPk6}tD#g1F{&u9Fl^0o24hIzP
zmx}jW_}i0y2420-14_cXy(M?G;tj&zf%G%*!c@%rx6~l(hyB~H6z_WY^o+s`w`=|C
z@$$k$e!o_{ew>T!MEWJ<4G0gP2h39EGqd6EO!^skrq2PSIwj%Zjc&!81AiCN&%mp<
zy-!_V?SDOrcQAZ<Mq!5AwSLl{B`0n3!o!;d#XAH(J)^K-<<a(P!dT5Be?WLRxIRen
z4uwz8R~@wS%>AV#JUq{>SMk<e0?+>={j5Cm@nq@-)(^ZTzyP9rez_j}JrHELy;eVS
zetE@9Dc<_<_XMwC;MKdIa6ovt2)vr&jfVdN@CFUMu)gN}O2YG7a%sie0RCR!4H<ah
z`GQ5i8(BZ_R#&_Y;qOiQal6(pJT5Tjmsh+s6fXn+hoql@hv&%PJf^p;zXP|KL|y~J
z!$sgF6>kjuACZ0rUS0i!hwar;iq{JN$E2T?XVLE_*6*&C+|r6S7XCh@AGd4$>gAVL
zyk!(`9Q=JrKLc;lMEmdc4hZkhExBbCZ#?|{NIwIwUVbIvLBHh{Zvy<Oq+dc_>Sopt
z`HfJ#iSS+0kK1GGC%lPH{|bsX34R;tXW%)#IHSSi=*s`1{ZDv*Y00gqc$49`lYS;%
znPKiPCE-E8<rHrt_|r%~E6+UMQn#>v$ZsXZ+Zg_I(vRD<e&H&bd3oXCx6xNoyeaTI
zNWX-<0pZ<^-(yz1!{E;#{j5BTe!_!(2P)p-@Mn^KR-So%Q@65y$gfNBj)1>E>BsHE
z^cyo_95n3j=<Bb!-zP6T%ri^zj)dPy`WblPc#CvA4hMvX<I`@%I|}{*q@R^%xqc-)
z@Ou>RX!r+`epa42ztnB4ADRzRypyqS?;`!UUF$a%8!UbvUJW!aJgo28>U!N-@Mn>J
z23}qLgonR$syg2IIs9(YFCniaJmmKa#XArA^^krBUd>9XH&VZ=>NlWxr@$|ee%!A0
z3)k0N-@M|Trg$g7KZx|R@{m;dccfHua0e7`x~lKh;PsMz23|NW=;!lh95|eY>o3B?
zHus;3_bc%FNIwIwUVT%)XZ=u*zbf9Z!RsgexIMOh!b9SJQ@r1RH=Fdc@}g@R#si9X
zkK$be-W<};z{6oUu8Y&tJlYS9QcH^Wcg4FFyn{(U1Fx=rf2ivB55>C<yhBJoZm-oZ
znpqg<g@^Dy#rrLIhmw8?c>}`3=TY}7-XM5~k$wgq&SPl%Ondd38@fuu!we<GyB@s5
zNk0Rx-gry>v8vw#igyEeN05Hp9$P=*A-@L|?_uzcB>fWd287qwG3{kAfGD3IJp%tI
z1cL@1X0g6X!owV|DBjlC{v8e8kbzfMzuT+&EupsekAm%im*Mu<`UwyDy{hti4E{0T
z6|B7S`7iaGVXpz<;TZch#k&TzIhORZ@`hn5K}mRc=hv#f*TO%J^t1BJ+xyg?SU>Q7
zqj<$Ep8rSsar@pK)#Isp+k?E~Z4Rb?UwGWKI(&MCTcrb>!H2@{&lMu=n>yk9MSNba
zpRW*I-qyZJgUe58boou3a^AN5Wc_XJo9@@x@>8ZXw*01>Hn#kf2J=6vLH(v=8_Pf2
z==wMM_ZwV(-A4QG=uI6Oceeidb^Uirqxv<te52csMwg$`X#Z_+`Axs!<s0pP8#dT~
z8(n_w2LFDe%QrfH8r}ajx_ra?&j$5xbo@5De53Vic=-m$f1}&4Z+iPv$)AreOOKn6
zKW+QBM)O~@!T!71nhhSmjLtT+|8DsBv%%xPM%TYl{nu--exn;){+r%^ZZ^8H{<ql?
zjV=F8uYaR|zv2C7gU7#(?tdHJ|24S(X>|MheILJ#_TNU2pa1vw8@qnhsD9t{@{RVN
z{TkeVe$(T>(d$Qz?mw~(UO)P#mv40ak7}_0tbSBu$1e?^zcrYD<^1Qrynf#3{=d=v
zccb<HruA=BzlQ7A;PKZtz5Q(V`Zrv^2FHJ+<FC>4*Z=jO#*Y7*z5o1g{5DqqM#t|r
zJ^mU!{%`dBrP=%6|BfFT8$bW+e~sza?EUYk2G8GC2cVJrAHMg0f8EB;pI1L>-Ny9)
zp8x%3-~ToH@$WZ%{MG3B+yDAcgZs}0ufH@}zi+z#HTw6PefdV~*XZ^4W*@)Je*D?!
z{nsNKynlOSgZB?KeE)Z&pFcI2|4|KYe@BjN?D+Kv2-wK^!;~pUG_-u9+s{U~UyWYB
zY4rSkN`v>WH+uhAqksQ@%QrZF8mxb#`Zs$2;$}xQHvTso-RSa-ZT}j*|F6;Gmqy2L
z!}V+M{@q6Zexu)iYWC&VY*7FIE#KJq|EBe8^!TIE<r_YJY4G^B(ffa!{rX$B!S6pb
zy8Zp%{=2d3XN~T^zv=U@Mvq^<>HF83ef%HQ;Qdo08(jW~2ABU||7mRd(dhj%|LZ@E
z9shmbk3Sl{{yPO5;6|<=HhTSbQw*R+{(hs^Pn&)GHG2KE(d8Sxew%c8|HAOOK=OQ^
zqaycjJTqdQ^6$_~d-VAsx8XUA^gJPH$Ne|!I_mi&(e~)_??w30zyDdJJ-Yn7D-6#s
z`u7)%v`3d;G}0dZ`>9BKbouD|MgM-ZJ-Yme2tQiCl_Tx4e2KpwvSy@xlkhQ3mFK^t
zIhOi3{Jr`j+e}1Xhq<=z*W;&xyQg>d^>=np>!&))OZbwXZTYX`|6jKw|LGltj_&Cl
z-P1bp3s>@LqWot~n=`cwB`9y_n=t?B)4T9f?6YRIb*slcH;ev<bWWe7w{K0Se@|yO
zJrvP4KELhbx2vyXW>f3m*1vx<=ilCQNJGb$|JU31uj3bbx(=P$)7`xNzq7lenfuR-
zuAa7jJY}<4w?F0l@pRs1&3|_Hf!#d^H*f!+*4aCae)X<d^`F(#-9Zm^Zrc3YrnOC<
z-rKbKWAj-)Z)sNjr*%Snb$;D!$Dgyp{xiw>FQ0E9%h~Gpf8E|Tt*f)YV^-7l|MtSP
z{oC;CYBv48?Z^Lf+Iq_?(5(4S$FCeU^Z4!WoYnOC_w~2UD)8rg&9?m{{mc3HAJ&xD
ze{l7oqjyeQmu#n-QGZC^wA;`2&hWZnvu;1J{ijuETK&6c&+6#yoYu7Y_jb&fHoLd4
zr?*MBpS>M33%xzlI;PL=E$_#gZTo}#dkXk@+-4oWoPS$CdJuh1Ew5&|{*U~pb+z^N
zHS7M1^6%;DYU}UBMbPHdzyHueM>F^T-X2`QYU1(R-G)QZLff>a+<wlU-8sFP&%f~b
zbyr_^^B({7_s!{;MjQVoz5X+^vI60Kj?K3J>hC*fc1Q1_&Ak1Y*3&(sb7u1%e@^e5
zMtk2jRH6E6mgAoT`g*$Ck#@80zbO9#eFop8+n;{ifHJkOyRFc-e-GWqA+KiXKiGd&
zvTw%Ye_X#B*W1>OAFOZA?LV$Rgy}bH|L+UaZ`SQke_#J$&D?+J^NVKAALDmYGw0vi
zF|DT;_f0ly|DVb$(5(4S>+A3B?4H@Q`nPo-+SK>&A^*1C-nK)V_4sX0Pjg>?K<}S9
zvx%=icg*SN?w=}aAxF>E@2`B@?|<mRvFGfznH^L2$L;f7C<MQn<@mq+`EPgge*TN?
zPe)hBEOesqdzQ_nKl1OUYY(&uZN~F2(jVtAO}qb{G5z4m&}~NjJMqn_L#DRPs;tD<
zY=6?+e#`}PmAysr{~Pl<&PMY%&Vuk$2Z!GuOivFj&u%rr>~Oxp{?HDvEwIhA?JdlO
ztK^Q8;n+KTIKRkon3dyv2#5R)Q&r!ZL=!dyvi?U5*TI+f|I2lfdftwjE(AuJ`El!+
z;ri#<7T6B39b_x@EO9)<R?<!#60TRYo<EFPPgzcQ-*%hl^)0ZKd^3?)-j8dgo_|6;
z7sf&Z%!^ykL7u<Fc8G22(6Bxkwo=bL$Jy7{Eq{+#@^m?ty2$vH_OG+D+}F+bZD}v^
z`uY(ayhxlhU#aJFsOKVBIAC7fdOB}}bxE_$vUS;dY^9z>j(xV0Z@{tCMe4aj%<;LH
zfSSan-wbujvaO^YjVlfD9I5B?s3-X@^WxUi<N1qheYOEx=dCbrsb`wwEL+La<v3bT
zY5zJq{_AG=mbDjoef@|IUL;PMw$$@g)RX#nz&xoZhDZI^+dMwmX4$%IJ+@NMBF8>k
zfd?E%>$waz8ZuU8UX$GoZ_2W!_HWyJ>34X1vaRG7jVlfD9H}QA<1UUxT;@qVzmDSZ
z{6)4t+kmYz6d9i!XW0th<ydMGr-rgD*=F0`7I}UBhz?#P*6SF@mwLX5dM<$l0_Mf7
zr}Hk4PqtaME?bXnWqfk%vy~hJj-&P5CFcH9mJ{B$-KO8;@yWK5Z#1qn#B-#cZ=s&F
zZFQL!x1Ju)Uu5gE4cI#GN5&_|S+<g;%dylY>G+i8g!gT?MP6S&qJtNSldgx<^KH~~
zNh};NFK#`Z4|sgC&9ZgbdTcA>lVhK)<Qs4-b&-1R`duBLAM*HQD>^vQxY7{Mk$Rpq
zpW`frMO@~^t*6KH7uou31Gdgbk@3lKmaSyzavZJa#F+IIPP5n|udg4`!HdL6*GKC4
zGt_fwEF3T|Zatlkd3>_XvUS;dY%AlFW1p?$8*m)0=T6_zdZs_&@yS+naH4UgA)X`k
z9K^-jWw40LytwuBc>W?=pKZX_`7|;<InJ_`EM1PH^_2GSnl1AB`Vk$xNSt)NrJf^j
z@pf4(9563#J)O^Ze6r25b=i7sE8~-6pRME@a2&1Y`0wiY{G7)pThYOZ#+8P6j?{A>
z)N?s3;xaF8Jw2Yk$kt~YuywwOj8BfUY$Z#V<7hosjd?vmmJ{B$-4=O${fG`;Bu=^>
zQqLA#j9(rL2h59GPv=V>pKP;iUA7+E%J}5iXDj&z97|o~_;$0H<8y8SHHl6ChsP(|
zO4`x5(h$#)dai?dj=&-=^WxUi<N1qheYOEx=c~y0<T%S#vUE9))>GQQ&dPFMH{Z9V
zy~yk9M|AKaangLHp6jBXD`4S(d2#FMIO~-^_e!(PvUS;dY%AlFW1p?$8*nUjk?|?*
zUuWn4x*5J@?diEf-Lh;e=|$s8Lp(?7xgP4dA{KF(7q^}s&tGKgvkllfbBB3LJ<}X#
z*-Dl!$Nz0TaXF-^UPWGCKca&diIc9B)N_5*b0sVsFfVRBoq2eCvdyw}*?MfHo<)v*
zwvunaakQR`$2=YqPP5qbygWYHR`QO<m4<kZ)N?fIxiS`UnHRU79?xH7>$45mI`c)w
zC&yW~lBLUWw4T!bb+*}#hl;$uenbZ^5`W!#N86>I8=#)6pfO-x+<H3m^Y~<&W$Uu_
z*jC0T$39!hC*U|*&s5CuDa#4(+iuef@c3j~$u}BT8sa%p&ka$}Rk4W6ytwuBc>W?=
zpKZX_Suip_InJ_`EM1PJE=iAvnr?jZ`uY(ayhtqi#I;h-4C=WW77Cabx1P>IJU-cG
z*}7~!ww3Y8vCmfW3^<O~^R_kO-Jh_rfSSan7v}NFwvu)<t~A7Rq@H6?&omZsnHRU7
z9?xH7>$45mI*UZcC&yW~lBLUWw4O`F+}^GwpeC_JUSB_=gBOXDu8!2R74=*l3kS@L
zTTf?E9-nNpY+be<+sgRl*k>#G1{_E0DeYfpo9%ohy%>*Aww3jZ#+8P6j?{B3>WNL8
z<1#O9Jw2Yk$kt~Yuyq!Xj8BfUY$Z#V<Nvmv!f6&;<n{F<I(U&d>H0`L$Dy8UV&Q;!
zaqH<U!Q+!{maWUyV_O-Y9Q$k~-+<$2JvWJY{Z^I}-nZSRQ#?M|R`QL;m4<kZ)N?%Q
zN&6s|d2#FM@%%-$KHGq;vt(p^a-3x=S-KocU6LN(3a43Yk=NIc=-@@-r0XN~oQQg^
zjfDf|#jU5a6pv4~S+*`)k8Ndqa_qB}d;^Z7^~`-&$LG>KKG}*6PBgAG#B-#clTgo*
zSj1&s+<JOEf03=vHel;46B(ZzXW2@YF2~V&O8eJYd_AtIUPWGCKca&diIc9B)N?ZG
zISLC0%!^x3XIUPfY_n`#wjSHc_~h7UEBOW-|F`v|d%>IOm0pg=C)-Ny(YVqO&yjj=
zgnG7M5tn&!>*?|QMYcZMfUUE9WPEa*Wh+^_97pT9T+H*&rn^6$*Vm8e;6-B5C$5!x
zZj5@agM|X-#jU3^g2yM@EL)eY$F?#)IriB~o&m?vdX9)$PrBE%nO^A?czm+0<Q|PH
z4e=bQ=M>a)T`b}<FK#_Op1;V}XB)6}R*Z~Kj<akfOPAwlJ=3_a>AQHWTjcfiBRY7I
zSoDc&rJi@9p6g+ufO&E2>8!-#lWmr*%hqFC8J`^cY$eZt<7hpl{p)PAU5`(%%;S@7
zW&NUYr6Hap^*j^Xwe_)x%e=Vt^mzUvTc2&f)>$PoJ~_^^l`LJ3qxC#H=Jr-N&0>qZ
zzJ5dpFA^tRAF1b!&}}pp4wx6Wp3bT~KG|m3x@<kRmGQ~3&sOpcIF8m++P`Zyy&8{C
zww3jZ#+8P6j?{AjI2&LQmw9pP>GAwUwm#c{t&@(7PmZ%}B}<p%Xg$Y%N3SOod42te
z4qhY{ed1cF=LT>##6kh{;?~nyoyRBJEL)eY$F?#)IriB~o&m?vdQOg6PvJC+O|QY@
zlWisMXk2NC=SV$ogOkA`F7x8n)8qMzY<;!?TW8J4_~batR<d+Cj@EOV?`S=XyuN-!
z2QLzfK5?zob2*%lkHJC#^WxUiS&PRf+bmm`t;e=9J~{T;N}d77(Ryz69j#}2Z62R&
zMF%GuR~q6uQqS?IXDb$QnHRU79?xH7>$45mIwK?FljAH~$<pOGTF)Nj^_@Kb<n{F<
zI(U&-^oeVwo?Ae-u~;ZzUfg;*qj-F>&9ZgbdTcA>lVhK)<QZ@rt>?xu$7i!$k59Mo
z_+%?OIMKM$5YLf%cA%d48F$BJUfg<mJb#g`&o*G|tP>fZ9B0`|mM+K9dd~Q+j!#}+
zKca&diAA5dR_f_NH+m*hz`VHibk^nZ$u`T@W$Ur6j8BezwvuPSakQS&{&lwLt`Dup
z<CAS={i1QDA)X`k{2A&w0gJfIi(5~R=P$DL*#>N#^&{hx<1Aar(&ac>&oME#x58-_
zTjcfiBRY7IIO+OGJ+DPQCt~4%d2#FMjOOvlHp|v!>#?njPmX=Il5fCqw4T!bU9;&8
zczm+0tY0*)G{keHo{ylOldy=(ytwuBc>W?=pKZX_*)TFbInJ_`EM1QO+j`1!!uz({
zBCoF>(ZP$vN!LT_`3dSd84Cx@i(5}8!{d`}maWUyV_O-Y9Q$k~-+*JO%Xje{Mb?#U
zdJK<Gww1PMTxp2sNIe(C4I3L_5tn&!>*?|QMYcZMfUVOS8J`?y*-Dl!$I*H|jfc>E
z7i(MO_4Ok<c#&B2iEE{vi=v+N9gu)|aqH=f<?+cj%hqM<v8{|xj(xV0XTWi^o{J-|
z?_xdE<9K|s6&;*tTxp2sNIg@~Z3-4~nHRU79?xH7>$45mI^!ebljAH~$<pOGTF>ob
z9^cAx!uz({BCoF>(ZP$vN!LT_xg_ej2^J2R7q_0y1RkGkvus_q9^1<J<k)8``34+I
zU6PJTSx$K0cAK8a<CASA-)LNEi04Q>mqI-^#Ud{A;?~pS`HO6QwgFpbQe=E`oMkIn
zx*SVgzU%cY^7{G_9lS^^`oy(T&!th%&9G3wytwsrCiD1Yn`P^=_1IR%C&xZp$ur>i
zzpZD}UBBIk$0u9S!HLF|hIo$D^Cr|Yi$z@K#jU5u^B39rYy-B=#*y*Kah9!Q>2mzv
z)>D=f-nZQrd42te4qhZqx*k%`0v-;wITj9>7q_0y6ds>!vus_q9^1<J<k)8``34+I
zU1WQ^XUyXv;WUd)Z^Gk~Z6)t$Txp2sNIf4$J-5IjF7x8n)8qMzY<;!?TW8bA_~bat
zR<d+Cj@EOp?`S=XyuN-!2QLzfK5?zo^HJ0@hlK*>#jU5a8IMo4S+*`)k8Ndqa_qB}
zJOhrS^_2GSnoVbUe6p>qUo@^X#B-#ckE5PjViA{laqH>v{6)4t+kmaJd1QQYoMkIn
zx*UIB>RIIV^&>iXky!MJYo(q~qMln}p@4aD>*;L4<CATct;^P9TN$4m`)no8faC8=
zJ<~ZJpKL`3CmL58;yF^!U2wDJ)>y=4Ufg<mJb#g`&o*G|Y#HV~hksu%&2g5kWa&yQ
z1(k;HsP%Y#i?ZJTpVm|Axd-aG4VXUj<JL2fe753o$~Mc^Wh?dcI4-i4)$k?$p42mt
z_1!wuO>}6~N<DvwdTt9?n)z|-ndSLiwjSFeTc54eGvL_SX83xfCH|h&Gt2Ai%6d0i
zNM2s*xi9Lu9p3YpAGe-Gp5JF1NPgS$cx5Z~%yR6qmDTVh{+`sc$m{FNdW#SZTB&Cn
z>bX5w0rTV5)7g&e#x~2=W$Uq(dKNkM*-H9>#NU&8I@_~uY(<Agt<-Y{>bV18S@}CL
ze{b34`8~Epwm#c{t<=-mf!Bkr<e!!Jds0uA*VmKv79kq6QqKcX&mF-kGC%J4^m+b3
z^4gK>#Wu@U>gjUqv6b|T5`RzX>GS#qvfd&@gI4NUKs|RN)=pfHnBz0e^Jm$*Y(2I`
zwo*@@<A7}?{hh<V+f+Z|SxG**ljil!%6cb97?+TG&PF|V#vCs5<Bm^{=P$DL*#>N#
zUBWs^J<}X#*-BEb#NU(g>GAp&WxYj+2CdZdFw}Dwuzcpnt!E(l<au1P&9ZgbN<BS}
zi)<x*U*hjcJp)<aU3t4CLNsWlo=2med9c#Vk6X_y&+oGJ*cRFPY^9z7$IfoU*CQ?Q
z_oSX#USC(%yTLlj;!@A!QO{lRrpNra^(^xIKHEU@+nvWNTd8N3W0$Su;Ys{Gsb`Vb
z*O&DcAsVz&&y!G3`c3hG`El#%?7?+on`P^=_1H>1iyZrGCH+9+?@2wKJy|!lqC=xr
z>Uk>axjSH4`8zR>Z(W|>V_RhFvkllvJ)Ix$da#xJvl4$#>gn?Oda~XkM1xl9c?Rma
z2Utbs#~q(O&mTx$dvU$kX4y(TU5-7rl73O*?@2v<Uf)30TZCxPN<Gg(J@+Km-dvBE
z<1@|kXW6=JJ+?)*Qcs`b;Qz6A9&nPQ#ogACGa_dg@{#9`XyA;FoDtC^do;T{yF1(l
zF|&Ke-GLD{U;_q3&LE;qmI2#<0Sk<6urZcxz%~xZ<cz>38~Cb5Pj%Y9+tM{RAG_W9
z{qCNArFvie>tEHHo}N}O9LRrEciy@A{gZqn%X?tNY!9rSuaJ5^LY7Kc-~0T`h%aCT
z8yGygtCQ7piE#u?ePZJ|d46W(TUg#^#Ei6hzDDZ#8?si#`rdlB#D|aR>b4AHm_n;(
z&bWjozqaw5)Uzeu@UdK%%!nCj^?ZZW^O3UFGV6Qm855tv9G0+#R?n7k_?zQ-EZcZa
z>KT)7YI)CCN1JZ-e6!T^QL-~<eQ!NW;%nH#@V9urLaS%YIEAJoxAC0Rvn1cz@-`!8
zq}B87QqM=rS}p5)>lv<4HyFbd=FsX{GOnS?Z*4p$^$ZhygXY7mw0gcv>iHO1EM|Rg
zJyYUySi%~%(CQhkk_R;Xv5n`Xo+<g}mbV!(BdwnAm3lr_)+$-wThE&K7KS^h7mT6R
zGi97ZlV93+PU=~cZ)<s*5i`>2`2nftZ^~NX&aNKbdM*<m!xZMQgjUa*aSKg;xT`zw
zy!`&TOun(@ZAQ#Ut7j$k{4H54WqofwbK*-_!xn~XU7f6+%Zy`a@>3hn$@4QO-_r6n
zBW9%4^P^JF6<MoheQ!Nm;=|osXJHIeX!Xn)m(b+bHlCAuw&WY`;ksl-%t))}$E2Q#
zthLPg-g?Hwr!a>ltfAGjWgPAu&tuufb5hTkd{fJN#yZ+`tLG=Ao~yDmXMJxyOX6$T
z!f>7EE3|sXj8kYjavRS{JxlVfEpIboMp`{<spk$^t7Uy}J(nKW)h&Vv%wPeno)zN;
z4&<N4^L0+@xkSE^<vlQBwg*<vPf0y@%2EmId+V7IU%(1BFxc1C$?CbpID)1=vGJVL
zGb7)^@-`!8q}B7^q@KHEt%~)%^=wSf{;qCIFoFrRdS;9ZX!0u?&q+NS%l81+B{O11
zT0Q?u>bWLsEwR40o)Pg0%wPd4X!UFu2M5RVShDe))H5RA#PXi8jyB!u`30%xZrPc!
zzPFwQ@fB=LU&`|pT0JAi2{av<jpwAE1^HH%w;3@bt)5?zdhU_68rJvLGk838gAq(%
z2Cbe2;|iMm#>R6}&tL=Jp!qN>t)AbMdhV6QB0G=s^Cl_rIV@ofTWIwRH@iHRq3Mrp
zJSX)`$v3yW&4?Lk^=ze{>#|nK`rhYfO?(T(L(~h#(CV2o&Y{UKZ9FIStjV{vyv>Li
zY4!ZR)bnw&R(P1}fb;peOneMen8OlUJ!{4-H2LA_-FfHc{dn??EpIboMp`|8EcHB1
z)=F95`~1v_FJTQ^7@pD9$?CbxIEE%aweg%hKXdXeEpIboMp`|8CiUDWYt^jpt!GPo
z_yn%AFor3#dghEvX!2_t&q+O7@(s`Ax@1PoNUP@smV;owthLPg-g?Hwr!a>ltfAGj
zWgMP0p2xC{=cJx7`KFfljCHi>R?mw_JrBsvob|o+EQzmS3&XQ{zCx>K%s7RnBe(IK
z)Uzbt+VVCdW~9~g5>n5DvR2Fb-g<^lq;4>VDa@hOvt(RDli%8SPU;yx3E!akFe|N|
zmz8>^vRKUe-g>6Q=dgq|Y@yXN{B81pra!jvoYXTV-`w&xBW9%4^9oYW$IDtJ>wD{2
z6W_w{$<zzR(CV2o&Y{UKZ9FIStjV{vyv>LiY4yB{)N@1D3bU>r-g+(*AHx*pu!L66
znsEzFe)yE`ymRyWC;7&fw;3@bt)ADEdTz>EDeHUdnG;{a8n!U}ovuz+&t=9jH2JBG
z=j8dBlW%Exn-Me8>bWHKJS1z?tnaO7@Vh*pVFVMHL91uMxPm6XvGJVLGk7Y#LGxi&
zT0L(d^*k(#MRp$N`+W)V87yE08))?mo<<(f^hY+HlX@oPn_1pw#Ei6h-bCtox~x^O
zzW4cA5#N}ur&BK&L91uNID;m?u<@MKvm)Qd@-`!8q}B75QqMDVt>2>_&gbVc@i9zc
z4ohhDtQoh^<cGiCop)|t50P(dd7BY4(&~A#)bk0lR?7O`=VwlQ32WHG@EKj5te(q^
zV`%bI8_&t}Gbi8D@-`!8q}B6wQqMDGt(x_{^=yd`pV`%I8OAV$R?nPq2~B=&<2k8k
zOTOW=xGtFyGt%mLN2%voveq)|d+Qk!pTZoLu!dI8mT~w8<9RIGcuwjWlW%Hy&sayB
zZuPv2)bniFnX|sPo+a@$Y+?9pp0CjA88c3y>Bwz7C-p4Jx3;{^h#6`1yqnbXiLzG9
z`rdkm&!KKGhAGUU)w5(=LzCaycuwjWJ{RAh`7kT3p7)Y^K1mjfS>IdFl=vK$u!b$P
zdWJc9K+_-Fcuwk>l5cK#n-Me8>Un>u=WokeCF^_ZSrgyF@Ojh=#?b1SGR~pNFKs+0
z^{mOawY<%U8EN%=u+;O(vR3%~t{&ccM#LvDg9WUh)w5w7yr4Ue$zQVZoYXTS-^B7Z
zBW9%4^Wjp@OxDU+-&@au_zE_r?}hXQw0cI26KL`?8_!8S3-YZjZ!=;>T0I{v^?Zt~
z)v&&|p5cqQzQP!$Fo#ypl5q`9erw}7sb~0Ne1qo0th9QrNIic?7K>TmThEmE9G0+#
zEwp-uFCh<T`ePf<Nj+2Y%`I;;Vn$j$cS=2fSJo<7-&@a`_!fpQrCu<GR?n1i4o!Y(
z<2k8kO}?$=ZAQ#UtLGl6=Tl{^@MT>+y!BirK87jGVF|6CHRBeV{P5-7dFST+c=C-c
zZ!=;>T0KvbdOl6oN?G4q&w}_0Hm2_t^aZqfMvN0^@-rLH$@8-y-^%hfBW9%4GnINi
zUDj$?-&@b%mDCMJFo7AgdKQc;X!08y&q+OlSK%8pA7-W1^RU$O_hhli&g1<1XG(ky
zOIX7eT0O&8cX=#B(;wS-PU@MGZ*F;;5i`>2d6v}k_hqe;^}Wx}#`L^~euEKApw%;D
zTtJgw*?3Os*;u|s=bQO3E3KYSmU=!z7F%L{Z#`q;Q<%dN*3jzNG7eump2xC{=cJx7
z`KFfltc7I0)$^%R&u7ZMob|o+EQzmS3&YoOU4>T9m~jd%joij_QqPinYs=e=n2}b`
zXGlGtC2O^;@2%(3>$|!|Fo78?pw+Wt+`xhSH}HI&lX@<ZZ)AB7jF|0#)$?ak&p(i*
z64pO)_YLp%pIiTc$GNYc6JNp_wlI8SS0}6IGUFK9>Zy(Aq@FqXmX@~}F(a*>7da^i
zo-J$DtnaO7OMLhyuCFkLDYSa#j7w<pYa7o=JzMe(|B&mF88IWRo|lw*K1bGCW_@ox
z6XG*izzR0d>KXh|r+*2W{K&?0QqP2ZGt1kIn2}b`%St_;D{B?3@2zJ=d}F%aOub+P
zt)2<v44VAH#&c57ihLW(+l-izR?jO+J#$^_EnPjl^;{x8f(guE0j-`D;|31ozm@CG
zoYZrPd?U+yV8m<>te#hsdOlB<N?6}p&z$%Y*06=)+qybgJ(n5B(A1|ko|Ag!<Xc+a
zX2guNdR|-V`FvTcW_@oxTjImFcXeBaF-)P=GiO{vlV96-PU_i`Z}<+bOJ>B3w0d4&
z>iGg$Ynk=E^^A#6VGc`JL#t=YIDF@L9?LeKlX}MFn_Av8*3qV0Jx`Q+zEF1NtnaO7
zMSNqr%C1gJFoIUkgmDH<M`7bRsb@vLjpc1d%t))}Eu@|=(zX7$tB1FqOT<Soff+2I
z)w5#Uz=8aC@qC?=dM=S~WO)yanC*ep^JJ;#i)E>V^}Y4Xh%aCT8yLL1tCQ7piE#u?
zePZJ|sb@yMh2?EV%t))}?WLYCk+mw;_tvv9J@4T<3nQ37t7pczfF{4P@toANv3&o8
z>yjBUBdwlymU_Na)>>kHZ#^U86PUpQR?zC%Fb@87JdY(C&q+Na@=Yx78S7}%t)8b!
zJzpj}GuHRkvmm~Ljp_R{p0CjA88J?v>Bww6C-p4Ix3av=h#6`1yr<Oj<+4`8`rdj5
z@1<@qf(gu^)w5t+L6hIucuwjWybs@?`7kT3p7)b_zCsp@Sl?UEg!l{=u!0S=dIs+&
z4`})$8_!8S6Y|Y0Z!=;>T0LW_=PPBcg7v-itcY(+*Pl}_7(uIN!Z?E_zp(L~)UzVr
z#_~2JW~9~g;Zo06=~^G?>fx>D67dmCU<M0l^{f~-a3KGK-FfHb^VCb^8(H21BW8PG
z^?bC{^VPCc!usBNX2ch;f(;Bl)YZxAxx_evrarOpoIF1>@+~ZHGh#+sJy)cjuaUJX
z*7w%4F+G35brwc2fmY9qaRE(!W#c)iXJh$RT$jv<8EN(0DfKL5ttHm?)-xhLff+1d
z1+AV9<KV;Nc`Vs@PU;zvZ(@1RSVx;~_1r7<e68%vSl?UEg7^wHrtc#>U!m1AVw^zJ
zk=b}o>RFI)WqF$sGt%n0U+VcfS*u}vZ#{#*q;4>R3Cy6?vtV37li%2QPU;za6yKov
zFe|N|8&c2L%VH7hd+V7HpTPoFuz^<3;IGI7n*PYfb5hTQd^5}2jF^#D&(o!zZ;-VL
z*7w%4BEB(Qe@(q$1g)M4;|!Yo!p3t_&x(8-%iD~Ykyg*MrJirpwf?57hqs<f#78iJ
z87!dHvtr!9f&7nk=bfA1Kgl<;yaz_i_Q2}-WU1$yWT}Mpz4gq9FJJ{57<{~|lht#H
zaRg0$V&ge^erDubSl(vDjI??_RqFYNvR1|V-g-8s=Wn^r!U!hN>X|VvpvkXnJSX*R
zEZ@K5x@1PoNUP`XOFjQc)>>kHZ#^U86PUpQR?zC%Fb+O3p2w1n=cJwy`6ib4jCHi>
zR?lZkJ>M)lGuHRkvmm~Ljp_S)p098tJwgAu^&n!LK+};;jK42+)zapo%N|?oG+u1l
z&R^ku75O$3c^4DozA0EepD(_>MJIo<tAn?mOT<Soff+2I)w5#Uz=7U>7=ON64PWG}
z=dt#*j>tEe$a`sGJn+Z)!0P!D@$IcLCu2QtJqzM1*qEN0=O?s!MvN0^GP8-X)$`@Y
zX+10QZ6@+ACdTHEmsZbLif?bzN&ndM`AK{P6PUpQT0JYq4IJqFr|Hkn*BqzyjL0{c
z$a`sGJn+Z)!0P#0@$Kz0Cu2SL^ON`rHm2vFd458BJrprcpvlZ^e5_qKErFga$hWe*
zd(J<naI5E=q@M4PpoaCm^$b2m-CzV0m_e&&!MK8^tFiH%)HC=rzCrV0R$4vZD)oG)
zEEci8x1I^{87yE08))?mK0_YR^hY+HlX@oPn_1pw#Ei6hzEkR1%31~Md+S*d-<Ym{
zp<Xb8R?mcS22Flp<2k8kMZS&YZAQ#UtLJ;9o`0-s{cBebZ#|cYk6;2bSU{_1#khe3
z`Ty3PcW&OlCEv*M9vCs(1FPrzq@M4Rr4rWn)-xx*gf(no_}Q*bR?lU|F*Nn5jpyX~
znUim6d7BY4((3skspq?8t(x_{^=yd`|Glf*GK^sgt)4mK5}N$l#&c57mVCqi;JRc+
z%t))}UrIgSBWo?QzPFw+@hQw<32SKeY#E3DIiAO|jpwAEG5MyJ_l$M4=~mB=Nj?8W
zcIK?_t!GJm4O<xg7tdE{^^6&(&~)TBo|Af(<Xc<bX2guNdVWIc`KPj0%lh7WhM%Ku
zFor42q1CfwTtk!J+IUXt88-L^&4*cO_54Sv=by=9G3$HlnG&DF64tPVR?qOi$pf1H
z*v4~G&y;*~%iD~Ykyg)7OFiE!Yn80;t!GVq3&YP-FBn6sXUaH-Ccm`toYb=>-`4Ur
zBW9%4^RrUV_sLq}7rJ_Q>$yyP3{#lH5?VcL#w|4Y;TOB}&dcwg%j6qd-e$y%w0eF{
z>iK?ID`kCeJ#*qqSi=^EU+U^)^;~8gLzAD{cut<5Ir)~Bw;3@bt)5?$dj7erRkOag
zo-Og=m$}Zu7^cwbnKLe-$**lZC-rQ}H~b3MB{O11T0Ore_56UWwaoh7dd9@3Foz|q
zq1Cfx9Da2?k7XOrNj+ooO)c*k>uA%hp5K;weo%JitnaO7Nqh}k7=DfCE3|sXj8kYj
zavRS{JxlVfEpIboMp`}pN9y?@S*vAzZ#~1WQ#TmH6z0(CSu(Dn$!~2uC-n@!fp5@!
zn3Yz~A4xs`LKcfz-&@a&_ySh2fx$OB-B!;f#t}6AiH+x^o*DTTmbV!(Bdwl4lX_ON
zR>k_>dN!u#TV36jU<4Cr^~@L-(BxM(o|AeumhZPa-^_<uY4tqe<RJL4EVjh@-g?Hw
zr!a>ltfAGjWgLEIJdb4?&q+OF@=Y!8SqsU0tLKHKo*$8YIqQ4tSrOluu9oL1jG)ys
zVVprrqp<Ou)UzVr#_~2JW~9~g;!@AQ)V03b)x%rQCE_EPzzi19>RB;v;6VQOc)rd_
zJ(tKgvb+aI%=W<Qd1<NVM`fvm^}Y4Xh%aCT8yNggS0}6I65|M(`ozX_QqPQh3(MP#
zn2}b`%S%1~O4h1a-&@bd^!zW^Ss1|tT0Jwy1vL4UjpwAEjph4&u1jXbjI?@QS?c-M
zvepvod+QkypTG<ju!2_4hH>zN@jRAnJSX*x$TzXPXRM=5w|ZV(>iIXaGh=;kJqzM1
z*qFW_@_dC>&xmmXO-E+qIjLtszLn)|M$Aa7=XIo>ACt8j*7w#k_z`u35lmnPt)2zr
z3Yz@J#&c57;K%p|&4*cO^}N2+^W(Bu#QNTPCd6m3fE8?@)id}Bc|g-2*?3OsnUHU0
zd7BY4(&~9*spsFyS_SKS>sb-sn695vFBn0qXTmsxCcm)poYb=--^TJbBW9%4^X5{|
zztgq;ud9c*o=e0>Fo78?pw+Wt+`xhSpLOS*m(M3Gk#A&q4~&@Yfz|UQspltTsf6{t
z^~{JbU<Df({Jg7^)pLn)1WkQn<2iYLX5?E~-e$y%w0d4B3WC3vwJO$s<eKl_b8Sq|
zFSyRa2qy5M=?Mo^YR0&Lx=*)iV%+UL_y1ow>b_$^qmH#UXa@P--1YC~p1TvnqgGG#
z=998}Dd^Vs)-xhLfjKN;4ekEymT`E(cpfq16M_@AN^`;mx*A<VblKx$t!6`)k9?Gq
zZ#m&ZIx#+S=DF{&dR`_9f`5?En)ST(Y>5vq(A8%d#xR9e&zx}yO<m2{YIXtYX~&N@
zh8M&yX!*4hzSw>*t)5qmq?R%%W_@oxGvW(a!3G8w>U7)nc8PHWO=mJOw*BYED@EON
zSbM<w;N0SWm5ycPTbzfy&n*wDr+V{`5?Hamx1NpZxo}svB^bd3+VeAGTtL%SF}Au~
zkmu)BMVCFc*yDm1!7pezH50zrelM+_*NlSTpJY<R`rdjb#AmR86>MP2_elj8B@bvi
zqlvNYKVF%TZ}v;h+v<6}C<y*p%qv*mThEI4#&lhbdcg?V^D|+bL916W7<bRmi=4Ng
zpZ4^&Wq)PaSiTqU{4$}H?YA90Y4toY3W86`+-27H)-xtPg*hx?4XvIn<M0yWc`Q$i
zFGf9YCc5l#&(waU!&CCjC-ROb#(k5pdfq~O`?O3hS<hR~n)numm+b1a3}a~ZOd03U
z^p+Fji=VgWr!Ds@&z5|{OO5w+J>f&&M68}CM?vrznY_&U-g+j)XRv@3Y@pRMxOAs~
z37Xz$Vtl#dyq+2P787|V6JztoORMK?qagSfnN+c!x1KHW;bnMU!WgE|>X|bxq3NtA
z#+N_N>lt1azo7ZhPWWK^y|jAXA(F4Ekx4P@d+V7IU%(1BFbF%{R?j8I5j362#P|xw
zc|8mAttRr$CdTHEmsZa^MM3ayGO1xbZ#~1y(PuD*Da_%1eqSjW*U)sfHs%%T6_4|J
zE?>U$Er#YpI5D>UURpix5(U9$Wm3v|-g*|qSFkZXSLpOEL91uPIDw`!voZC&(s5qT
zihLW(8zYK|vH9br)$?vq5d6DNx?)!cZ#|cZk6{XPSVF63&A5f8GrUrFUVG#2%Ex&<
zWAaTcZ;V);7@I#{T0QR@1;KyFq@4A<^(=|6VGF}6clB9@R?nDm3QcEjW1gSaKF;e|
zlW%KzV?;SIHh;XddOk1;g8!6B;Z=G*KZ%cF3UgROdp%S$ZlUQ6uiBkg{$94V>pI7I
zJ!A4sq4}^pF}D3)T0I{b1;KyGq@4BK&rjlO*uwB?JU`*!`N=qirZcxO&(Edfyq-1r
zww5<WloMm~$4jec76rlQWKwwbp3hI>W0=AmmT)JZPpBEU&~%2^=+0~7<DJXo8~>8?
zwt7B0lHXOt#FX{D&(ECr64tPV;WfKD+4XjraSW|)>BRT}$NBTKB;R@>?|fox{&;Ei
ze0~%J|1FbR*7MdgycXA07{e6iaDN(X{rf8=;~JXI*2Y|KA9kGAbNSkxZ!t6<!ill%
z_tNV5;wT6{FOyQ%^VTybzJxVwVR)TRx7Bl*aSTmoYGdm8@Z-FmCHdBtH%8<WWAn#L
ztLMw3AozkzYFW=)&v1$7CyZeVb7-H}EE(6Zh2eF(b?o;~ul{2-Zkc>z%lns<h}E--
zg5Zm?X3F~BdgjEJu!b!Ruh-RSH?N158OPAlNGHa&|D3olI;SJY#{Qgq%bs~3Ejz2{
zN5!8nNpQ{j-g>sghu7zM3S*c;dp(phE}`kGC&p(_x!!UqwkgLn!yDiiv}D=|A8fyu
zR?m+`LGWdn6tlj!o+<G;EMW~>XrJ#3Z%7`{bjA~7+kbBScr^T6-?3Kflzj8^koUQH
zV)aySz9NAo>wD{26W_w68+G-HU;^#)U>V~Anzo9uy+368M|rH(Y*hQ?<Hqv8ap%|M
z@$)*idVVqrg0IRVORV4S>h5>=#V^i%<%swM=CFh{wCiolI6QGYk7dSI&nHc}-pa0X
z-K5^fOvzL7%_n?_C&uTR@RWmA&re4|@HIJJv!3_)*}kl+d-(E>%P@v1w0h=@OK6hn
ziSd)qLp^P+V}ao-@C%yecEShS@1@oAi%}4KT_(k>@2zJ_d=5)k!xmaS!&j0AG@UVH
zdxAY{N<A(1cwkPx<%AFE#Q41CdflT;^d8^4wwg{`$~%ABT-|5$(>*JP&P>;i2pb&V
zIP<I>XFl#6j~{V>9X9`1KfVqAuO2$IvA*N*p|K*Ef9b;i^Sl4kje|RnXwzSo|7V}F
zaVu*0q|uYu`lI_sAOETx--iC*xo>rIb2N8-Z22$x|9rpxuO2?McYW)R4I@Q3R{WQr
zUwif)+_AcEMEhv}Oza*EEx*Bzi3fIk8~R`UU)jI5f5+O!@M&P@|5fk*<G}x2tA|#1
ztZt6|_&sK>KgWUpyAN)xt?xOovbnZ#`q~E1!(*lYCoUU<q3Iv&n0Vklz75x(!(#l(
z{?#;HKd@(@()9hRUw;OICmxvoiQR*t_xLvWzq$7K!)ph29-G&nfuR!*y#7v18w_p!
zHu!JXAN@J7a>j-{5#%al=U?>y`TqKIbMLOT&7B+T>7n(5Ti=<ZO>nIEe;oY$UEQ;1
zV{Oms;{Aug#b>1dw&D7-x_|V<JU&+Xe~R?2U8aB5<J;o@!6V;dGX0C||M`CXU)?|D
z9k_-6-7{x9`d{9E-nF)S{F%D&f7|@uxw?61+<X}={NFzR4<6V&w6VH=V6+Mg|F_eB
zdEaMq<i-D@{|C?iZMgpLT1(ds9INmDcU^kq|LysE<dJTV_xQH>f5dOUi|fDDXB+%q
zKd@`<36p(d=U=@3Ki^;fuOHaEwy}O___r^u#lrutBe$Xd_pKh-b9i;n;{E5*Yk>9N
zw)nrXXYJsQ$F1!gc>!wYUtE9A_v_Dr)%|Om>FW3&U>5#w+x6#QI^`2U3;(y#|Fw<P
zLkI0I@7;M9{%@QA8*69C%b)Qdzl+yDo&CW!{QTWKw0dZL^U(U(hd>wozis_5pMKu6
zXYj!v>b~%Q+x$P6?q1!$zVA$&@38QH+x*|PV`byuzBT#$+o3~)k6;}u{vQXQf7rJ=
z`29;hhI?pLKL@n<`ODQ{8-D(-J>k&Wfr+>O7yfUb|MvMe`4sfN!K=_?<@$q7N5J$S
zaey7(<J-{x>A`*LJI8<eF5dsN`fP{)s~d;b^|Oa-yH*Cz0XzTV{kQY|{kOsIUn}yH
zz@D!Q|2rqPq5szptRGrQH`Y%d{R_;(|84Vstp8WGUVSdE|JHxo(EmFR?%OBt01ZET
zu9dg*FRuUR`}KeA^tA(rRwmvBTKK<h*PpcmhmZUO`ojNh^M6DB{yzA<t2^uB`eXgK
z4cCA9?B{_^`6YPlNnqz+T>sDa>;LB7%{BQ8@Q$_7d-#tP|Br+BKc~3Axc*rGZO8Rz
zb7j~1=x2ZQXMml5as4^puRn)2SJKrDxe4BW_AdM%_TTpO|IUL4cCQb=d|=W4gXjNt
z`mbLCVb6iZ^`~=Ud;CANwm+5kAVz)*Soptf{%@?M@&}l$cYsz7%4bp5^-Bnj75|Ti
z*Pn-X9C-uG!vAgS|IHEqS2p-9>{#)?bLq(coBkt@bbGwVx8eGq9^PTspP^dQw|M<O
zd`&WRXLx-c-8cH!{2Sdj`q+-k^>f7kLHi#w*Z<L+Mju;#qx(i5dyj8J|Lcd}M*sA<
zCDZiHd;gEtZ1l1DGrDi|u^pHG-rT#o>);uq2j%hL_?&;g@PsXcq4_h|G4a5T>+7$L
zgNFydH(A)=_`+YaWQ^v|*v^rG9T$J~`w2&mbPor|7ycgMjqWgirXCzQu0Q|u^JnW5
zmtdQ}xc*Le(EOQxbm+hMyK#8&^X&+my#5_AksUT){5^er?Tp3ihY6$1pNZXrq4&7>
zyY-pKNrP?v;`%VKbnJooGq!VNV8_MZ{i}zLY~jK2#m~19*M<+xpW&YIM|NEN-THoy
zi63u+pO1_F9rw*f=AVrxX~)H1`~KpwOT^&#qJJl?+ZvlcQxD2cJFfo9{i$Q#>YTyx
zW8B}f<<kd;<@V<@S9Yv!Y^<+soN{R6@Y<6G`5n!lS&y5)qn|xr_&e;cUxL4*@2?GB
zOB^r#JF7pl&X@k(cW~!v$L;lDR)1!lFaFB)Y4hO7Kh3hW<f4Di$=Pnd`EyQDc3k{j
zJ$-duKK>_nKo9N&-5gy17q8EzJ81q*KN<ulabUW?D}3SR@Umg+E&QJDsQEMf=<w&I
z_?^lJ3RXAdX2K_|?i{>uv-t~ur#oo=Og}o^UwvV`wqI@@kv6vZ3xB6OX#Pw;I^v)H
zKG%I3dw6k>Nj87s?{o*vpXo<O{8N9ozUO1i-i5#CWE}f@_`7?3_+=^slWqQ@f2SL5
z{!BkQ*1vmqtgdW%Jo<+Xo4@dPx`XD=^rK_`9$MeOc9wh_!OH569UE)hm2LAE{!Vw$
z{F#1qy1!dr{K*8HzwmdugXYilqhtOa`C)#UWb+sPPIu7!nSOM{KY9JISN^fsmDPRn
zzR%W&mbZS!+5CmS(;YN_rXL;qeR;>~&eP-{M9_bd_u$5vE7Hz~<c*2JD~WCeTW{g_
zbVtpf=|{)=S$>Vvb{_jfv&~=lJKaI^XZq12{nhs(MoqT)3xB6OX#Pw;dZfRbXKpU~
zcf>5Oe<M@I9@>0)KTY4}*?KQ+WyIvc@nih&pJPdn3@pEqU1JY_S;x)at$zf1@YlJK
z{$BVy=DQ7!=Ksk1@5BC`G}#st+IoxY@rWrS56z#EU1JaJIQ_hSaP)_rL=KKG`gzPZ
z8<;;fo}?Wg`a5zlmf*qhg};;5ZH>*JsRu{>>wexz|BhcQZ2sc<Fx^4(XZq2R{^a$9
z{=PB#!9bh8@OQd{=Fjw_NBXN*hlRgGx4ixhXN*6x`D3rwMhqStUtE93jbmi~Og$(&
z?fCHL+xQh?aD3tKR13|YsRyU}JAQ?*`HSoCbO+6!=|`va@7C*w!4<;hFZ`YEp!qZX
z=rn)z`Y`<HyUkztJKaI^XZq1;{_6E%;jj7S_0QrbIq`mk{eCetcw&N`Z_&>a#xOL0
zrXM|`pI7AV_x01|HV2Eh`3rx|k#U+o<9mlAJ8rLsN4~^qD`;?h@p^4&+Q=jGXJpsd
z!-?yKd@;)KOR_fi9i04g=<xOP!tXI3ZE!Td?YQ;x=Dv-?Hdi?~zVLU{mce85XRu@9
zfgLx0&sg0jJ-u^nT|Nn~p@ZWKe<z&lhUU-IgQNcO{@Uo5is>Aizxa7K-9htb`q2^p
z%-_w?Z_6GzzUben7WVab&*tW8x;|XW<}dnp=*j3K^JjG5=wmxB*WalxB?iYA{X1&g
z;Ia8L*fH_Ij@!>S{bz)BtZojzQFd^A;qQcV-O&7*dQf&wz8*Gz_2YrN<@=K+Ul7}R
zi|fU7N6nw<M@N31@$+u$-OSyG56G|42M@?+F>U_B-{}sTKhuvM+0R>_51cf4WU{Td
z@O#9Rk%#8b$gZ)6cHFMds~bD_+V373I5@tzK9Bik1M|nmleFXJ?~XlZ$hV3Nzc_Gk
zeBtk;bz5WeXX-)OX~)f9`Tcn;bZ~s(@0c$(Fn?@3$;sCj>Bhl58*7`JE2{@q_nmpx
z;B}nMv-K9&V{>Gj=Fj-v;mD3#KM(vqeRbzy`S<V!#~1xPG;QRO`7^R>?BO}~^Ui&T
zH%G3^Hp$jo^s_lKPV;Ac?{IX^es8SFw<L6*#Tte$`g_ESX%Ek{-}+0p-a$6;mC&}{
zqQ9rPJ$!vJ{pj$xT(7r&OOb!7C|y6WXK+{E!ryapuG??^oKw`u^<(t?x$fV?*}4Vi
z@W5+b!`55$^K?hepXo=3zkjbE(0_Tf`^SeickW%=zq%rK^{(zaxMxLgPajOT^^WHE
z*{5vWikK%2Cd&hxKe})9@w^<D%ZC1AEBb9?WB=}i<-hQEv@3>>&7a|(@ke(3SlzXM
z{lNOB{F}OF@yhDqLwn^qa@O!>b(w1GE&Lw$(MINvjVEcxR}aZQoWA4mq4WCv#*6F4
zq_f@F{N1^CbwfV8wYH&O+cfeM(2g(q_uT!PesuWrPk!X>JWc<IqL+;I1Bc|QeE871
zeEH4c&DGI=6@B6NIXOArujdqH{!RLAEp5ja{XE@4^Jn_e;m@<32lvT8<k|g=YBbj7
zFZ>;KW%$_q8SWW>WUns<{;rH&nOE#}-TJ=nL$<cw!tZe(ZDjt~c#?Kp{64(@fLx%r
zFuW++@rA#W&UIt+XZq2x>xKHe>)?L*={NF2&*m@uo$jFdGyUih{-zu9>HB?adj>C)
zZT`aF=?<Dd(~lnE@8MmCMw(Y9+5CmS(;YN_rXL;p*Ijt+RoXDx<}dsmdNTUR{2ARh
z`q++-{N^$CGtQ1L{2g^|_}Kgz?iqh%$4C6#`gS3^O4#v*zvI5y$o#SKB<(ma9ELV?
zsj%Y<e<z*m#^%rTqr>a3y)aw+d>ePl>)-gS;mGE9{w{vL4V~#8nLm^JZR|b1c0k{V
z-!=Zn4q3$JFRl;f=s3-v@x8;59Y1YNzae(wmtzUD;|qU>o{c^-e@6F>KDOiXFR93%
zLFN7TiA%)b_)$KewfOtSsH?-r=HJHJ?zIj1Csfwue#p@m5q5mhzeCSPADKU+`$iwz
zas4B+T_E;stZw~VH5zBf7ygdAHhgUU4EKybvg7vWqRBUS?D)dpao=oY{@8euc6_UU
z&E$~=w&M$bC!OoY=Fjw_!{<NNPW*Hm9$)x7-9fLv-Dj!}txm3D^B4Y3I@68KpXo=3
z{drDbAY}6w{!Vw${F#1q#NWs7TiGq2;65xjgN(-7{CW3x@7iiQ>We)#e{4KS@9|CX
ze|>+tZ|mO~>^`QcAK6^Ie#6l1@Lzrh9R29k)|st-N9+H|y4%?1+jx@4%JJ>+e{=mT
z`KbN4L*~!X{6F4*|31I{@BU~q`kQ%IhlT&+{Wv)?e<t_a*p7GpPX6{d{P{cY{cV0t
z()^p;Z(}>Y_4{LI^2l$G3xCZY^J|jc<6G8_y)x`pviXbtopg5GpXo=3{VAVHKCr2`
z=uCL*_4jD&&+BpfJg3~fuxF3F1HEhIa4O%aWABwtxIL`LqMxUCjP>L6qeK7jd*Sbd
zQC|NhrVWNRUq1+U<g1CH|BL<|ST^>+{2SXjGO*)YuP4;xu~!odf6bo}%f=phk8gd#
zdrAk}{6+tc**X}QKZ6|;5A1l?zj|-ecn90@g})QVF*JXsA00mbx4s-;*V=Bmg>2W#
zp)*r^*VN`O{GINg`7{0K(BIS7*UqrNXAkcN86IEwJKe!v{@UMLM+~<43x9tJ{&w$2
zjJUh#U+b^w?hgGq+Ht);9P!JAOtSSC{X5-3^Jn_ep?~_lvs*vcCSIf2{Dr^M9W;NY
zA07I;*8Qb#^rger6q~>Bce;b-&-A0i`~6qs3!LQbw6*m;2fBYRofi|6*9=?lXnvp5
z`E35YALp~l2h)jr(#@Zv`Fp(o{qDT>hrNF}+Rt0p<})_rR`er2uI&E#cV6{aJ&)J<
z=C@rtM!jA+qPvIJ^F@D8?+9<dO!Hf>)i!_O?`-~#UOzV1HsoI;>V6V>{hfaHVSVQP
zxb<)6?eI$B^>^{}Z+Zurf8+iRU&Yz+#r5Ib{hNMt`1~EcK8*gY-|O$u)}Qy|*1z3b
zLc<plUVj(Yhv^+;{*C!NygV#kKTNl6u-E(>?3j39$MyO!?F*6Z_|dMv6Uz>UHh-{V
z;=#NeAN6+l>S1_&KAOLa>%&CjFtq*|^>^g*u(&=<w{5W3{2T0;cwooni(wB9-)K8Y
z|3&q!NIQPC{+(EMFtquD9TN}c<@nIwaRV&>bMn{T&p3Nyt^0Rgb{*cIu57KQ9}zg>
zTfA+(@v{s^=lAhbIn~+nZO#X~e<Nju@0K`a_v*gQwVUr=-?wkKd{yqjjZ^NhJoeq=
zI@PQ1eE&TC@cFQ3?ZDcGeB^M~sJ?TmmsdYF9f4zI{`uv<_51Mova-HQzjxr!*7s)&
zFHAURe$9J5j%cI_e~xy1ir-sP7XEh5onQTK{*L&+`?Qf_+3|Vr?}wcFz<b?e=|uT6
z)k!xQvB&-MV$9zayUGlI-psqd=e57h$$LKNAxn=q_5KgP$3vIy`H%-aVClr=n=C!-
zK}#o|v^-X>LA|!>vAHHUtjWjk^sA>R>Z0Gb@p`lS;Ngu!dl&wXTu`k4&g=8R=8r5n
z_RyX$_Htg{mg!#6Z~Z$ctNT}$$HE8Cm&Ntz{JDSssP(V_%3}4M_j*|WcJGI;?zgTr
ze{4L-qa8Q9ZT%yjA19BmXzR^;J+}LLCI1xJ)<?W2uULzEbp84Nu^#pm+7hy;M^_KK
zo^03kYUKGbcEwuM<NSSon7{bnoowX)c6{;k;{16%W&Z9yZH2E4-MQAiUETebXU7-*
zpTGXw^{4w4d-w!c_`f~ZpPdKyrw2FJ583+~R)K~8=EQWGKhuv6->=(x|7iF6?uoZA
z7QZiJ-1+tT)9Pc_mqUBkH@5yT#(C}d;`+Wl*Z1x%yy3HD;eY4F`Q86p*B|{2cxe6B
zqvOT@v*m@u;SK$(>yFipjrFySQx0t$UVGBu`g;Ed-R;!-Pvovc%YVwDtzEyY<71x}
z966b-KkwI{5lhA%nm=PZM+SC$tH0L|>|K){SUn)G;pNLU<<Ibg`p&z*=lAtx|Jv%N
ze3$Qj@qA^cJ`LA*ZG9%o>ND^DkF@OAL-S{B=g7d0kN7)sba?$bCx7kDwPWS;SeEZt
zk|P7}asK|k@8BN&_O;>PbiMOO{9ZZp<mKht?eE?ev-u0ZyP}vM=k@w>Dz!c0@9{TQ
z({HhS@-3E6x^efXdG_3TYmc>ro&SjI|IWSZ`{aeb9WNht!@K?G!-L?0Ht~e+!FP9e
z4=y0X&s``8Lg5LrD_-y7mv(iuY3Iac;zQW(?dGK~>X<{DzqBJjf$bTm>A9TjzJNrQ
z%=hZ4iEm-};!ejhjG>*?tLO4E;ew(kWxiKWiZ3}VVQqfFzIv`86D}lr!pC*_dG#z4
zAHx*pu!NegZeD+`D6tn7JvH;adRpSc)2I`SVG6gz^w4u9iM@#E$(irfQxadp7KZ!K
z2m9)|vP`(B=!u!{)sqsR!xGl8g_^H!-g;d{VlSrsv%hEm5Ff)7=CFiYVtVMgs>EJg
z`-k~ne_G<h1Jnn`Fok{fTumliLi>mLUOgr8HEdyc5Ph((o~z4*OKSfx->WAjK8Gc&
zVGH}}xrR)*l<3K>c;l^~FJ3(b@fB=fa3$7(ef3;ZCR|$dM7ECe=T|~}1`Alh2KLo+
zEtzl`(Gy&`pFc~)M=*gIEMQ+f*Om#F6+IR6z4dDBd{-e47{LT;f9iI$>yO6i>y3K}
zL+u}1*IBQE_zE^KxGMUfYSHs->^%3^8>Z)e5`Q`EA6v(%Cm}wA1*~8LRnLF-(DPu4
zzr6O()q3_1@exd51`DWqzS2X_LnZzS+CR*9_m7?L>f`|<m_XI@4Tm0keYhd<SJeKo
zb)EGph_7H{=f4K}pz6`>X#H$@o+GhW(*Ch^oO%-CGg!b1HgHQ!4?Q1|*eh%QT(f8Y
z5Ff$B##g<`)^$FCef3=T(m`+)trzpX{ZkNM!N$&iwNB3x?5pR#qUWkwFI&f3FXA&;
zzzR07ub!953;L^Ry{=APPCZM+M=*gIEMQ+fODaY5RLu9*tFiN4gFIja6FAV5p8NIL
zo?m}2{#--*$JTZFQxIRl#?F6D^ud81=kKSF6k*rY{;_qOdJ^I@SilN4@PnGTY~Jh7
zqa^-X+CSIo*+0ZbFo78?pysQaSI?s*{@U6<%y;(>@xisp14b}`of^NM$B3TmX#d!{
z&UzKZSFo}3Uk81#uRo6!Jxkg@wvJOzLVN}bSiuJN_2-c2xvuulQqTS&K7t9%U;+E~
z^9MxF^|XJO@9rPsgX@w9j9>!$`lI@<ul-}|I_p&sU%|%Ce?9cUzIr|^dTyZoW9vBe
zB*bU1fE8@uK#%kK{7KPsL+zjI_v|0yBbdMp7I2`)d40aJyy1Bx?H}g5`^V0A1M+|o
zOkiI<r-+^#Yya4~&UzKZSFo}3-w=JUubws0bE5W-t>e^_5TC&UR<MD6^*mGh=O)@e
zH|p6x#78iJ87yF5Jr@ywZmRvme0TrY`EE=eFoFr}tLNhKyV1?Ge{5Z6y$a$h*x30`
zL?7&{=klWG=Gs5Dj#E!Ud<F|x!3Or#a|O|J3+<nq^z0wvBbdMp7O=0LJBprLYX30b
z-9L7|o012NU;_K<xs&L*mG+OV>#SEnd<7dj|IN?``|5eL=vmhOv2~n!65=yhzzR07
zubx8moTUA8^Pc@fd;}Ai!2<Tx^E%OUvi1-2-Th<dy9If`2qv(vp6`jCTWkN=y3TqP
z#8<Gf^WPGEu&<u0UL*)^qy1y+IQ1mNXRv@3Y+zqKR~J3E)&99v&;B7kf(guE0sHEC
zgXp=P_7C&j{bT1_CJz|F1oqYQbJ25q?H^m$S+9cl3O08Blh6nI>UsIag5VC?Kemoj
zPeObK3s}Jh_SN$W(erECKPUI>AL1jJzzi0!ubx+mo;zy)FyGxjcD`Ga2aI3>`|5d>
z=(&^jkFD#hS3!IQ8$17P&<FeKS&<J++*$j_)^X}dh|gdFE7-ukdR{I5oTB}6+n)VH
zd;}Ai!2<TxqaP^wb?qPKyZgt^cRTWc5lmoTJwLu|5Zp!k$JTY$t02CDjh+AY=!1Rr
zJpGc=JK8_Cj#E!Ud<F|x!3Or#^TJC7!KvCmcj(za#78iJ87yF5J=eZW5ZqP!hxzXQ
zvGe^JdB6xJu&<swik`b^|Jb_DdKJW1u(9*s5q+?)p6iI7yKDd0I!-+a@fj>&1sm8`
z&x=LRJ+y!B)U$txk6;2bSirt|-Y0tQsr|!zcmLS=?o1vqf(h)a=d+^cUfMsluCrbR
z@fB?B{HLG~_SN%MxgOqI`^VOC>Pd*tU;!)Gz`lB3F4x2RX#f0r&;B7kf(guE0sHEC
zhy1<izS=*`clVE-?=Iv4BbdOxdLpUU{j`5<U1z-t;w#wL`6Kkffu4;(t8k8+yiK?D
zxr)1rp8HE=Wa~KfB*bU1fE8?@30QLKxx46jfGR(=Xa5i%!31WofPMAcQ}jGg^i<4u
z_m7?LuH*qDn83bz?k##ABziJi*IBQE_zE_5{=1<M_SN$U(GzR`*g8%<3Go>$U<Di4
zSI@`g_49+Zf9~G1e~6D@0y9{^fu5uMJ;gqc^c~Uj5bYo4yZgt^cMtM_5lo=wtJ_kl
zpZY%M&7SXHkobp6WM=C+>s1h6!3GBRL?1K-OGoj?^n6kDJWTXNwvJOzLVN}bSiuJN
z)$=9M^Kf<YUOoGV_y{I2g9YrX=gXq!5u&GJzPo?yeD@{~7{LVg)$<k6^BdYfwyv{Y
z1@RSZ?ELpZAMC5==kj{&k=j4Dj#E!Ud<F|x!3Or#^HuTZQQANE?b$!XM=*gIEMQ+f
zUlTo#*8X9>yMOF__ahG&!36fz^9|AS80{Zh*IBQE_zE_5{`;d3_SN&Rmk5H#YX8_e
zPCW_n87yE08`xLR+vM|Xzp4H6fS&zBd;}Ai!2<Txb2E8A;kUGZnD6c%JKqDz14b}`
zef8+?rYrJaX6riZRoMCRTXdguhY?I*Up?Oze-hPW>w5JNU%|%C|2*`;zIybt^Q-b-
zWa~KfB*bU1fE8?DUp+sR>+=rL6Fi^1oO+gsk6;2bSirt|J}tiw?i4*0^WFVy=X(Kp
zzz8O=ub$6{o?W6Rvvr;R6vS7svGczWeXy^de-S-v+CR3AQ%^#C1`Alh2KLqSucBwS
z_RouY_7Cw9Okf5J=+$Gd=f5hSC)uO@!+dxD*!f;e9x#Fl)cZGeTiS8%ujftA*F?`=
z?H^m$S+9cl3N|o!3HqSkzp0y7&(}rIy7rH)<J6N7pTPoFuz|z-8GZhIL-agO`{$)S
z`-k`lCNP5ql=j@(y#D-1^qi*s!+dxD5FfmZJYWP9sP}K`=GF5p(X&tc$JTY$t02CD
z4Gdn6J~+Ie(O0i;i=O@3KemojPeObK3s}Jh>iwI#dHwm0=sBSM^NODRLwp1in85<-
z{hPXZ^|Yerp!N^*-Tgy+@JjN45lo=ozp0y7&v!*ns{Lc@I_p&sU%>_juR<Tx`!{v-
z>ba17zUA@SKemojPeObK3s}Jh>UCE)ub%ITKO5RVukP7D#78iJ87!dQzp0y7&;N*?
zP3<4%yZeXu;5Fm{BbY$Fe^WQFp8pj+hqQleU1z-t;w#v|pg<qg`!{v->iNFtIjsF-
z>p1l!#AmR86>Om1zp0y7&p%1*>DoW9?b$!XM=*gIETG=Mshd}i-V1w%{8usG-9N+!
zuOkl_!3664o4R@R=)I>;kpD7U*IBQE_zE^Kcs=@{-oL4vSC8I%dZzpr**Z==3Go>$
zU<Dia&`yq9&)-P=S)wO+L(l#pK7t9%U;*|1P2IfyJXd1R7Cja7-Tgy+@J8~05lrB7
zw$%9b=)KfWl>ahY*IBQE_zE^KcoX`d-tVTH*PkED_2)^VC$e>%dJ^I@SilN4aCm={
z@AcSEM9*)Fp5PCA_7Cw9Okf5JsQ0_+=Jn^NqUXt?r((Xle~1tMh&*5f6R7vQ_0jWx
zq9@b-v2~sGDu}OO1A{lC59<AHee_&VUjIBr`^VOC>Pd*tU;!)GK)v6skDkL4|2x`0
zZ|T`T#78iJ87!dQ@771p&!k?ztNp`#cmEI{yp=p)1QV$DyY<oYbJ6ou?H^m$S+9cl
z3N|o!8~Whzem&pw{U1dC)3kqV9jBgz_zV`Xf(_LB%li0p74hfk+COja*+0ZbFo78?
zpx$5BM^7lvuiw-DVZOV6h!5UD9x#Fl)cec&=((Ke`F-slTi02yg7^wHFnA~Wpx$5B
zN6)jQzR%G9v2~n!65=yhzzQ}{?=S14=c1zjnc6>P&;B7kf(guE0rmc}K6)-8^?H`}
z5A)sqLwxYZ<N+g?K)t`LkDech{y)(Ev2~sGDu}OO1A}*=59<A8ef0cL^gLVp$JTM`
zNr=y20V~)*y}zuFo>xo!bF_cn-Lrp)k6;2bSU|nMtdE}8Nc?lPf0*y?AL4`ekOz!l
z0`>l~K6)M_@wxVot?R5;L3{-p82kzPpx$5BN6%{|{(0IzwvJOzLVN}bSi#2j_tf)x
ziGRNK&!6_}AL1jJzzh~p?=S1)&krR21=>H%clQtR!Jm-_j9>!w{<1!Lt|s>ry-@qd
z)^*maAijbP4Bm@AsP~uk(Q^%nf06c&t>e^_5TC&UR<MD3e_0<r{~&(8So`OFJ^P3F
z2qrLt1=Rb?`sjI?e1794+CR*9_Yd*G`^f`FFoAl1Ssy*G5j`)}{;_qP^(u(3U;~3c
zM<3Mt%lhakB>rXEKemojPeObK3s}Jh>iuPX^t@K$U#|W0fu8+Cd;}Ai!2;_2WqtI#
zPU2so{lk2B{}3O1kUU@n6R7u>_0f~aA6#Fl{bTDo>s1h6!3G8&LLb!o%lhbfz36|H
z_K&UO)RPdO!2(vWfqH*gA3Z0C{#R@N{6)|HAwGf$%wPexzOY!9_rCx06ZyRHYqWou
z@9rNvUqv1;f(g|7%XIVV`8~Pcq0s)Zb)EGph_7G+gAbz*>iuPX^t?zukMvsYA6v(%
zCm}wA1*~8L_5QLxdhR5j|9PGE&qsRp5AhL9U<M1Q_m}n2b7zTvz4j0D-Tgy+@R#HP
zBbY$FzpRg*QzZTk+CR3gvt9-96>MPeQS?E*zpRg*mq`2@wSR0Kr=EoP3>L704b=O~
z`sjJT#J@@V=dXJ95AhL9U<M1Q_m}n2^ErwCL+u~tyZeXu;IGL8MlgYTe_0<rUz6+A
zA8G&Cy3TqP#8<F^!QY?{>iuPX^gK<jS8vw-v2~n!65=yhzzQ}{?=S14=biHRm$zvD
ze5_~x5Ffz=X0U*Ie_0<rr;49%)&60=yMKreK29Dmf{BfL*6VH(|2FL(Ti02yg7^wH
zF!)>aLA}4Mk3aX2__u5S*g8%<3Go>$U<Dhf_m}n2b1#X1hxX6k_3R(wBbdMp7Etdm
z>!asB68}!^ALhIJhxp(V<N+g?K)t`LkDlL<_)`1F)^*maAijbP4E`Q{Q1376qvt#F
z`|=-a|JXWCJqhs{EMNs2sP|9x(evN841#xQ|9rA%{}3O+1ZJ>+s^#W#4}<snu^Z9z
zZtWlDyZgt^_YdR&BbY$VS2yqL=PyY7d$fOSU1z-t;w#v|phh25Ew}99&*w$YpJ@Ns
zI!-+a@fj>&1skaO>gM(5OA`O5+CTr;vww(>U;;B(K-F^V9{zk$^!%Cj5A)sqW9R!P
z@_-RcpysQa*PpLQ{Cl;3Y+Yx)3gRo+z~G<J2UW|-J^cBy=y{*^kFDd>lMtW50#>kr
zny+qNf4(O1@7MnMRL}k)K7t9%U;$Ojul4ZftD@)6wSSoJ?jJkfr^y3GFoBw{ZeD-B
zA@Lv3{;_qP^(u(3U;~5Cpbx5+5BKor>!Rm_+CR3AQ%^#C1`Alh25P>#dHwm8#D7Tp
z=U;mE5AhL9U<M1QTJF%ppKpquztH|+zPo?yeE&)wFoFrxe0B5s^Bsw=w0~?}XT1vI
zE7-u`-_QqD%T0Rt^KH@dVeKDV$EhbFK7$3UU;{N@-Ms#MSK>dS{qxzL{X={N6PUpQ
zs+ODf@TV0$f2sY$e0TrY`Tm_eU<4DW`ReBN=YJ&rquM{VuCrbR@fB=f@E_=ds^w-q
z{P~{f`77-oTgRy<AwGiztY8B*U){X^d|%@KTKnfed-f0U5lmnP3#eLd)5D+t6+M5W
z{lk2B|JeEdi#%Wi6R7#>=Jn@?68|ynA6wU1uY&jrHZb@c`k-ogx4h@#{r%zxqUYn<
zKemojPfC0aOIX7eYQDO8{rR!P|E=f=n|}SXOneMen8OmPmfQB|pC5^yzY{$*^S%Ak
z5+D9|m&Y=UVG1>0-Ms$%RN_A&dUED_^_0Zdu!Z60(Faw_i9P)JiRk%z(GxS@t0yHs
zhb63G3pHQey#D-5;y<bV^M#)MLwpQVn8OmPmfOlb|K8`<|B0S|(Eee**PoX7@Qc(3
z#xRANuWnv{ej)L-_7C&DdP?GJ*uwBj=!2?dt%pB97d`)|{lk2(o|O0;mav8`)O>aG
z`g6jqg5aODf4<zae~6D^3UgRO)pDmE&#yrA{Im8C^S%DG#D`y@J}`zU)O>aG`g1{v
z|CII*^SydX;%nH#@T=&9s^yM7{JDVW`Ly;A^Sydf;&WKS8n#gL)y?bAg(dzo+CN|G
z*+0a`FoiiRp=!Bv4}UHsdj3WGhxuNATH?d6Qy&<^6l%V@dHuPl#Q&@I5A(fxO5$tS
z!tfjDgR143J+40&5k3E={lk2(o|O0;mav8`)O>aG`g3uK|E%`UH+%LE@i9zc4oj$7
zPU_J=7ZW}IuKmM&uRksE;kT#{jA05jU){X^TvFoyL;HvMUOgr8HEdz{ZS+Caa*H1R
zTtf8xr}hu?y?RpOb6CO}wovoc&FjylCH}v(f4<YRe~6D^3UgRO)$;Zp&#y~~p3iCj
zFyHG>OMKW;9~i?FYQDO8{kg2fH`+hU_v$H$uVD+r@1hT?mfQE}pUa4z|JMFtzE@95
zd=5)k!xn13x_SM%oWy@#`{#Q-`-k`#rZ9&kR4upa;ZG=fzM%cXe6K$(@!|ha9~i?F
zYQDO8{kej~e^L8~`CdIG@ilB=_`m3bs%5!{KbIFhU()_zzE@95d=5)k!xn13x_SL+
z<nOs(*8cf^&;B7khAGTp302Ft<s9DMzpg0yzoPxae6K$(@!=1s4~$_7HDBGl{#;q~
zd{z60`CdIG@ilB=_(SwT)$+m~@Bds$^n6YGhxuMTDe*ZhVGUcT`ReBN=c*F_b?u)Y
z_3R(wW0=AmmQb~Put)z~Mf7|_`-k~ne_G<hA5$L~!xU=1x_SM%y43cY+CR+q>M4n@
zVGF~bpbx5+Z}#x#YNG#J+CR+q>Pd;uVF_#4Ld{n<uRqt2_-||f{IqBP5Ff)7=CFjS
z<y$@cxw`oC9qk|Hd;MvN5C4z)z!;`b^VQAk&oxC)tNp`#ubz_l8n!U}8Tz1V`A`pk
zt|5B9tNp`#ub!0n9G0+#E!2E<^ZIjbiT|GV&(C}I5AiWfVGc{ES{{3G|Ie>$iJt$_
z{$ak?pO*OW7t{yFFol}0ZeD+uB>sQ3f0*ypQxadp7KXtsx4z$V8LF1cU&imxbwtni
zwSSoJ)sqsR!xGl8g_^H!UVpA9@juZ1IibtTt7n<`7^X0XB~&d}=;6<GMb8hlf0*yp
z(-I$Efcn4~rcm?M&Fjw%B>qR*Kg{>)DT%LP3&RVd52}{1hd<XBJwMj|VZK*SN_-AW
zSi=@-zPfq+xsk;GMEmDLJ^P3F7^X0XB~&e!>*3E0MbA&Qf0*y}rzJkTF!g~kOrhqh
zo7bNcCI0`kf0*ypQxadp7KRr=A5<;x>EX|fMbFQ)f0*yplM<i964tPVny+qNe{L%A
zKiB@bXwUv3K87jGVF^{sMdh9%@9$qX5k0@q{$ak?pO*OWV$=u5Fol}0ZeD+GuJQ7n
z6w9B#VfVe#&cAoZ#HTQaC9I)pxoVG}?>7@YCy1Vw`CfmPzR>B9U;;B(z^#0H_;X9q
za{<v)G2g4FAwKvbzQPD5Q2lwvh5h|=3(<2y(UaM_ySnvtopaj?;w#v|;7jO(ny+r&
z{#lmz3yGe{)^X}dh|gdFE7(BQ^2{Fo+)DIZSo8#6CNHO+CE_EPzzh~p^VQAk&&d*h
z5$zx5yZeXu;49<-BbY$da?2jqpOZw-MYVryU8g?<@fB?B{9i>M)O>aG`g0qJznJ!q
zt>e^_5TC&UR<MDp<-I-pxwYuIxc1N2diD?T5lmnP3#j?(=Jn@x5`PKpALhIJhxp*@
z<N+g?K-H4Um-u>re%)5|TvGeT)^*maAijc)o&Ou?gPN~yUVrW&@t4y6v2~n!65=yh
zzzQ}{wcJs@)5q)2?M2U}wST_Zvww(>U;;B(K+RV-uRnK`_{(VjFyGxj#0TFZ4;aA&
zs+PO<`1{wdiJr@9|Jb_DdKJW1u(9)h8+}mo)y?bAoh3fh{;_qOdJ^I@SilN4P_<mD
z$Lr@iiJr@8|9q!s{}3O+1ZJ>+ny+qNe?EU%dH+HChxzXQAwFoy14b}`s^wxm{5eJR
zUqSoF)^*maAijc)o&US&gPN~yUVrW)dakJbW9vBeB*bU1fE8?@YRP&$zkXfxTuJ-q
zdp-My_~OLw_lpWPFt|y#U-Q+?>(8kYe`V3L#C&i6M8qdBg9WUhYPoU`e<IOy717f$
z->WCMDZf9$2qrLtny+qNf9@vnR~0=4^Syd1;u{#;41F+ys^!x?{JE>>xti!nY&~Z`
zXT%q<f(;CA-s#hPb@Tdj4~f6J_7C&D{zSwlFoOlGplbP14}b11daj}U!+fuv;1<*e
zMlgXH)O>aG`g1RdzozyN^Syd1;u{#;5`8d&s-={BhP|(U?kRe%rTt^;dHaX>0#>kr
z!L2%dny+qNf9@mk*Vg`FzSp0K_ylIKfE83N|JvjA^SwpSb+mt&@6{75Qy&<?1ZGh4
z)y?bA{Um-#`-k~nJr(f{3{FBHjG$`yY!83#D|)W0{bTES`-k`fR<MD=$(=sUS2wRe
zpBC-c)Ba(;*Pn>^1ZJ>+6;v&s>fz7*MgR4+f0*yp6Wp5mzz8NVgPN~yUVk1adTyZo
z!+fuviueWww?Q9_plbO{4}TsYdTyxwW9xbQhxh_kuz|sCJAIn3ZeD*LEb%we{$ak?
zpNRMbX0U)2R4osZbDX|2kav~Od9#0SE0)+BYyU9ct0%Y}^??ygU<Nf`-Ms!hLgG)<
z{$ajXPeptKgWIDIMo_gptcO1jm)M(V|JZuo{vp1A6>MN|hfbg7tDD!K$4LB5wSSoJ
z^(P`eff+1d1y#$Vdie8biM^Tj5A(fxf?uOPFoFrppysQa*Pj)Mzq$4g^Syd1;u{#;
z5q&U%s^vF(`14y5dkgI!ThH4+#22uF4Giwo>C=35^ZK(>;%}+_!+ftl5%CGkU;!(r
zT2_1bvqNHUrTxQvub$w})CWc|ff>|%b@TeOSK^nof0*ypQxV_5;1u-12&$IdJ^a}t
zu_tN&*m~anA-;eWY+&%~oj%Q1H?KeYCH`dXALe`giHJ{N1`Ak0)pA-7fA&f2t+juc
z@6{9Bh5En<CNP7VuWnv{HYENw+CR+q>ZyotU=X1XMo_h+J^XpR#NJl>$JX=q5Ag-8
zU;~3wJAIn3ZeD-Rkoena|1jU{PegnIGg!b1s+Pk&{5f4>Z?FBse6OD1uG9xcFo7A=
ze0B5s^F)cigZ2;ey?QF*8yMUTeK3Nm<*Xk5oGr1xru}2<dHaX>0#>kr!QDH3ny+qN
zf1V=ochvr2zSp0K_ylIKfE83NPwwGQCb4(Y{$ajXPjC<F10$Hg3~Ii*dHs31#NS!_
zhxuMT74Z!W?ukAaLDllq9{xN{Vo%ZjvGu(DLwo@%*udakoj%Q1H?Kd>lK5ZO{$ak?
zpNRMbX0U)2R4vcw;m<QA_Ac5#%=hXE?oEAQ1QVD+%~v<CKe@z5+CR+q>ZyotU~nJw
z!3e6B=k)OBxe|M-_K&US?H}R`SiuGc_wDp)zPfq+d6C55Rr`neUVkFu6PUpQR#3IP
zpoc#%l-Rpz|1jUHC%7N=fe}n#1~p&Zy#BmQ;_t5g!+fuviueWw_eUR$plW$Z4}V@N
zvG>savGu(DLwo@%*udZcoj%Q1H?KdhlK6XS|1jU{PegnIGg!b1s+L#u@aL5hdoS%D
z=6m&o5A5<^hA~WG4mDrhy#Bma;_oebO6Gg@G{grFqHkaX6R28V)5D)aV(%k*GF#W#
zKPB-sY+)GV8`OMt^ZN59iNCMtiJ9;9CnY|IC9Gi!Rm&TC`13}Iy`Sg_AB^u#f0l`l
zVG46tLd{n<uRm{<`1^~Vn)zOTTH?cpP#+k>6sndt_weT}68iwrlQZ9|rzF0HEes!u
zKB)QX=Jn^D68}K$ALe`Yq{Qd2gf(oTYI%DPf8HUn57Pd5SkL|;K87jGVF@)~-Ms$1
zN8)4cALe`gX^9UXPJLhuQ>a?r)x)26OYDQSf0*ypQxadp7KV>NAJlwx^ZN6CiGPUp
z5A(fxQsQ%1!Wy<vwY;~7Kkt*+hid=)M$i5sK87jGVF@)~-Ms#MNa7!+{lk2(KP~a$
zBdHIJVG32t2YUGPL5Y31_7C&DdP?GJ*uwBp=!2TCZeD*rBJq#V{$ajXPfC0aOIX7e
zs+OvUKOdIZ-_ZVfbkF`FK87jGVF@)~-Ms$%wZ!YcX<akl>rYF3_!#N~W0*qK^3fjt
z{FTJ&ziFK_->au2zJ@IfAB#Sy`ReBN=Wium|4r+d`CdIK@i{DE4O^&MKGwsZk4von
zo7UlP_Us?xW0=AmmQeH6&Fjx6C0_qc>zesqe_G<h-=aP+hAC7npXlMw-%IRoYX30b
ztEVKshAj+N&<8bN-Ms$%lf?g)_7C&DdNSe*SiuI$2b{L<mr^Zt4}bnqVpl}Z67#)!
zV&YSn!xGj|^VQAk&u1h)5j`#Qy?Vk`>H=e!!W^oWPxbKU(-ONXdP?Sd_0+_-Fx-JY
z7(>lhH?KecF7Z1=Ps)6+o}Bm+*06=)&SC%jTMvIeE3rF8&ocA9dJ^I@SilN4Q1k8M
zPb2ZWL{G4*%g3u{iTDU6FoOkDE&tWSpU+9`n&_#R@72@T`PRq-MlgYzuWsJ{`I5x%
z7Co7*>->GWAilEm!eBT0plbO-4}ZQWv3o>MWa~Kfq{Qd2gf(oT=Bu06pRY;$UeOco
z>DNEY#K$m&IV_=S`AQFezACZnqNirQw|`pV!@XS|%P@v1)O>aG`txmxf1K#aneWw8
z5#PXI9e-g2Rm(Sf`137^Jx%l^ww|+pa^g!^!xn~*>-1^9x_SNip2Y7HJ<H7Z`V$kM
z!W@>chN`9Q;m>y^cE9LpneWvTo<`rm7^X0Xny+qNe|{+O2SiWFe6OCG_!fry&<A6v
zTE5@IpC3r<LD7>k->WAlzJxVwVYt83r}^sU_2;J&pKAXw->WAkK7~0fVGUKwk9+v@
z6N!Di_7C&Ddcp(L2gWdkIn;c0^ZN4(iQmxvVZK*SO?(T(gXn`XR4qU2;m^+{c2oO@
z`CdIa@g=Nb3&XV2r}^sU_2)v%L2yX>hxuMTG4UzPVF_!fS}q{x@P5Acf)aaJ`-k~n
zJ>lc24~$_7bEx_1=Jn@ds$Khs`CdIW@huED&<A6vS}xMVpNmTD8QMS0_v*=sFJTQ^
z7;bj@G+*7k{#;7ppP>E2e6OCE_!Q=_gf&zxm+0ZoB_;Mu?H}fQ^@NA04~$_7bEx_1
z=Jn@t5`UKV5A(fxYT{cM9!4LGp=!Bo4}U_5JzM*S`CdIa@g=Nb3&YbpeVVUsUVpA6
z@lVwLVZK*SOneG+Si%~rmMiq|=ZX^hB<&yOd-a58P#+k>6y{L#)y?bA)g=D6wSSoJ
z)l(DS!te>`gE3SsSLxx;RVDVx+CR+q>M4k?U}NV$v(vQ%HDBGl{#;w)Gtm>-I?mVg
z3Go>$U<DhfTCUl{pKD3%Q$$a27I``KEE6BY6y~smny+qNf37F-zax5T=6mZEoQ-}M
z!31V-OG^)bt}C&>D|!m%d-c@Bw=jGn{S0HM`EF_2af0i9q8my4Q$<h8e6OCI_!8Ez
zh2fJreX8XKJ^Z<$#6C^*EHmG$Cni3HIV@ofHDBGl{c|&kf4b;tneWvT{x<c2F-&0&
zRm)9!_;XW<{XNlBGT*DGCccH?lhFrbsQK#V^=Db)e_#8D`CdH*@fB=fkafD2plZ2g
z4}WeYvCj}ak*(vr{-ng`u!J>iq2{Zb*Pq)+{4+&Q_!RPT>RBc}hAGTp302F<J^Z<~
z#6C;()XewRt0g}C9r^~wFol}0ZeD-xAn|`7dUED_^_0Zdu!Z68q7SN;+x770_7eMS
z(GxS@t0yHshb63G3pHQey#Cx-;-91a^VFXGLwpQVn8OmPmOJ+F=S~v)T<ss`d;MvN
z51&STU<^~J`ReBN=TwQ$wSSoJ)l(8*!xn~5M;}xzcj@6zB(cxa{$ajXPfC0aOIX7e
zYQDO8{kez4KVSRj_j>jZ@i9zc4oj$7?$*PfyG!f~w11fI^`|91{C(;JW0*qCS2wRe
z_mTJ)YX30btEVKshAj-Afj+2O?$yJedrRz#w11fI)sqsR!xGl8g_^H!UVk1a@h{f?
zd1lZ4AwGsF%wY*t%l&)!^8kr`iS`fkz5cYshtHxuFor4Ae0B5s^H7O@srC=^y?QF*
z8yNfn{=x{VmIwFn=OGgNGSQRRdd}AyIq@Z|VGF}&cltD6-Ms$%hQz;I^ei*q>rYI4
z3UgS(8mg9u_weTt68j3#(=y+yCwvZl17n!N9BRJ0dHs2;#J^JXl+5?)X^0P=OW(i<
zCQ!9Jx`#iHk=R#>p3K&D_D@NC4O<xI_y#p!-Ms#+O8l!uPt1I;KPmA!EMW~>s9IKf
z_>)NNYeY}@JbZWhvrK#pQ<%dNYQDO8{aKUvLiE(k_xjTkA3mS@z!;`bwe0NS&n}65
zt?0>_@6}ThU&9uLFF+sEe0B5svo7(k)Ba(;S5HcO4og_W7OIv#J^a}#v9H(ud124~
zAwGsF%wY*NU){X^?3egAX#X(Z>re0^^uq`yFoUY)v>yKKlh`+ko`U&aJvH$y3|~y&
zz!+-2x_SNCkoY%=o|O4sJvs3utYHhomvs76OWMPq$4l%Vik@ZWd-Wv5XRv@3Y@p_=
zo7bN+B>s;?Pw>*7{Y-oeQ<%dNs+Pk&{5f4>-z<7+=6n4KUWR@c!31Vd^VQAk&l4s7
zEuyDjzE@98d<(;u)6X!5s^zR6{+unbZxuZ$^Sycs;w#wL`Cox=Q1jKz>(5gp{%xWs
zvUQw4PgCM^Si%~%P_;a{hd-IbzFqW$uk6>)5%CGkU;!(r`ReBN=jjsv4$;#v-&?QX
zRp^HiOkf68%Ts&!^E8Qlr|2n|@6}Tg-`IIyjXoGb%~v<CKhKi*QuHLYp0j^);!9Y=
z7KX3s^r@C-^zi4I68p!ZXPNn4e-h#|SilN4Q1jL8X#R+V;B)ftmi<uT-=%}1%jd-1
z-S-OX8t1kx5g)+>X0U*^#?r39_4lS9h@N+go{ITiJ&m33we$^)U;;H?-Ms$%MB?8g
zdNSsF^%TTcuz|tr&<8s;qQ~jak44X)h@Ob~UOfr%87yE08>so}=Jn_QB>qpee_r3S
ze~6D@0y9{^EitOc>CaC^&!1`kFyHG>W9NGV^??ygpysRF(fqORWBi50zgPQ*`CdIG
z@ilB=_{L7(GE^-;lXG}KKlXEpeV^!wneWw;5nsRxHZXWor%UtI?P&gJ%josz0w)E*
z`(?1ie6OB}_ylIKfEC;lqk5eEbAsskbJ5c<->WD1L;40rFo7A=e0B5sb0LZUfaoch
z@6}Tg-@xFH&<7*fsqy=BLDBO;(UUOWt0yDAfE8?D@a9gR=BwM${ITcP#U%bi+CR+q
z>WPR?U<M0VLDg~*IfwW8by11^3+*4~d-Viwp*}Ez3Cy78tDD!KOG|vE{lk2(o|^a;
zhHvfkEyEb9mP_{V=TZ{;VbPN^->at}zJiUN|84jNHDBGl{#;JtKO%Y}=6m%d#AmR8
z6>Oktxoi)ALW%uL(G$GA%gd{0iTDU6FoOlue0B5sb7hJDsOYJf@72=~AH0M5zz8N#
zwOp}>KUb32zY;we^Sycs;w#wL`QM2?sQK#V_2(KA|JT|-%=hX^iO*pPYuG~7a<v}*
zTwP-SM)ZVbzy67cPhbWMSV7HKH?KcS68|yL(=gxbPw>ao1x7G|8B{IT?%~gMB=+N?
zr(nKUPepuV=Y1FYU}WQ-{@hUF|5o%Q%=hZah%aCT8yLL1)2IGizlT3J_<!u337q6)
zng0Wbj=E@6)Y(-u9AX0wWbPiFfhZomDi}bt6|qy@T{B%wAECQ?=ozlywe5jMadkY<
zD7wzB;>LSC(0J}BqETF(Rnd6Qs%X5&1O2~mp5I$tNmVg#Q~Z~Dt>gpq?dLrzzvq`X
zZ;nb;v9>odJqG&A>aoz?0X;APBar9I7uoT1GHd@uK7P<&R*(LE%HITS&;@;vYZ+LE
zKPR!aH!(dS`pfD`#QHvfaRXYQ1M+<NBJ0m-to@gm9uNIx^#o`ifeEO65c)u_Wz{nL
zIhD2jGSg$BzpNey?L9C6BQODZzI>7OXB}%FGd<deC@)z(2HIPo1A1TpaxH6?;m=yu
z_A5+Jg#NO660yF$*gl{IIv~%NFS7n@V(q`m#}E3;>Iu+30uxaCF!X_3%f@B+GsxQB
z%*PM<%j$8^-U9<L0uzwu%NJRHPG{}E#>dY`mN|aV-U1!a0|St2*}M#Ywy?JU&BqV=
z%leau^?elE2ed#3<oWVN)}J$3`>*rygZ{F50<@371k^qTeIVEJ*k$<hIM((XeEgul
ztR4sLJum<xFadeKe3A8Mh_&C($IlJR96xApfez?_0m!vHaT)$RiM9P7K7P<&)}IvZ
z^^a2?CTN2$$n)iktUqV5_TOZBeDs&q6QX?#rl9@_=mWWyr!2#tr?R%+VtQ=!m(}B<
zy$^<945lE@moKvZoXgsOo9WS`<&7T`?QPHneJ}*M7JC`~oWt7Q!t})GFY8Z=_WCEW
zeLx#@L7p#PWc_(2YyTZSe$Zc5Pl)z0n1cGJpbz9)&R>Q<&tPr8%f}D;%j$8_-UmZ4
z22+sd%NJRHp2ONFeEfWRnd1lTZO{dMFa)`lXD`E_3s~Fl@$rNHvi_uKuYU&H2ed&K
z<oWVN)}IP%|9w7w&|g+hi1sm<g8FBn59C@#mf?@X+Wvr#AM}^i<D$I}hF}b)AkUXC
zvi?+A`ycZ0^SNb?AGEhY7xcjp<XXm-;m<g0`y)Pn&|lV{6z%o@!uA1e&;@zEe3A9%
zBG&%LeEgultez0<V=x8veb5JTEfdS|r^ed;gpVKem(}B<y$^<945lE@moKvZ%&_(c
z`1tw!GRF_v+n@{jU<h(8Q_JvYnzj93K7P<&)*tN)q~8E7&;dP==gSvaf9kCLPnn(o
z{blt;XrF-EjnD^LAlEXx41eZW+n+H#4*JXL@z6d1BQOEAFOoi<FJEN+Szzsd&h!}Q
zFRRBwdk6Hu0E|GcWydo7xrDX7mFY>)UsjKH6Sfa%fez?_JYT-Z`g1vJ{|i2T&|g+h
zg!T!jeF^$N3*=fZU4}oGv9`bD;|Kj^^>}C>fDxF0+LuWm&zCQ<{#?b{r+oaNzpNe$
z?H$kq126))mMfRxkH^~nijN=km(`=i*gl{II-m#geEA~l&x=_5U-R*U{<3-^v`;|o
zE6@j8AlLGOW%%<#*7i4i{Gh+A9uMsUFai@$`zq<<`SL~9pO>)qzvbfx{blu7Xzzd?
z7=RJTwJa{fpBJ;XzvJTv{bluNH)H#N7U+N;$n)iktUs?{?SIe55BkgMiO@a)wXZ=R
zXn|bI)odNIzfb*g*7gs4{Gh+A9uMsUFai@$`)|_6^W}@IKd)x(5AyMY{<3;3w0A%c
z48REFT3)pbf39I||H#J=`pfFkzfSp^pbfg95AuBZBJ0oVSo=RQJt6wb>WR@l1@&)0
zA83PI%WIe6&u-TC&rFYt{<3;}v=6};OhJ7=>ErqGMb@7;vG#vqdQ9|})nlW*3;JLP
z#vs@7#%1_(Eo*xl)03jVtRDS;uzf%qbU`2F`SL~9pX*rrzw+^e{<3;vv`<0(o6rZ^
zAlLF1whr0nf8NU4{*8|x^q1A+qkRa*U<&HrB7HnxzR3Eshqph1jUV)v)nlW*3;JLP
z#vs@7Z_DuK9jxtPOizmbvU>DyWBY(M=z>0&<+}`ju4j4<XL>^Pm(>%aeG2NgKp$v>
z+@E(Y!=HDtwns2MF8a&r@zFj6V=x8v?~p#8FJENG&wE(=yYcaZ{<3;(w0A)t48a)W
zTK;nx{)DXUk$n81zpNhpyVyRU4Z5HY@_hLs>(2*S`@8e;gZ{F5Vzf^|J%K*Z2Dz5^
zFT<Y?u(tQ$;|Kj^_4sHXf-#ta`u9j5&zCQ<{(O|RU%|%@`pfEZ(B1<BFai^hYx(dp
z{P_rLdrziE`#$9(tH(rp8+1V*3_+eRUu6A>So?c1Ju&*r>PgXF{{hAgXoD`uwS0UT
z{(OS9y*Jb2qra@45ba|y1@#|7AIS6Ni>yDNW$pig>9NsYR*#GJJ{W>An1Wo(r<dW+
zXIR^#m>&H{%N##wZ-XxAgCWTC<%_I8UtsO;!^aQ$%lea|z5ZitAJ7I}kZaku41Ye)
z+TNFsAM}^i6QX?#rl9^4=mUAae3A9%ORT-l#}E3;>T%KD2SYFhQ;=)<;xhcXiM72S
zA3q0{IeyUI23^nxLy+go7g>L9X6^6K#}E3;`jeu){=e8hpbfep*YcHR`14iP_5pnS
zpueo15ba|y1@)goAIS6Ni>yE2VC|3Q;|Kj^^|)y7gCQ7$Daf___cHwXI&1qtK7M|-
z%<+TvHt2#r7=k=szR3FXZPxxleEgultUoE*>p#c#0d3F)xt4D(!=G=lwh!jx2mNLB
zglHdwDX8BHeIU=5FS7o8kF{UP#}E3;>T%KD2SYFhQ;=)<?lSyISlfT(<L4L496xAp
zgD&WUA;|OPi>yCCX6^rpk011x^(RGp{g>E2pbfep*Yd+<`12#y_91-ypueo15ba|y
z1@#pAK%OsOWc~RWYyVI_e$Zc5kBjy`7=kgFf?Ui0F2kRnvbM+Y@$;)?jvutQK^OGF
z5ajvtMb@8`wSO2NKj<&(Pm1>Xud#hV8+1Xg<rmBF=a;PQ!}<6@e_1^t+Q(oD>c4?L
zkmt)6S$}@V+8ccQpuemh7wvs81Y<A-xt8B7!=K->w#V}E^V?;PAGEhY7xcjp<oWVN
z)}KGI_Q&z@gZ{GqXul)<255l}=z(0z!DaaKN7nXurYAsuSv@h@r=b3Oj2qAfdA@v+
z_2+M_{RvEui~h2De6$b27)(L^52TN4xosK#{FSwR1k+=pzpNe`?Oo6ZLofz;zI>7O
z=ZH0$b|TZ0qQ9&j;~?d4fez?_0m!u+#?~Qw|M_s%_K{3ag#NO6Qnc6qh;akjpbPSR
z`6BDjJy`nzrpHHrSv?WjC!qEx(q(`a$h91~41ey<+MdMpIOs2{$4C1RjKLJt|4jOL
zzI>7O=iaRS$xM%l{<3;(w0A)t48a)WTJE_Ff9}QFp2GB`=r5~B{|m+qXoD{3gFIip
z$og|%*8WjUPl*1qdSbLsLH#!918tCNIcgdH+=sPY#q_x7FRRB#`w)!56x9Dp`gp#4
zk@e?j*8WsJe$Zc5kB#;&=z}2`gIvq~m*LL?SliS1_(6YJJ^J6UeLx#@K_BG#@<rC4
zm8`wV#}E3;>WR@l1$FJT?Dt?z&<44d2Q9;&2eY=T`S?M9Sv@}5hhPk*pne$X<N5MM
z)}M#6_G|d~L4R32Hrl(O4~Ae2axMR~41XTN+OFl}2mNLB=!awbfHvrYKFIUsi>yD#
zvi9rv_(6YJJu%v+pne4Ofi}psJbW4c7_9AjK7P<&R*#SNAsB-xsNaqB@qGCr>(3)t
z`we{jpuemh8|_`t2SYFhxt8OX;m--I?M6O+&|g-Mek8UJXoD{3gFIip$og{<Yd^@x
z5BkgMiP1g<^}9nKXoFnKBbVXN0BgI6k011x)#Iam2*zLv>h~ahJYT-Z`g1C4|7bpb
z&|g-MjrK0+gCQ7$T+5@D;m<18_Az|?puemheFe4;XoD{3gFIip$ojLEwcpIg5BkgM
ziP1g<^?O1eXoFnK>Sg$|hPB<o#}E3;>haM&1Y<A-^?Q*%o-bcy{keywX{Yn?gZ{%+
zx$m#D(cT4pFa%?eYdMmwL-zT@yR)`u@bQEGvU>D;WBY(M=z>1T^W}@IKf1*}3(Ln3
z`pfEx(LM$Be}F#F2Dz5|uyx4#b6?i>v3&fXzpNe~?L#mIQ&2yO^znT8BJ0nCElqnI
zA3x|XtH(xr7xcjpj6tsDfovVJ{yd1aJ(G_g^q1A6-v`?Vv_Tj2L7p#PWc_)VrD>1n
z;|Kj^^~7kOg8F@-541t9<)LgH?wBU^{gB77wol;W2mNLB_-G%3F_?n7PWpJhe3A9%
z1dDzCnvWm!m(^pVy$kwa2*x1SavWQS?D#pJwS5vFKj<&3N53Do4`_of=z~08zR3D>
zvZZN5eEgultezO{Q&7J@^no_WwG6Oz$og{<Yx~c9{Gh+A9v|&PFa}dle*o#@`SL~9
zpVKT&domwC=r60sMtc|Z!4Ql=u4NTlhpa!RvbIm*;|Kj^_2@@q`+zp+f<DOe<%_I8
z>n!&BFMRx<zpS1F?X?GzE(5ec2jp7Tuyx4#vzE0zi|O&uUsg|u_A!`(`h(yb$n)ik
ztUntqO*@<EvC&^vkB9aF7=a0>J(zTHE$i7jWc}H|+CGixG0<ODkB#;&=z}2`gFIip
z$olhWi+RKJr06fJ$5=`ETc87aU;uJ0gKQnL{%m4x&tZBZ^q19>qP_l)7%!j=x**S&
zFS7n@u{7;mrpHHrSv?WjC!qFEq{{#;kZXAiTgQ~-@qIIEdmhu{puemhAMHah22)Uf
z2<hYb@<rC4$6A{9bf(8de_1^a+IwICMqmPREoZQG$ogZkw&ycF+CwQHSv@A&+n@{j
zU<mSj`6BDj6D&=82GbLxzpNhZ7>pOt0v*r;xt258I%NHMJZt+*rYAsuSv@h@r=b2Y
z(r1D;$n)iktUph+G;NsaanWB^kB{~t7=tOOKb-V&El+0akoD&&tnIUy9uxg#_1I|d
zf<736G05}fi>yDk#lAm;=}FOFR*!CA`+zp+f<DN#oXyrDJ3l{-wY`Ap3DI9xPa@WL
zEXFlxfey&?<%_I8&$cw}IZTg-{<3-kw2#09)Q*EbkZT!c>yRBk&th#aWO^+0m(}B-
zy$1$h1STNQmoKvZj9HrYT&71mp7N5_W1zhSI-myz;5d7IzR2pCve@^%Fg+3a%j!wc
zUONHX2ed#3oMx}j7g;^?7CV;m@q_-ddIGeMzy#DD0e#>O_WFD|lpgkxxa|AhcUqcO
z;o}GWW%W2{?|}grfeFa9@WpV6o&FWyAAY5!X`_7noVd*KgZ38afF2ltyV&dVMb@9^
zS?v38`1nD8S$`6=*B*)O16rU1axETThvfO#Rjlne)8nDPteybvBQO!`AAmmadF=K1
zawvbq_Iib-X;r4jLVsC34%&NQ07hT}axGW0b;uqsUe4NXWqP!emN|aV-U1!a0|Suf
z%NJRHUTv}8n`3$+^q2J~L3{0FY#-199gu5z6<dew@!}fRc7l%|^q18WpnU`;V*RH;
zAIS6Ni>yDpElsQO@q_-ddK|R(zyOTE1ms#?!`30|&udxRi}?6?)H25p+FPIldSC$Z
zeEA~l&l@fFdwhKSpuem?3EFF`uzf%abU?1<^=uuo&$GROwVmYS2mNLB1ZW?DiCF)s
z&<FB-`En?K#PN$Cf2a8PL4R324%&NQ2*zLvW@5OWs&<(8yB&ode*B$gdi2wlH-1dC
zw?PjKzz7uUHfH7Y@bje^rYAvvS${MW+Xb{i7xck~o<sbei}2^|OpnX-gy=7;Cr0}e
z)K`-}6STpGo<sN}j$dzMdM;*qT=bXK<D-2D#$XESYe*l@moKv8nV+A`GCd~x%j&Vw
z-US0N0uwM3lj*tr@gw~C6w@=u^k{3BH=Yf&w?GH<zyM^gtVL`cTP4qbKFQkFnVty!
zW&KIS`qp7wgBIw3pJK1imqTqY;m_wR_IsC1kB9!UdIGeMzy#FRLm$Yse1@$<_Wb9w
ztnCh_$3lNuJuce&U<k%w3i5pUBJ0nMmZn|8^ynLwH-1dCw?P;5!4TwHKF`)6`+nsY
zSlb1rCq{o+e^Rv9H&PxZXoD`u^W}@IKe450JDDCI{ZsrN;xGS}y-de{LbQ)XKTscp
zK9Ik_mn`}5^JUg{7t>>-zpNe??R_u=WAQp7X0Oi|;g7zF^it{4zheLThNWqjF+CpI
z$!@Oz?ISP|>v=S-&j9%v|C_BtcK`Z1YkN7<W1+vS9tZ6`FaRSk0eQZB5&lrr`mgZk
z$CjpD!Sra4p}b`E7-(;S4(Nda$k+Tswhmc;e#F{d$@E0%FRLdJ>)VX&16rU1@_hMn
zD1WFG`)rf=qXYl1>g#h;BNLVB@yY6JPKj8*JG*n&=+29BbvzLDEfpyEzpAfSMklKy
zm6jyL`g>h}WqRjqWqMn+=%YgV$7ZJ7O1;LmLsJq;>aWbsR(4VnO6s4UnHDK1slRDY
zP(}a9=*-My)3_IZs-u5ry;>2fm#B>Xv589l$VE9Qqkp<G)jV&CEuf74+H`$Th;E`v
z`qwn+7Jn+Ge@At!K9k=IRMNjrnBGKX^v_nexwA84P13VJmC;|XF4T+SS*(oync8?8
z`ReGeE3SV*as6Y8>u2M?&GfI%_;<IB)OWfqr$EZgKTLn6UZ1Uv&exlhP)Wbi$1mpp
z*komHu5b`g#{Y}vW~K|;d8my3g{gLzzY6KEjVpcq;?A%uu+{;j%=np{t8QzRug>^k
zw>~FS);~J4&_=#O{?APA+}1jUsx$s;)6M4%Vp}Mqf9vE-rQUi4ua^GSBY-mc$7-`<
zxp!V@1Sz9`YG%5M_guu-=Tt|3Wvntjp1=8~l72S*v^jlH$A5PH-eSH={!h{PDb7I3
zjQ`5mWUXGE;`?m#qKy7gx9vMf%IM!wnQfPVGWy3Wx%(z!3@M|(UQ_%1JFb7OUYRO<
z4oWHiGyU~lV(f4#qn};9S7&!riZ@Y|(a*<E-ui4aD5HP0*6#F2o$<rw|8^-T<Nx&h
zRCTtd?)#^+)vaUmvvY;F-_;pEc>lUOK0n)hA3~Y&!}QO%g@;#Fn*X7{Qh0zvCI6Lv
z|AqNKHZwWN9$?MgL{!OtrQbhg{?E>|-oR4HfBp#HoLd=Fa{ip3Z~Fj*GXAsoe<$Z!
z-$7MIe|@gq;~VOnKWuBCfI|LXJYSt{xdW|?{_OZQy|q?&6ICVs<Fzq<^sO{+1!eT}
z=Ql>POWxc+O6hO&9=0<2>+Au<k-6!LJ2x@YbS1Bhes=t7Rj=0lt94DE+Yvi~GXAss
z4{hYDbNsE(wUe)s{`#&q87MP;`1=<s>*t@}QCa`k9DDe7x^VWR&iskThsUd1#mq@m
zX8d<ZKo$K={@Bbk8$*@+W0b1sAL)>QD*BoH+Vli_2cVvRf?E~+-2Zk7sG>iUKVEHl
z22>UO<bP`ds_19_^S3XGxBt~Sf9sHdQvPqPOw}gyw}4d9&&U6a+nRtX`ZM`$-at}C
zKl$I9fGYZ#|CN@9_t+_@D*Czq?GjK$e<pvX@bHy#{<kKel>co$0I7<88vm^csG^^Z
z|4H`!SM!y^S7524pZnh~0af&8@@KbIXGSk#Um`NXp5?2sJ6e^VpZ{0mb_x04CIQq7
zsJT6adbN(nla<EyXe6MJ|CMdq*qf?_$I?~NpB)d{B%q3ZHXpI4x$CvLdTnf^>8oi}
z(a-&Fmw+nzi}FXZqk&rUPtJcNprZec$&>w|ivHaAZ%RNF{cQZtRxe?<qnjV*Q_TN%
z2`J=0dp~d6w&D}rs_5tAzg+^V=x6e)JF3(5ViKz8=l-`#Ko$K={&;n6Y_{g+A4pe4
zKli^~0;=d|^2aK3O%Iac7^aGT?ti-kRMF4mSJ@+3O=k}Drz-lh`KL_+s_19_xBEDQ
zQs<xT5>Uv0_B_gt>g=4NpGV^JPn!f3@_)SQR;P<!DyxcqI{vmMpo)GreE1!Q(aK!@
z0!$VCng4APP(?rYe{_U@B5!1_TCX>rtW-sR=6{<6RMF4;&wkQI+0PSpNI)U~EA#b<
zmd`}0qMwidb_uAWAM)87kyE4WB&cy6po)Ise=z}7^fUkGCfb|<siL3z-!1`F^h18j
z9UxWo3;&A=sG^_wU!UVwFojRls-mC!-!1`F^f${NnW{7%j8jE_&i|$aRMAiV^G{?q
z4!zkQs_2LREeWWipZs?#vvc`(z*Nx>|63AJML+q^zR=WVUkTK73qlqB@V_MiRrE9e
z*;7B&>B8+#rS3nsOF$L>Pu4#2>(5=uen<LdqPODIGjBP+_4$+Fi&QU;5+Xm*zf-;V
zJ>E|Br(yk_>i>WDKc&m{tKNTAm-(0dX}`Fm<M*!iuiNq4t^RKJ|8D0WtIP3U`q$m~
zzw9(EJO13!{pT|F0{r-ONBeK-a{F&Sy*vGzyIlY2UH0F+r91tnciF%6uRH&{9>1$j
z?`r?qT=p;h>tg=BtH)2*`%gLlcXj-}tLOi&kAGbrf4e<?UeV?HkG-P1<Ii2a{%*(5
zWtVlAf4Apvs*k^Z&tLW9hw9@`_3QU;?_YF#|KzT|e^KV&-Tjkt`*-JGzmNaB`ua=x
z^>=ywr`q_7zVpc3`-3g>Pm-@MzJF(ZFJE8G-}`>PzSuwdLB77&U;9zMzS!UTalXFz
z{)P9Ge0{NhbRb{fa{T&fzP{Ms`dPld=%4rVe0{NhbZfreDvrNj=Ie|7tu$Z1*~(x4
zc)!Zm7yC!Q&eylB|F`-2Vt?y*`TCai|2|(|>>vFhU*EF+Kj!O;{jERc>x<`)-k<aJ
z#s1M>^7Sp}|9{Qb7yDa(%hwmLpFC|Kw|^D;M~CI>TkiiKLG|MMl0^Rq@84T@qk11S
z@rJ&5KVQ7RU*!K)J$oL6Jxf%t@lQin$LX^XT#H!0$mg!oS6r`%g+Bf~PP+s|eXr|h
z^4oknyjuRZOF$w2*>{sq&uyKV&3!0Z75&-sXl)WuML!$=b5-^WQPa6!cT;Qpw@W}F
z|K0h~$=X=q2O3q;pUwYm5>Q1yKYT68XWxy4AF5VGf5HDo0;=fe^UwTf@zXSl%|C4t
zP}G0r#t$F=%>zgzppgIk!{|#N1jPxqD*Czq?GjK$KOa65mGPNN#D$H5{<lj&A^+L$
zBhNQJ2PZazD*E~OZ<l~7`kDOs!bh>PO`wW??ti-kRMF4myEE+czg9Q|sG^_y-!1`F
z^fURh^Qs?z+a#cn|2t~cOO!tUX_tUP{?qsIhy$;J=RfTdP{{wOO1*UisiL3H|Lqb`
zML&~Yn`S?+S2+AC_5Mq{1QhaL+`i2n160w^$A7y7RMF4nv)`l0dCw04s_4)BZ<By3
z`i1|^zXhVC|Ahn;^1u21f8$g_75&-uSJN99A^}zOv++MUGqzpn`KMh13i&@aKRY+m
z^i_n}6@V)G`S@>_fGYZ#{K}3>ZL%`TzLB=^#VT_>fBY5CTXFx}CXkJyrGI*Tdx(76
zr!#^tEMJ<YRLQ5q_Gw9|*Yj!UW49Y;tL%w0_ORSSWvp@H+Nhu1xodRiMN0z@e~J2*
zivQ#K?0DTK0ijQU=j9sy^9zUS6#Fq9wp*y8Kj(i_0;=d|^V3|F{T%mbb#iGkVq;hp
z{oMa{38<n!lb?NWe9nHw{BKP_A^*45mcG2du@O|!&&Pke1XR(_<WG!ND23)X&Q#IQ
z{co3mD*BoHdTpw@i+x&Pq%t}>TirpsfGYaA|Lqb`MSrvW?01+7=a0R9{+9E<DFKE2
zZ~gLe#m;})C7_W1a}(^xD@H1l?6z0-(0VojsiL2c|8@!Vy8fp76Qh-}?d%7<_%CqG
z%<kmh<zHtvjv8k|G$E;y58oa#3H5qD4Sn=}8s9zeLq}?C51~KXKiGABF#%EE>&G{~
zdvp2x-cWN6y{=#E-wpqZ3E(_Kpx5(h=)?Ay+c~H9{!mDukzbGBZ_MovO*!<s{`UJr
zQvzZ~0DC>3hCb{M{F&wE9ima+<Kw9@KNS+_b^T&I<@Sd{4$VVKm3$i8BRgJnPN>)O
zY3QT<qc+og@MasPQLo7UQA|J;{n$U4{QO+2ihi*_6cbQI|C0ateOVR#P5$Q+P({Bu
zKJe?e=I3fv(J%bZrq8AXRM9W|=d(|feC6i<Z2D|UKsEofomWBqjqzVdKr#Pu`mnSC
zRM9W|FD9Uh{-xt@eg;xSf0O^Y1XR(#<bQq#Rz-i4|G5NI(ZA$BKLc9&WJ|B>7uWYq
z{^t{DzPMNDdU9!d@ZDMI?U7GFvF#zo*V6XLcJHMPqG&!%`^)Vn)a(0yLm%$1vbTR~
zJ8JcvVzV^r6}f*jQb<4*{n$U)*D$%cFSS!ezt|s&3G}-D!u-@Me`IcQrucY3&tFfA
zeA;b~b_w<R_G#!7?}tv#&WpK;vr(_g{4|$OKo$LBe!8SG$@Y=4YK?ueM@jz+38?0O
z^LO?s>3<;s<@}$+7oBqR*`kX6#_?b-pMWa*#r(5vZm#0i$a^LIFC?Il|4pZ+OJ@?Q
z=;!0VT>`4;7vq1t!oI1XGFKIQfGYYM^G_jxUe_<qr;Gmc$A`DFE4AVUf-3p6Y>y5J
z^?E)Hea+8@ZJnQHw=rg>*|Xh^dR49uvb#x33G}-D;{LJpIMq@PVy;r*crEg2>>r0n
zsMohoL!Y?6H#;^VuH97F9zuU(dsOlXh<a6?uOHoZ$!x7o*Un1%Ur0bT|Jmh#^K_)%
z@u84Fulq0NQ*r*}&dzL`t<KGjRHiGFJ9jnCW5q^MC7;H8bclp{J)ed?93L9#?5K>*
z&rkLI{vpO!>-InbVqk%de0%hKP~_9Hzl=@J&ov!4L|Urk)4F}yC!|U~t=ng|%D&Zy
zp1l+shf$S$+H9Y;38|7#%l6?nP51|93qP5mN<OXIr+q@bo=-y`{rm*`4s6%0O>b*_
z3cFF?qw}-S$G1mzsa{Ax)c5N3NYnA9`F7~cY;9X@I{OIld}CYiX-t)T`1X)VsMqrm
z`o=1=<281QJTg5$HM(@U*z?;%=pX5jfT-`){X^&*o2bmPM~T>nP5BM$r31RC?{)nn
z9THGQ|J=pwb426pI}&SKYfZyno$=oxfnN7t93N`a{FjBZ@2MG?8=I(3RoEU~uT0Kt
z8{yx6)Yu_Z$%mboOC{9n`3QadyTJJ2daCdXaH78F=cf({sG@&ob#7#4c4T^{u_w?U
zs^}l-kbo-s*<}NNPL_ZBckX92Rnb4vA%R}kFOCnD@u}K$ZI1m0z1RdJmHGMvyN=kk
z^yOx3mry025y^ylJs+X3Qm@a}M(69w{ytNO1VnzletcUx{hq5fKO>@w{-*1PTmrqW
zU(8SJlyz)7f2M^`QMKthJ4(#gYwXKk=jSThmQG*QnV%#R>h<j-^cB+)H=RU%ukYV%
z|Huz8B%q4^v6;!q+<_YBv8w22yLQ_IdR>2GeiHKY6K&(Pur^5_!c!%mrR{P13H5qD
zLLVDm^HbA2D~_duhN$oT`Kf&Zs_2LD<1<t2=&i8-#RL@d-<@Tj2A-^LYg|UCqF?N}
z#ROE*FD9Pxdeh)n;Cfp4EBv1?Bp~X0ef}i>#aSs%j@0wNB>{E(Z(O;FT_L-0>vjF&
z{Em(P_6aNvrR=X>&qwHMN+<iNR52@w`d;6ko3}@E0;=f8dBoCYQ0#hhq(cIV`CmMo
z_5A%K;ji$&n1HD7_5GKIPs`y<HUC=@P{sf1G=HOKJpX2!O6Nlz5>Une?N$DZHHDiI
zD*4|b0ag5Gznse+dtuLm74~3N^pA8%Ko$M7)veW8{x}$WB(nJ=sMqy3t_M0Lpy>4g
zf9OI?NZV#B*>4&1RC;}T2z}XstW5%<zSqx(#M_yw-rs7xA6rO39sjc@kLW$+rX4^P
z{WSi?C|XKD75z|NI2tQ8|72sQn1Cw&(=)-%PsY=MOcnijsJSHpRrGiI0$o+~H+>~@
z?hHs3{rvI%t?Uz@^X$Xy%?YTYf22bKs_4%?UNPlPR;SoS0ewt?KjGi==bw%1fldiD
z4<`1vO51~7Vm3eOm}R1h{%n6}lYlDv`EZ)6?P4FL%?}_|^k@FJNkA3-^qx}loeA0w
zs_3WR)Xd#lQELAs{|gtBECa>-Z%IHE{n`B>+6YS*lPdY&ApuqV&#cd#iK;mNv?id8
z|Lh*<9RGxAAq7?RkFbwPv`e7Z^^5aq_GzbW+t@n<<0JDf`^3JuH(l5oy?;Cq`LOLF
zl~Av5AE6KFD7QU?ez84@35fb$f4_h~_tA24q0;fWLjtP!pPgRt9CBwzy+0obf9d^#
z)&xX-&yRop^k|zsKo$M;yk%<ws_3WjpKXNP9-xYTjQ^GdRMDS(SJ`-VEBgrS_(*-H
zEACpWqMwa_{#{Hh38<ofN3D8^c-&y=(_{+G{~Z!g%75{8chd$?MgK_Ve`^A&=+ETS
z{h=oJRU7}U38>_M@!EyoAyP$uHvZcrpo)I}ErQu`w{QxlivH|t5Zff6ivB9SQ`~&&
zlieUyMgK^L1bSV6<9-(VepYsSv|8IX-FT~sKh0j;rIpEtf8ne|LcN}k(1*_!%((d7
zq-^I<>G;qg0g+#?-k)f<e`LpsOJ><O#I=0fpy!X*jeMH-kK0K|mF**T_2yJYTK1OS
zKc8xBAIXGzeftP~?b6}L7E#};`@hiFIF>XIFqVL*?|J=AheLL!hW$D;9od!9-#lM6
zC7_CaG5$&V(#)ia{>Jz(B%q4^y#GsQwW{bZ_}@rC8U0Q3PxG@d%II&Nf0`0dMZXyT
zbcbx|#7GtWjqzVdKo$Kt{~LR+Lgx=15>Ul|zWb{9dqBe9kq!xn`d&Z(Z1#TX>_?&T
z-ys1N{coCn)VY3X_P;3sRs3gP^j@bMtK!ZbQ5F3o9THGQ|C0QXZPn@OEPDiXoNobD
z^f&pROF$L<P5#r*M%8y_JAf+soBeN2Ko$K>{%_q*Q;)L#Hz%N^|0CiqubT7E*6m9P
zDCPfLm3;t^JzmGZDh>O9D*9>uX-z;C{n`Apb!L9HKB2JxtqJtH|6-<@t!`uAgfPdi
zBFE>aT)f9$nHn*h4hHN`Rq|m!CM%VYD*1FwM{Gbv<|`q$^Y#%fC?Vyy&zirt{U@K0
zs@sSCG-~!pYjK8CC7+i4r$a)j<WrpAn$pQl2&&|Bd)ucuAyx9Zo$ZsG5LC(M_O?%R
zLcN}k(1-mcf3RVZsPFar|05j|P(?qx=EXO}j#ZoP1@*lDVlEo#lt5ubsj@x9@q@02
zmy%FLzt|tx_w4a|5{(3u(Lc|wC1=<Vob#`H9v6306q=9R$;x!@mtScUDC2*<y1>5@
zr1`f}c?PQJpP$BOMH?euG5^_b)~;(lkw6}*_&+u?#dhCXeQD2C%>NDvsNz4}53Ox2
z+zwS{|D|)Ftt|<t;=ee5s87^pmyW*b%%6?(he86X`A>J$mkt7|=x_L6NI(_++5E%r
zkS$rC{iTY2ntxgoP(^=-)6YZB|KfV?QgOqE-p%1p7xAxf*>e7AxD#-B{&-&4%#Bd^
z)hH<ZAzR<!bX3`VI$Yi;n@@JBK%WmEo1d%COkq2V?c4MBk2)lvivA9l3(AhC+~eR)
zbA~G0Bma4Vr36&bpB=wz(-YO%TD>yO?n$!mMrYr&f5`D9@>|L{|JVQTdU5=s<9C|`
z@^X4UAK}+jwKB(kYl8pUK`sMP-}CcphXhp7-z0xz=Ni-8Fhw_KRM9`uA%R}ke-1|b
z5ly51QWiLyPCN6K^EE9Ue?-Qn>b8FxSFg(2iF^;`N6Y*-Hc^{o7xJQi&+l&?5>Q1y
zeGIS0?&2<tuy+9(zbm4Oemnrzl7K4u#rO_y{>ZOCccrGen`wU&`lo7RBlVe)+T6^@
z;NbdABjQ%Is2BZ%FH(Q8Q;Paam@_k%%#EyGv)<${#9l9JBI*y-AM0;^8L9r_{9s8q
zUwg~TXx6Tia`QzeJ>vD6U!k2b&dcOpPvqawx4PM#*7>KCzs2n%bc)ySl#i%ytbgsD
zv;MUh0OIwf*T3%0S^v6au79+)Z9QM}e0}Sn#>T%`WcBlYygz@PZu)PZsSS?IinYsc
zf9luBpz#9B)t|l2e4~5I@vqbPqki45U+h<d8#ZJjWc4@o%ZZTfFScLfbvDunlI`Er
z?@n1ipC`Ei%U!>iU$b$%jQ+*^+3RFe$Q@XJHiO*>>(34`cVPW%nm2%K|2Fw&Q<!Xj
zvHh8E*$jFo<ev?|JFxz22E7y3zy1!6zomJ!yYZJz0Cyn&4R_M^-*^Y}-*_jjfA9{j
zKRXUBcl%f8$4183S2f}Y$XA^$vi^(xhh9hYmY=_Pzl+$%NbZdFU*z7g^>3fObJoB8
z4jjLE{`E`lob}h6-h`9ge%nS@51MPUQ*$wN?$Gwf>ohii{Q9>ylIPbSVv+4H#^1Kl
z?OHee8|T5ez>!`5QolwTSqifKh5p9t-~?K>|5Cp@Y5h2XUiSKPM<4n9za{@oGy;~r
z{!MpM{;TPOY|CE%>ij+=+rK6M)ieNP`!}}#>fGsv^!a1!`j>b9yR`oN0Fc$+y8i6+
zLt6jR`g5lr(*0Z4zr5>@rS<1dKkoSYDShewjrHf#m+s$^Kcz3-zp?&&`qKSd)}Nhy
z$<H5+_2<sM<oma*KRf%9@84K|?(FLh^v_*A$@g#3j~hU;``<+~6P3~Y)Om-FU)k%3
zgEyn>`sXgH%{5lv#qpoN&K<h{GLhZi<TiG7{xCl=c8B_B=fBJX_EuAF0+sboZ2#uh
z;hRCW|J($7^m2r6guDXT{$l<4*E#h1#eK?V-FzGrUne>(|9(!V`-$T9^8NWMHm`xb
zpR``?^+f*J+biso75s~CYctc05iaUG>3PopHT7^`T`)e6?yn0*&p0u=|2*_e;(l;q
znBs%r;<G4T`E24IaN+{A0~eozb{C=@IQU$O_kqJB6dwQ=9Ew*|h`YgoQHuA1iw}Kd
zrg!Br#695P!zkVd4nLgY1K@%|@rq-KyTO6uDBcSW9Z&IoaQ6um4?F_xz@Zb-4xD%-
z#RtK~0g6|iL<~UlWQrqj=oE@$aL=PC9$ZD-2M(V~@d0q*G>TW4#NFV)YKr%QLu)AB
z4^FJ5_#n7<9mPZIi7`05f#L(;!bXZ$3=(&P1Dhz`3l2S+;{BlW7>W;o!!EWvxNtGH
z`z*FQI53Cp4i42R-VaX9Q+yEgc2KNeLJYu(1+)VfcT&7^7jX|bcqzsEz~Rd%J^(IU
zPVtH>h`YgoD=FR!E_i5n721IV&qF(K==l`y2Pa-Y@j-C$g%q!R5pfSV_%9Ui0~Z(3
z?!{;a4!#8Kz~Prtd;nZ{8O1BEChi6YUQY2|aOf2j?*|WFK=a4qbBHT1B<=wRpG)yR
zaCn5`1K@%~@rnv@H#jg#@m_Ff4DH6z4jibW9XPa=;{D*nHi{2|ixU*DtP%HsgBMY}
z4;<c(c9Uoa4o;yRI6O`90dQf4;uS7&H#l%H#e2b_S&H|A6LS;~)X@$cnnycuVh6<s
z!Np4`Ub#Tr0}k$_cpo^ti{b;|!lmd}q5W|WI5<l2K5%#p`zyFGPVtH=aW^=ymEyhN
z&^C(qgA)@J57f{O9J&bYz=`b?9|RXCDPB26+yf3yQ@jrxo}u^vxZqMecrn_6!?P5p
zza?6~Bl=(hn!l%V4~#+M4^-}g5vU)eat91S?T=J$g8`U=)}N@}2NTfzGnIQ_3>tr-
zau<w1{WdChz!22_O64{ffNwEy>VWNTVg5yF15?<q@5V~_b1!p<4@bO!m_H}O>n}xo
z7~(4n@l}Y|Y%tert5>hvuu)rWu3M|E9@I9iTc=gDty*?G7wbES_DACNZbK~UkH9=5
z^sGQE>hFzM=+O)D{Sga28%^lhq^(|Onp%ws;(3YJTSR-I=W4{F{u;zW&u+wG{ckA5
z*B0V8BNlqr!=E+lH7;gU(+6_?ti-%3^c;g&=s6y-*ggY=`bQxa`c@a>wTPi-{Ti{o
z*oOqy3~t(>RhdCe{_H_}q33$ULeKjO@m|DY{WlciPZZ)$B5wFIi0x%ELF7=*pFtdF
zgq|&kg`P7D@epF+*I9^#zH<ukd5DESYu7dOtkpJb*pv;RCV%#!z0h+LVxi~eLcG5a
z--1}^`F<h(A!6arTKKbO4d36)jU;GkJP+geEA(85Sm+rm#1n;h3bD{LTZrq3g`Ppa
ze+{l-yZ-8R+8T4sy3C*@JqOTU=(!cK(DR!@e6SGThFIu19On}vz8m6(p4CFnAlq`Q
z*}Lt0k6+TWfc8Sq<%orz=NIC|LVPu1vHn*T;#VVXj32(02HAX={j+JqU_s9coDT^-
zM<Et^9#Du^7UE+N3q8jb;u8=z{9&6ald^hngSL7tn;!H2>_&T`=UT)<&vk`(Pa(b@
zvC#A0Li~Qj4L!n|HSEERHRigF1w8{ePZWAqAr^Yp7UID|yaln)^SDC%c*G4oeAKX^
zoDFI=o;PKOu%+>{7wv_f8xRXUpDx7v3h_;dg`Te#;;$iY=;23I(!-ouyLtok*k{vm
zBLGv-dK#6#awQw|e8KrZ@%b96)U?ZS-YxV$zYs4L;;Ru0{jVy-uSN|0>({bTmg&#_
z*|cGGVf(JYdA{)TD8xd~0}AoVLVOHjq35_ld;;QzpA9{1eqh_TxIgYjd!grA#6r(?
zg?LXPz8<mA^WH-Ie#8wu?D)HyPgOLDZX7I(w*g%D2tBJ13q5NK@n9j|f>`KzTp@lu
z;)Wh}e1V>|%%9aQ;~(vXo*NJgJ)bVb`wH<*h=rc77UHiVZs=j>ZJ8cEe%SWfSny{E
z*MUOMS%`(6^9u2BA-)i?&@)<y#}Px%`n6(vWui8*)7||1xgYI?o?8$LJwGVK2MY16
zh=rbC7vkR{COsS0uVKeK@+X_m*+2Z0duhI!!1b}vGlf{_nJdH#h4^yBLeKLG@e2@(
z?X`)W4`h1ye7<_^U~xPjM0=s<HpD{D5x9OA@rpux6k?(0{)PBx#0@=}KTJw?&|B9s
zo)^(x=(!rPSpPMJcy}Sb7O~Lt)<XO?#EtQj?O%Kn;@gYe6EBRPmAGFa^c;g&=sCU+
z4;126h=ratg?Js}hCl3lAlp>jA9nmLj-NefFZ5iGSm=3QA>LbvZ$K>ce5w$C264k5
zw!de3vVS(=gnMZ`58^(K(6a@x&~qkYalJNFsDBD#p~o)7=OPw**a3I-`VA(thW)pZ
zE{~e8ANQfX&~p=Fq334ALeKs}{kIScJxL+{KH`QRHveRL*xA4acHH2XYg|wICOYn0
zZzlR+0$!!#h<}Llfva&}Oz6L+5brL;*CH1B-&%;@hFIud$L0sFpZ?h}xV||4aUV|j
zc?@Eq=lDWAfLN@5RUtmD5U)lo^sHv%A9~teZ|*^Rq33$ULeKjO@m|D2&kcq66NUJb
zh#Pv?c3r;#qS{^m;=ZKNvjwrxb7mnPD#T|Y7W&RD#7{>o^sHm^C-g8;8wQKVyM1Uc
z^xTA4=(!njWBeEDzlB)nNec1z5kt?~^~|4*Y}RD^<LZs<>X#qwn)k<H+^-dSE<`N!
zj3E|!CJOc25eq$TA)ZCt(8EsMGCj<vjcf`n>N$Y+LeH&;g`VFa7J3dA>i>*b=+SWh
zS;U7UZs=jx3z;776K=pY`?G-dLeJ%hg`Vdl7WYdQ3-vEUEY|<ZLVOM4h8{M5ay|Ti
zCWv<Tru!!=aQ|HBISR4R^8my`&&opmLlFx-#}?w_5jXU(`IGeU4aN?4{Pe9!&u+9A
zdagw*^jwEn=-E@Ke-~n*=RJk^eTW-+*zqpY!%m#m4zk-Pd;m4+8NmAqLeDD1LeJVl
zJXnagAQpNaSBM{v7<$&TyKb2t@z2`A{Ja<Kg`OJ_3q7AkEXMP`Lj8@1g`T(&e-*Lt
zhhL?!)BbFKUxTyB=J7m)_gRFVvk(hC=OGq)h7pVPUr>lIEW{&-8+z75&suhOvcm4)
zuAukN#QOH5z0h+DVp0DC#6r&j#6r){3h}Ll_?L)<9(KV<{;b2@(xXVvA{c`!??dGd
zc<~Ck^MPyezLn5_9b%z>4`QMJorU=NLj0eIg?@Jbko0eW{-g7L4&Z$<;pZyELeE;n
zLeF5K{xOJ!KC2Kv4zbX)X*JiwPlefeXq)l27wv_f8xRXUpGGY7>_aTpe`6uOsStk|
zaYGLq`k5Xv-X4+nX9({b3O#2b7JAM@Ec6T`7J4oy#1|Ig5yV0dI{_d)VtbvO*Rvn(
zg`Qgw3q3z5#0L-yJ+~I(Ul!tDAr^Wzu}ziv!>(O0f^zQ{OyK=gp=S!Q&@+cv__I)`
zzZ9|1=N01TA%-6Q78BFMZ&kD(KL^oX=(!EC&~pUd?-hDhAQtPtcOgEi5Z@PZLr*rI
znI18IHs!b1BH9Z*S0fgBu0bsH>_#l~yrB?ZTZrF`xUs$1UEa(eF@D}dpSu_9TZ#9#
zg`Q&&i~8dW@c?3>XH_9Stq`w9+|ZL9U$TF=L;M}N-1ymp_Cn9~h=rc_Ar}7ZE!2M$
zvCtP4;!h!NY_DwpVh5QH<7W`>_X|B+5DPtLA{Kgv5R3Icr4XN0h@XbIp(i`OFg;@Y
z*!l6Z5AB7Xn-B{<HzO8$_9GU0zEy~CDa7AJ+|ZNl?@W&vKjV2l!*~ut=(!NF&@+Zu
z=$R<gZ$~WjxrKNZvCzZrzp?Sm-@0hOzaKz*q32e_LeFmy3q1!Bi}nAx5Z_ja|Ax4s
zCp%s+Jz{%JlRwdBdTwCF7Cb)y?glG39Z2yxd%?Y@Q+w?U;t=SA`$5B^`Ux-;#B+)U
zUSB-Vh_HcU@z>_l8%4Q4MExQA=Xxy#(*3tA^=bY~wtoltEzip~zc&9{y#DE3uK$$c
z3nLb<e<=S}b=m*qF8iO>W&cNX+5fp+_P4t1-(mAN#>b(y-(9``bUXexcQ8P9{C2zl
zoY>{|yR6Ip$9LKPiZ1(a-rW8Lr2X$O0A%}L*1<s8{&)5KcUP~!!|qeg9~jn$nm@ap
zKf2xi<@W1x{^?+#?EKN8d%6GoAMHQo@wdzI-{tYQ^uP4KoddG-r|RdQ-5$To{_RH3
z(*2D?oqwr*{O$VswY`1P=O5jkzh2Q{2bSIckL~aRvi-Zg{wv2%`vD~Df4AeO+xf5C
z`;XoBzhZOy9YEH<D>iRw`vTJaPw%#W+to|YAN`(xhudG${&hS5cRPP|yZ@fve)^El
zzrzd2j{nmC_8UOjztaD<2Fm)^!M<|*!_`BbznAk*+u<iYe#-g3JO9e#Z+HH6d;Z?<
z{WouJ{|2V)@#n-Y`(Jiq`y-g_`Y$`ZyZ-&2|6RTPPV8_4Uv~Rj?Ki*l@oTfy-Tc$-
z{hx08cYFLTpTFsD`<MQA=U=zy-`(CnyQ`1?-RdvL-(5TYRKNez?fl>E{@da3Eq(m%
z_V{U8-Ho5_`?nV$eg4xyfNcMA{C2nfyB)vXp8s|`{x`2`pMmW7IaV|**eHq*b^q<I
zK7a49`$*3}-S#h!--UsLP<H%ud;B@M!v>ID|H&N=pz{5@%m3u|yRY>4J-NdTNNN2Y
zHh^sZ4)#my?_j@t|MvFF_wSH^Z2leWlU{#^%`e@*+v88S^LMxXH(TvDpsfF0_wR55
zCA<C(w=d<lUzh9eaQjty{i`l(e+4U>f4A4)?bj~<{9d=8|Lbu2B=28`0U+z&UA_L}
zJ4`^b`j1yU|9<zs!{JYQ{NL5b?+(LPUVjGzWXFHE$Nz4RpJ%i`d`jnk#%1kaKsNs~
zx_tkq+v9h)*U#M^f9~r3$9E7QyZx*V14z1mm*;O+Tye+7-~GINKVz7_*TonoE`ljI
zT&3~|9N3EZ?31$ZU-50Cb1u<*I#GKDF@7d7coxyUfM{Jv)JKSkLkvfW-Z;_TN;D>j
zX^j|dC;C%FXNG8AOw{Ixah(|KAi4`gYZp<!jF?<b46h`5R}t;!6O9)V(-#q=MWX)_
zqVqDM`EsJ>6XRDBgKLQHzY?w267|;+lh+f&Hxj)!5$%9zyp@<<M~vQ1^xr{r{+(#P
zlc-%!jNeTRLZbU#qV;~F{y}2$A!7JpqW4jveFM?>1Tl?>(Wi+1XNb<{h~_?`_61^m
zBQdy%=zf`KeTAssOiaE;48KnF_7m-I5{+*Y(_4tqcZvS@h|Uj)=8uTlPl)jWV(?R<
z`*Wi83!<J9lV1_T-w?gu5$!(^jXx68KM|w95dFUr9qr_7e=rXxYIh^XM-qd35Z!wc
zt$P#oqln3Uh@npO?oYIjCK?YSrVl1Y|48&7LUfKHnhz&x#}eb?h`|X&_e7#KK-5nr
zCZ`a?RYdPJqP?1EtR<%Fh|vb3KS*>QO*A(XwbP058N}eRME6Xh^#r2+Bw{i|44+K&
zo=UXOCK@&|J%<>bNA%ApI?p7U&mw9U5aZ_%gXa=mhiHuw^>JcSC5GFGUX5sPCmK`4
zbeb5sM1Pj()QRQ}qP9SccM^k3iSFe@>q?@26)|}pF?<2hdlAuIBpNRvrY|K%R}=kL
z5S>>N&1;C-zY^ou5QE)B_w_{UjYR!T#N^Gy@GV5|I->n{qVW!5x`!D32ho2Q(Rnw~
z42jx%iShf0!3T)$hltjPiTX#0$;XJ{$BAA<v_C~OK0{1DON{=D=zpH*+(<NUB5GeI
z#xXJYD$)HK(fT@3-%m{bhZuf~=-ooJze_Z}M@+v@jDAS;e@t`^p!~n+_-O6JaT2sX
zj~KMRfEcuHL=0MAL=0LtAqK54AqK54BL=M)F=%}SF=&02V*R8?W%E`5nkQ2nf%YjB
zC!qT%inUclA2d#-I0UWJD2_qLq&Nk=)fDS%hyiG>r8olZbrdI{yPjfg1JMVKjTDEV
zHAry`I-4j?LGRHN>yIG@pt+gi2(-6QoPh4>6l-S?ebBHd4nga&6vv?RIEquyJCkDl
z@x%Z$pFnX0+E1i70o^B2tPK%;(D-MHL(qCM#WCnSh2j+So=UNP7BK+Lvnh^1`)L#>
zplegCokR3N<6MeE&^nLe7<8UaaSD3pQ>;IO7=Y$8DULvUnBoL<pGC3uY@!bu7f>96
z)^jM1LFYn>Q_y=Z#rg;_08NMD2(&8{C!jk@u{K8ZL1Uca5VWck$Dp&7;uQ3@QLIl8
z1JJBd9D(*l6epm&onmd0=!3=-#UW@-QyhcN48<wvxfJUc69dqkr8olZIf@g|ty8Sc
z6MfLwL2(FLmrxvo&H}|L=<TFf-$e{S^HPc<(7ue~1avQ_Si6GggT|E<hoI$A9D~kP
z6sMr~Jc{+_69dqE0mTt$zmVbtbYDcV_Af*qG!`iiLF>g7$Ds2Pic`>gDaHEBhyiF`
zO>qR;FQ+&G-B(bo`9vQyUP*BXTCbuw2AyjtPC@V06zl&=3_$ZW6i1-_T8b0U-A%Ff
zI-(C6uctT!tv65{gU%Z%PC@TliuE@U1JHak#Sv%+6epnj7K*jE5`ECPj^Yrs-bQf@
zI&Y^q1-*ZxSbqmG0L?uVN1*-h6epnj9~5ivB>JH7E{a3Yx}M?~bly#I3VQ!Zu^tiw
z(0mWY5oo`c;skWxN3r&Pq7ND$pg07r4^kY1&W9*YL2oa``iF@DXnus^2(&*+aRRy@
zqgcCv=!3?`DGovF6BNgw6H%Ok-X|&6KSc~c^V1YZp#2$&6VUxE#oFhHK4|<G#UW_z
zqc{eg&r_U&-WMp=ZzKkw`9+E&(7uV{1a!YdvG!%64;nGWA!vPt;uv(kN^uH$H&d*C
zjTnIDe^VTR_SY#+K=&IIYx{{lX#5YwA!vP*;uv(kMR5vx-=<i<g&2V5cPNfP`@0k;
zpqo&veUIpa#`h@>LF)$;$Ds2=ic`@05ykqCi2-Q-gyIOa4^W(d?*CG({gmi~#?L4Y
zLF?xf$Dng7#VP3hf@1xb!~ir?iX+he6~zhY{+eR#H$)#aeoJu(TEC+>2A$tioPypT
zDAo@W1JL{<#Sv)#iQ)uw|4gy=7orauw^1B|)?X=(LFaE2r=X{;%Fg@s!-xTB9!_xt
z+DA~FfbQKW){Z3lpmBGKL(sYh#WCotpg0A+ds3|5ix`0By(x}B`yVJyK=&w$wfhi#
z(6}$fA!z9o$DngRic`?LKgIe3hyiFGO>qR;52QE&-3L*uJ(%c&#!8Aq(E3M;W6=30
zic`>g2*vtCi2-OHLvaM!52H8%-G@`G8AKm6j-@yRt>Y+;LFagiQ_wqsV*L@s05nge
zI0EfQQk;P90L9u#L?1LxrZ@zxQz(u>=TQ`=ptp)*{ZwKAnx|15fwoC;0=lay*47Yx
z&{#`x2wLkXjzMQV#VP1*pjh8X3_x>`;s~@iQJjG8qbb%NL-avoGsPijZJ{^@ozp2!
zLGKKTb&D8)=3^<2K>KkNC!l*K#oFVEK4?6F;t;f+NO250Pog*ly&;PAe<lW?`DBVC
z(0&TV3FtnRV(l!V4;p7v9D>%<D2_qLrZ@$?b12r&B?h2*9>o!8Kb_(PbkC<)dj`=5
zjb~CEg4Qs_G3Y#t;uQ3rO|gCfF#yfyP#l5wg%l^C`&^2(5uy(o4#gp8RVa=@XO!X;
z^u{RG$B6-GRw<4^dn?5W=x(D}n;`n2QKL8nt&1p*L1#O~Dd<g7tWOaG(43|?0__=!
z6VP=j)-ERcpfO8v2wHO#$DmWEI0e0ViuE1D05mV5I0EejiWAV?NwKzz=!3?k6o;U7
z8O1T^TuyNcdRI`aUr7u=)1x>7?W-tGK=*kRYtJY8pz#8VL(qC5#WCo-h~gCV{)J+F
zkr;sHiz$vk`y~`7p!-sawU-fn(72jnebZ^#aXmzAy`0Kp(0K*L!H4Mn-^#tj_!;0W
z^!@P0cZe<+f%<o;+yO&SOQ_ri126@x?@_%ECZPF!D)+z`G=4zkE*OFO52@S%Ls0t>
zmD^wdrl9p>s`tSJG=D<n9vFkh0V;RF2-N?V${jESwVzVC4F+HeT0f(DA51{==Tz>2
zF=*UM<t`Y3`Y)*50YgyxC6(J?0H&aoQoRo*p!q8*_rMr5eof^r7=ik4sN4ZVQ2Q;F
z+h72up!GYd_rU}-e^2Ee7=y+isN4l3P(MiJ4j6*kAF131126@xKT*98CZPFeD)+z`
zH2y;6E*OFOZB*`nA*lV8%55+JQ_%Vw)%#!qni~BMqzA^JaTt}mU<B%iQ@I0%pmqe6
z+h72upmjH@_rU}-kEC)Bj6vh>RPKTisNaLi9WVs76;y760hof;J*nOY6VSXDm3v?e
z8uzAh7mPstAE?{`Lr^=4%55+JQ_#8()%#!qn)ju04~#)Wr*apJK>dDH?tmev-Ji;B
zFaT4~dH~h?U;>&)Q@IDmpz%N|cfknMA4KI27=qe^soVwwFa@oZRPTccX#OLWdteM2
z|3u|37=ijjsN4ZVP<tqq+h72upmhw@`(Of^52JDqj6vh!RPKTis2f!7fFY<IOXW5g
zfGKDlNA*6KfadX3?tw9AoIvF+7=ijDsN4ZVP&<*zZ7={+(0U}*`(Of^161yTF=(7b
z<t`Y3`pH!8fFY=zLgh9XfGKD_it2qZ0nJrZ?tw9AoJ!>`7=ij}RPKNwsF_r5g8`U=
z)@rKv!2~qdP`L-jps|+9T`&UmbyV(vA*ijVavKc56tp%_y$>d!xsl2}Fb0i5DtEyM
z)HhMN1BRgXXezhC08ByaF;wq^321Jnau19_V+)nLU<B%?Q@I0%pmqk8+h72upk-0L
z4<?}bSSt6x7&IP7<t`Y3`k7SjfFY<op2}@708`L<0@eFq0-8^xau19_<4IKRf)S_>
zQMm(#p!UyHZi4}sg4UC%-Uk!Vd<vC&U<?{hrE(XHK>aK#cfb(T&Zcr348Rn$o<{XP
zn1H5D<sKM=#yM2(f)S{nOXUt2g4%ghZi4}sg4WZi-Uk!VJfF%vFb0ihP`L|6p#DrM
zcfb(ThN;{J126@xXHmTmCZPFjD)+z`G%lcW7mPstIaKa|A*fwQ<u(|ADQG>H>U}T)
z%@Hd1z!)?fDtEyM)GJi(fFY=jQn?KVU<z7eRPTccXpU342gaaLrE(XHKz%EfJ75TE
z+o;?I126@x399$O1T<?@?tw9ATtwwA7=ikBDtEvT)F!Fi1_Lk!ttqPa!2~p?soVo&
z(3qie7mPsNrE&)hLG5BHx4{5RL2H)keJ}ydIV$(S7&Pis?t&4h&r`VrhM=~C%55+J
zQ_#AE>U}T)%>^p=z!)@kQn?F8puUUB9WVs7OR3xj126@x%c$N56VSYz$~`a!jVq|!
z1tU<ulFA)01T~M!Z7={+(7KB1eJ}yd=TW%_#-Q<hDtEyM)L%g54j6*k3#r@&126@x
z7g4<rCZPE*RPKQ>Xe?5>3r3*+Vk&pQ5Y%2m<u(|ADQLZv>U}T)&6iQR2gaarHI=(y
z1nMuRat91S?G;pRg8`U=mQVFQn1JRhsoVo&(0CP<yI=(B*HF0whM@LpD!0J^OhM~k
zson<@(0mP*dteM2ucdMqj6i)il{;VvYOkYm8w|h{v|dm3KA3>!8>rj^W6*dbmAhaB
z>eo`a1BRgXCMvhV08Bya%~bD$31|jX?tw9AyoJhLFaq_rQn>?$pmrUV+h72up!GJY
z_rU}--%jNo7=y;YQMn67p#Bajcfb(T_E5PE24D(W|4#Kjn1JSgP`L-jpz%&BcfknM
z-$msP7=qgMRBnR-n1a^3son<@(ELv-_rMr5LMnH`2-M$0<qjBv+Iy+o1_Lk!t@lyA
z4<?}bek%9C7&JaW<t`Y3`Uk1p0Ygyx5S80t0H&a|m+E~m0nHCnxd+Ce@ewL_!3fkp
zO63k1g4)NZ+y(<M1+5#X-Uk!V{5X|+U<?|cpmG<CKs};z2Mj^&lT>bl0hof;r>Ncs
z6VUuLm3v?e8lRzZ7mPstvsCVYA*g+h%55+JQ_%V^s`tSJH1|=t2gacBc`A3o2-Lqo
z<qjBv+Kp6hg8`U=))%SX2NTe|iOM}N28}OKdGHZ>Umyjok5cS|321(d$~`a!jT@-k
z`ZaM8jKP(^p>hY@1E%2MZ}Ga|K2ZN1l@EgfcmOniPxT961g`i4mD}KMFaZY+QoRfA
z1+_m?`4H%X`$6MRR6hZR;6c#(Gu1DGF}U(CRPKO#z!V(3jp{vcAE^J8%7?)KJOG-1
zqxuCf0#|7CJHs}(8%)4~!>HZ`_k!BtR6Ydy;C|3Jg6b#05IhK4ccc16Fa}p1N#zc>
z2TZ}iyHmXf?gRCEQ28(zfCoTx1=TNr5xC->RBnU2!2}$*7uCDqUQoL?l@EbFxF0nB
zf$Ar~5IhK4M^XJE7=tVCL*)**2TZ}i`%=9J?gMq5%7?)KJOG;aqxuCf0$1Fh%588r
zn1BNhpn4bF3u;GG`4H%X`$6M@R6hZR;6c!O5Y;b&F}U)<RPKO#MEr0n-vg%Lph4vx
zxDV8irSf4g01trXaa6wmM&OF$soVy4g9$ir0@b_VUQl}kl@EbFxF0l5r1}Xk1P_AN
zBdLB7jKP%yRPKO#z!V%jiRwLYAE=*9<-=eA9stc#sD1&Az!i_8avR(YCg8v-s&~P?
zpmr*i4}m_oA2d#*`Ux-u4}z9S^^0H(u3Sy!4!8$Q!NE0D?}7V3eJzy_g8_H|G}lr6
z0vLfS)>F9+?gkTZU<1{=;9gMMNaaJI5AFwzL8_ksL+~JIZKC=`Fa}pXn#vt;514|3
zkD+=G+z0BLseBj=zyqMUh3Xf;2wZVGmD}KM5x<9yAA3OmW7KXxXxu>Y1Q>z`LF*e-
zzZ*=!q5V`IfCoYQf2e#9sC|>-VK4+&e2dB*a4)ETo60A^2wZs!mAl|R(D)9OFMu&P
z@LekR!2O_^P<i}hqVW`>3r3*+R4RAC5Y*11avKc56tvE!dLK+c^J!G>fiY;<RPKTi
zsGmdS4j6*kxm0e00hof(Cun;cH&MIz3q<QiVhFlp6z>JKaf*jPAKVWbRjQu=L+~JI
zZKe7}Fa}p{qjCq_1E%2M1l4=sK2Wbw`7juO2SD>8s$T#jaK&~ix53?D0uD@4y$kLI
zwJ9ne0)22lXiQW61Q>z`!Dur*7qDUro*w{rg9$irI@P=2UQj!O%7;K7+z%QS)lYyS
zco4K6OZAIj46b|}l{?@bFa-zCq<RnB2kMWf@?kIl4}j(qsD1&Az!guVavR(YCg8x6
zsNMzlg4z(34}m_oA2j}%>L<VuJP2A(rus!N23J0X${lbIn1X{(rFswC2kK{0`7juO
z2SD>|s$T#jaK+Q8+y-}p2{>R=y$kLIwR5O^2=u}Ipm8qMPk<qK5VX#t`b97XS3aG}
z9dHkrf`jK%y$9|C^=DA|Fc^RbK=YYYzW_$yieW0Z!QEg24m^wMU2rd`J)6phKp)%>
z8W&Lg1Q>z`LF+kGzX-<Q$_uI70r!9@IQU$u_rQIiK0@WgU;rKfO^50izzDQ<Qn?Q%
zpt%d{0b|g(l*(N&0`<$N+yO&SyPV2xFaT4~x`OI`FagagsoVo&(D10-1tU<uipm`@
z1hwZ;xeW$j3R=&ndLK+c^95AyfiY;jkjh;!0`(VBxdVov_AgX!g8`U=)*{vWU;>&i
zrg9IALE|M<?t&4hzm&=yFa))iQMnBUU<z7SQ@sx+p!sqt_rMr5UP0w97=gM^<qjBv
z+AFEt1_Lk!tyfXK4<?{_4V8Of3>vSdau<w1{a>lv0YgxG4VBwq0H&bzTB`TK1T=S3
zxd+Ce@j5DZ!3fk}Pvs66g4!FX+y(<M1+6zyy$>d!c`cQDU<?{>qH-6EK>f{B?tmev
z1ypW>0hof;Td3X#6VQAsm3v?e8rM;|3r3*+HY#_(5Y*mI<u(|ADQNv0)%#!qn(v@;
z4~#)$50$%M1nU1z<qjBv+J8{F4F+HeTJNNKA51{=T~zLYF=$**<t`Y3`n##z0Ygyx
zPb#;;08BwEq<SArK=VCR?tw9AyqC&dFaq`WQMm(#p!R+$x4{5RLF)rl?}G_wevryN
zFb0heQMn67puU&N9WVs74^z1f24D(WAEA05OhEIaRPKQ>Xnc&yT`&Um8>rj?Ls0uT
zmD^wdrl9o+st<S5b%<kwqjdh9|JJH9Djg^tC><yrC><yrC><yrC><yrC><yrC><yr
zC><yrC><yrC><yrC><yrC><yrC><yrC><yrC><yrC><yrC><yrC><yrC><yrC><yr
zC><yrC><yrC><yrC><yrC><yrC>{9wci?`9QI56g`Wj=VHvadY@$F}r^JDAeOr^f=
z_B;HY*S7jG_;;T1?WdRXV|sq7I$ImN{T_ecwK+d#tJ~b!nX&5l{A~5_JME6y75$iT
ztFs*&_d!KFKPvT^zV)Q&$Jk_LZmti;L8lape$;om{W^7I78U&%o0;BP+t$gwzyI>0
zALF&LdTnO9GQ0EdKlgSS7X6s6OjYOH%2>bdG(|sVXD0jAifp(P{TQ2>oUHIYrrlub
zgR&++))}KSGn0Lg^X;XW^J8qH(zhpKFe&E;y9v_YSGSs0^kdEK4W>S*%lU!p+dfFU
zV~ROH%KL@+{iVEL*v60ko>6oA%hb$twNI~G#DtRbW3*Nm<^9T>ALaez{QObgPj2y}
zyq`=n1v^z=XRz_X{V4AzYnt$*yq}yOALaezydUNL<h&o{{p7qK<^ANmALaezydUNL
zWby;2rbfTtG8ES@O@8$C15-sm`u!GJ(T{$=q!WJBs|$6beE+3!eVh5gPd;dR?c0BI
z<6~i}-`}pxkB?Ee@AsJ8_}EdI?RP_Rez1c}dH;i^PIlhW_vfbE2{At4M_-?t7X84v
z7!8;{{kM61^!vGKoAJ@_Z#U%T4>msf{q2ULAN_u~E&9>#Z#NYE==Zl9ihlI_+YLoO
z`u**Oq96VKc0<vRet)~6;m1XDGt&nDXS7dG)RDG0UuU<*41UkBzuj26zU6+Dx5x5+
zl()y0{BXC8OwWv0N9wcHYCrEdV*YSv*gLoE13?RYK7j~7Cg-Z#jLek&HbnU0R_YU_
zA5HTI`7yb3Ti<?^FxxT2_+T?fIXsvn&3^DXq`zm6ydUmN-|sHD@iA7L9qaE;XD^Kp
zjE%mZL&W^i@Pi#^`gsm1`qA%as6{{eeGU<RjMc_x`}sOf%n><1YSYzzzaf<Kqr4&1
zIRD7@m-2>C!w>qXdijWu*k6|X=<jd5X!WDNADJrnkqwi6{zij>AASFc^_(Ar<$T)e
zM>(JJb5)u+`1mO2(<VRYuq#gH`;dkGrJPUkee`sl-}kr28t3cqqu*}_75(V@TkK-~
zn5v9b#>Zy|4gArEQ)!lR!w>#dTSnhcr;Ysu!$J&?zP*GI<AdKSEbpH#y&skN!EciF
z>%@V)bK|4Df7+Novhh*gKW+FyL!<lzzt~^kN8f*fzu-sTKcZalqwl}bpx{T}A7Ki9
z^!*V=_yHsO{s2?-qu&oOML+ue08{j%-w!ZFKl=RuQ}m<X4=_bP`u!_&gdgLTI{REy
zjs2Kby)xzY?@f^Qe$;pMt0Te>J~XoN(XU1n?vG`Duy?bovpXvN{3UfQ<D-9fm28AG
z`;kqh2A^5`cc*FfgYP+|AB{t8pC5GFkB@B6>C2g<#gF=~zMeta`O(iaNGCt~^|bwV
z{piCPq@5rAe45(M4?chN;n2vPpNcWj@7Hu<d^G$JfAr}os_=vMk$xW?OV0;2`O&Xa
zM{fRLKV4aV*n>GD_80ik-yd!$_)-46X?}dL_g?zm60yIK7e?s^KJNiP`hJLr@qs7T
z`~Dcy#*e=KKw%p{`udn#^rPQDP}tUwzWubv()ov&PpA0jg8KB@mZx1DAN~FbDd9)+
zZ@={OLAR41{d$mT>jwr)KTacU{pjN#Ic@TT|29xRPaWCr(8iB`ojTh3(Z{<=TR-~!
zH?<1;OQk;3-*ZR@Kl*kKY3D~j&mrynz#P(#r(=!lIp#t3TYUX~0@5@-8h-Tc<iYlc
z7C-uV^2qzaf2*<IcbDezA^hm)-KE8kKHps$e(>Lq8n4aOYtv)wXOni;tChYy+_v|l
zk3WPfjxTI{j91<2^mui8tX7@t+ac1)kA58@!Vi9aIy1eswyn>PxNZID<0EdXAN_sA
zZSrI3x1#%apXumFpYAj5{pjC)M);BaG&1`=&zXMyFmluNE&0J7=I`TYKll|&eta-L
z`ugdyHh%Q!JSwhU^W%f>HKL|JnfIgLpQbDBFZ3%8{W&ry>-_lW_ov5}#s|B;<zG!V
z=T^r0`lhlNA4Na<^)S=ckN!PfZ=8>@@i99y+1I;DHla3+58+3@?ka8l=;K|b=*QU1
z<Yc8@o0;z8+d<9qN6wFa-D#TUk0n3)c&8D5u=k^;#&(QMvbQYz_Pr=Rfwc3ZU*C&r
z_G6(oK4`FivPrg&|9R!x`#AYB_0oaTfzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}
zfzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}
zfzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}
zfzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}
zfzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}
zfzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}
zfzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fzpA}fjh;4E1&SxCmw#-VU5>1LOWgK
zk$pTBS*Y>fi$54FZBV-h`}Y{_p;_A_m#SL+dcnJ>NKn&^tbniM?(F~F*JmN4k^LdK
z{|G7%YDCfRa2E1@d(dwW`U&d10b2xDP^jIFk^jFBsTTQov=iiQ`C|X5KF<FRUy+3^
zd>hs$DEghl`f;7Ii;jlpVtt)_nZWu4kxrJ|(d;$2ZBIJ;eC@m&j=26h<LBRa*7<9n
zJb%i3{ENq2@;k1LuZQ>HZTQ05<0Z3rJNtJn`=9%EPxk-)pZLQE25-MgyXT6-2d}yN
zK31|zyYQ;u$qS2vC;Ogu!aa^X^5~O}d(Ho2?@IuqD6+q6CPNsKFb)tfAZin?fPe!6
z21T5NB!tre5n(+t2}wxg$Pfa8q7D#aK#+k&#A_W8SiFO)2<tT<A}DI`055cK6;V;+
zfyau@_p0mnlF2abjl28(r{Lw+@71eUuU=PIS9e$U3?v6m(u3}_&fDqmdNZ9>Vw>;=
zq^-V0GW8?sPjn_xI?(_kna)xm&L%vEXb^ZOFcTOJ29q!w899Wx*d9tajA%H~2*^ee
zo=4;YFJUy%7^3qfLuo9Lij2;RaRe8tO)-J+FW9~aDB%)pUrLyVZ8tCy@*@S{3xE>|
z3yF%rODHCq1ipl@{*a*U%OIEgEiz3Z!BnDY$S4C&C!9f4K{S(S7SZKIlQvbRoqbZv
z;QqxePJO@HekAn##B+vk8MUeR@OS*pKAU;Uy`TO2m1RfQYajHlxZ{TNS{%6V??p}1
ztPhWDRkX{}?C$bC!OquxFnQbH1E2lv)=OVW=n?p!i~Xt_KYe7@!-4Auo@5f2hB|(8
z+r@|fHuR>qFFohc?T0TgZ63ccZrQ^>KH2-8M+Wb#*#G*@0oVVqp?S`P32$#Joi^$9
z_1!0&(Ba|9!TjRaN10w8bl~4#&3W~bHlH>>=lfTaP8fFn4|kQ9J^y0a%JCVWmVcl4
z>7xZFz0xJ;q5&J9JG1-Yu6I0BP}{FX=PrK_Jv8#porAp3#lNxa#9niq&pfbnT~|kk
zOY_$(k58Ss;?(J9?0zBjqTc76{+BJ?TWq;6_(9ymBRkimeZHm3y+?YlE#6LHiJIon
zfzu*dPE$H&qR8){gDjf-8Uwl2z|LwKRHE6D$F&xv-Z}$2b{YiQMA3hn+7-?Imj?Fx
zl72Khl?LrH)5#*5{hs7cH2F1DZ!~$8fqb8V|F6@47R}CK13OO|`2W5^JjWT-+tr}n
zeFpJdX|R7w4dO#$p%F!<bOZl87|1_1@P8v2juy8(13Tv$$n6GklVfc(`*#_{L5_3L
z^b-u?bDDuaUmNU~=MD6)Fz~+z`J)j<rUM4{zcXl8vVlB@45vmKCw?BC%%jCYo{OT%
zAJ^k!6Ap(mCeLd!ZPD9h6Rs;1|B3=}p?;j&ga;?Ohao>-_tPfqI6<-gX^?C3<V9(r
z0&xTCb)BXJ?Wmv0)QjttCm%}ePX_As;Y7>!cf!68Cp@<EmELd3PkExJbQA1vguWa4
zazDw`6ZRdj&-GrS?=Ol&rP5>hqp)w!Rr2w$??Jn2x-0qZ3dDQtf4&lIfIJuW0}GUZ
z<8v<jtU=tk-o5(%wbf5PIR1T6ujr=iaD1MGeQ}+VABCTl@ZUc~$p<SCt6<+D4?s%n
zPXKZaj<EfkNT;Qs{?kcU7Z1>JBGZpt@AZn8=QEj>VgI`4D4y-Nf&czACFk}AISz8d
zLMajcFF-tl7c2Q)R9>csAa}WSw!Z%)!GAX<d0g*v(650$w`(NFVT`i#ngVeG+7-M?
z3BE+VM`6cVpycbIe<Io&n5hI@@4c`eK%BX~FCz{<9FOeJI@qa0+*rSk;ws~B#|~us
z*Kpj5RlTbbx4(0oPjBo`2l!Km_^|!{kh|oBn9}PC#Mxv|uQyu>-iQ2B`00@oM@o+>
z5F=4<u$vOpLLLwQJ$;nm6UY}L&c16K>ra9|_VbmT>unEzM3$1rW53+qMEM!O1f1h@
z2lVTPDE&4H#GR<uU8Mwk{AQzFHM5lbN$7u!c6oX$!MBi?!%j`567YUJ4F3ZcD)~C-
zcW)-=jh~0W*27fz??69i{cqq;E&3s`dTI-Ozq}}+#O-x(zm*dwN_?EY2s@4|m7K?u
z1?H;~Y}R3k$s~TLxe)&Cp^)AuvTSzqXqz>c<hx0Jh~#?PGHv#y=1g144RP5vZHXn*
zmbSP_md#NWpJ7X1&~%v1X+AH(mOj|#$grhl+BDLn5}7vZ@I?LEEQN-&xle~edi_6b
zYrM^6o*32E&_w;3u8?fikuAN?H`t<D;a5hzUs*P<DefUNStD75XIY8V#uV9pvgg_r
z3hC#H44Zv{`7#@YQ@<XorMyDQ)6Y8%b!A1Z%VQ2ittlrl?j-n-)i^pC(uWeF^b>?=
zzdIDt$G@Me$8B6s;!o5=)#z<nOLYZa2!(#2zAE&(a^!w(h+lSn{9HDVd3n?}WhTbi
zER>UTOkDLg$+#xjg&6;GD5Uq%p)#%weJ+jic2iz>pBqSJoFq?vB^1(+zhO4}4d!fH
z$`aY<7F#HGRkF`5h&NxFU`xvg>t#b^99=|SD!+ot>*Lb!pX{@;pGEJpS&0|etWWLO
z{rn3ry2<7%WK+Bv3Y{*;oQ8eV5U+;gV2I6au8JD3tVHwW(V|5rWvtTaqH;IY5iM3@
zDbKmLajfL=IfdlaB>y9h&&_13{k}iu__WYO)JL}Dxh^bT(Z-F@wz{~u4km+$QE?gi
zv1CtsJrv?GQ10gvlC>w9etvG)C(-K&SmGu}uVa|(H!{h1K1Mcoz7Y!PW9LuX9By;R
z-4WenR-&71Sz8IwWB>8(GgGLJZ>f%GeWsf7ro4S@pYfCYA(H<QedaLP%6%sks-^Zu
z-Oml<usqhTp}kB;O2jBkN3M*0I-Q7ilP”+>_^<&alvNe)y9eb`JTXGDqBwJcA
z6w>FyG@e(P2RDz;T^e_TWl598@naS=jmwNnNNQ@$rJbKf`VWzQ-M>O1eGaSZXV_8}
z#O2wvxC_j-6q01<=L1sJ=SQ>U1|@mi%G6Ck{XIxb5oy+@(#Zs8VOzqvVcT73yKfjT
zH{}#3JwhhO#_*I8#$YO@-}5^^Onwq=j|t=DvD6iODd7Mj4ZH-}osH5I;`lyeKje&U
zq@m6y5hBrv1gAy_F})E^l4EMBYf$L*)Sv!N#5Mz+rW&O-#GOs#CK^tZPh=(1&)bCZ
zl59K?r*n9NFq5b}jE~$LLV}(|t-~_T2-}_%woOB2eagYsOu~*tXV&Mad#&=GHci=n
zHpwxqeT3F%;^iq?CJ&LE^7SU_LqyYRJ<0wXp5*1NBV0h+CxXY6v_AJd+KLB9L!X}H
zy=<A16o@jQexAnmC4@9JtWTY>HH`40dTGPP2-=+7!0Pwedl5g4C>8Pq!l6W`fzJc}
z6v*=dP4kaQbXfo5G#HxlRP|d@hloNI#Geu&3ui}$GZEWG5xiul6Sa#XXHhY4{0z4J
znfzapJ&KgFe~BXLG$SgBqB|>!EGBA*fY$ff&v#rSMDPrS+%Ii1pp^KWk^-KOQ@5sP
z*B_VOC5q?$&d*onIYA~lZpy@c%YUDy^E?J|K94`?k0^hR{20=Q1ba)x^VsIr6wl*{
z)>`rMxRz-m{N-bl&z<$hZ{Q=n-8`<<A%o8k@>rLN$1+b3WuN7)(-hBsYX}^wtv^>E
z4JrROqv1b*=X&|rmFG&Cenfs9^h4lzztn->-c8m29r*foDKmfAU#;z<U&#K!$x9B&
z<p2AAT|Qf}A^%dvvmT?C?bY@8xw$s9^Wgm#&~?)|@ujr8(xKY19=Hj(1GpD>5coAP
zzPBpZ2G|AIAD9Ci2P_880#*T60@njK0e1lR0uKVe2FCY6{lG53{=gjIIAAew7O)Dq
z61X0?3Ah8e7kCi(H838>Ya3t}V9C~x^nT6pk6|nVu?WN>5Q{)80<j3hA`pu}ECR6z
z#3B%jKr8}(W(17Z|NETE3xF73!FM|78RV~6|L=CIa*X_>K}zL4qLQg9zhr6$y+X#<
za6_rChc2c+n!X|L&&YDKQEoQMG4i^9M!pXu?_WuZT=&m>7UjvrD0OAx_5W_UK$TJ?
z7Q#<PF86!a{~N_+6>d;5Dz8-A<UKNZze%p`f8@u(o8G%FxZioh39H1S`&aimykm;=
zP2SIvWsKJUe@l7t-nG<|_v}ojez#~{owUz|IjMumDj#62P8d(mxNJQ{6K}k`rj_Pg
z(ZcS0uY5(;p(<zZ4fCBfSD4PRzD_pUcrAT1=XJLaSTNjc*HV|oi7fB7?uG7btFTy_
zSzK9}UVETr2aDS&+G$jo)nfMgyyCr%Vo@h^RW(IBK+;A@`X-;TaFX|tXA{jKx`Ks-
zR}#$)<9YKcZ1Al!FY)t;<`Y#C{am`16sp3MuEX|1;3C57v5i}=;DorLeoJjEC0GX8
z3c?$)y%Km6P{Jx~-%NNbwr>O84!i^C2d)N6xHD{<H`nq8;X0zf6Kx>6izqzZ9md>)
z?R$au5#Aq`{{Xf(0v{y&2hqdejbIZp9szD9++v{oPXjrNAES-OiJoA=PhbFfTM4%j
zZ6|sfGKL+r@eI*UqGwr1_#Dx0@Xr&zK=dN`TEaa<FN0@zg*IL#+RFmMeb|02OvamZ
z7#)+|Ajz9VZxQWB?%Tk32!q&`Z~)uy0{;b!go7mbfap+I#)q{1Q5gR*wht425+;|M
zpAvl*A*&1HKd0?4!uT&~`zxY{Yd4>F$=~L@z4^ddSM5po;o24(b64IvtJ|nIulY20
z$V2CLKjG%luCZ+=7c^<}<ZU1PbKlAqZU5GE!2P$MRj{~t;G5m|Y&pBjck?^X%362d
z*3U<!owcc~H1PbSR#VmwyYId?JAd`&xGn2$S?Ip*`VlElb-2H9<r%wLocHvF_og||
zTAaQu%iX)}2W=<rdik#Fy(#6b3nvCbcjqlVZS=mmYcDzb$4}1blJa<q@7<SA*tej%
zN8g^yw$z?GxmE6$>;As(mGwQVW-WbW#L|uv`}xo8vMBit|Ih~?Z9OCN%EHj_OTNp9
z>v{J4SKDTO{bkS0bIL<qmv5c7_4*@wQciDY8L}rMyJ=k6nXV?~yY9Mh_U1Ews9t#4
zk6S)|@B7s+4|(k1^UdF?JM+<frAtqlS-m)WUek5`rX*jme$c-2F4%tMddHp_PkuJA
z?XI4#>%JfG>9Uu0m@oRrsJf=-^bXBE`Em2)-kS=}=y-0gw$Cohzk2WotKVsT`Hykm
zOvzd&-*XxMc>nU3bDMqtNzeD)kE|FxV(*E&7v%M;y|S4|uDsypZeNdluBOWamo4#p
zxPRIi|5$5z`mQw@OM0&OX2~6QZ+Y$Sp(E|)&2c|;(ZjE<8~N{x6ZapQJp8U(Ul{aM
zXw$yJ;>(UacFw+a*{#Nvp0@UmtVh>9u>JivX5Mtdp3a}Qn6|#@KXSgmuX*dNbxX(X
zTK{p<tLr9bzjM}!w7=s1{l%Jl`mP^VcEKmPr+A)jGvuQM<zZc;(dNx`(rVdAqL_OO
z<ipL;_0LU=E>Aa*w=l5ZgdQYC^T+6Yh*n9__2v0hi_$Ln9zcsI@+S=Z`6ubOiK3rm
zp#KrYCtAHjnnw5Y9D{g{G|;a!i09V{(d~!G|7h`izDacX?bhh>2MqjKY+(OXnkYr9
zSI(=V$)7Zkry9tQ8u)p@z|Xr4?6_&)M)PNff⋘`!fyt?YpMv_7e^IkISH5zkz<p
zK>mb5e|W)Qzkg}4ze)|_e1^e(mvh8u@f>O3&#TnlXvfPT1N~M8{$F5F?`DH|{@dU<
zYHqN9HyYSkX&}GLK>m_JyQrH)jYE!s{v-qY7Z~ivYJ>f_#9%-A4Eoi325}x`Q14NL
zc&;^Q*K-E`Y&WR)MuT?c8Pxl@LH{us_}_!hFB;VynKl{3=e{WN@hB|knlgUT5??p<
zjh1O3yVF1(6UQPDi$E*_u?WN>5Q{)80<j3hA`pu}ECT;ON5D7zV}1Rv3n;IrWjdR#
z-(<2W5ZT}XA~-;0G)KNQMU`v|o}ZVV#QDH>Ko6dewvWiC%RoJ)Uwc^htA0IyALS=q
zhU#h6d5WjYUOiQ(E1oWo^|T0%((MX8r2y%2RV9CtzRsL(2k6O7_n~A;RzTm8Q;FB}
zd)>-D%WE%HJY7cXsRn_AnPb-T+j}WJUeC{d7_H~`K%bV&m&ua6UwA!#>cy)5<Gr3=
zc}>r5u1iwu@Ok|{*7MiBvP_R-<a&PZt4dybNUifv9(j|}dq>IHuKO*;^SXbnTGhkP
z%{^I-*Z;S<OtrfUus<*dI1X40oCT}`t^}?JZUXKA?gbtMehrMDtm<t8>;mi$%mI!A
z76WGitAHzk>w%kqJAiwE2Z3J$<ENm0U>9J2U=DB`uoyTCSOr`OTo2p?+yUGRJP7<6
z7(W&D1G@nG19O1mfW^RBz$)NM;CkRD;11wk;6dQm!1!sXAJ_%hAD9Ci2P_880#*T6
z0@njK0e1lR0uKVe2F916eqa}1e_#%99IzNT3s?nQ30x1{1l$4K3p@z?8W`_E{lG53
z{=ghy`kiWA<Kruau?WN>5Q{)80<j3hA`pu}ECR6z#3B%jz<+xLjMo2qH08My4OsOc
zHVA&r`v0}}sB(<FT0!1tk@u9masjc9S|jA|ms<J8k5KvYK9VfwLpk4lO3vuOjV4Ae
zF7J0q;`RQKm_>OqF-l#Tc>Vv{6^f6<_Nb4M%l+Q<|3-1~qCQ5xUnT36_tfP5F1fb<
zi{wv-+<WlTD+*nGzMqqGpy{0NSL~KLa^0aUW3>K%66N(Fl5tb_*ev5yHMwFcb5fiz
zS(>%)o9;@9zw>q9qD~sU4)CsX=Zb9m!nG7#yOmxO=%B*v6^UZbzeLu2DyZAzZz#$2
zBmIc_%Z#veX4tmeOpoAk8?=%1EaJ{?BoSgD@#jExE}@es1H6PxqAa3pq8y?jM7cym
zi8u|z#&E(BM5Bn#gDetUVS>@LJ%;FfBDvm+(}lG07ov+;0K7Pizm&G~h}>Z^-pr?s
ziA04&MJxms6Uu#7LR3n08Bute9L_{+_kP~fzGGsGj=hIDOgE++Zae+Ks=5PDwK{U*
z$tN7{m)`lu3;XVCz98kn*Up`JYsYp=9{J&&N53EcWlFokYv<jx`u6lsN~dj!3w{&&
z@Q3|p#~t3Y_>J-TU+r2w;jyLdzO1TgdQQ)4i%uG__5Q!^+_~U|`xm`^YQ`s}A6C!)
z=$ZlkBlAM1d~#;jdrtiB#T!rFIWqB^o+nHkeeV}7et2g6@^(*L;c9!zrmp=?{NQu{
zt;rvM_FSvcbKekU?d~bu+WS`1<+ts6<jWJ@|K}YqtnWT+!uijn+<VD`kADC9qt{+0
z*E#oGd+qn{zkf%ojoz`fr)K7LG<E&z>Mbi@-W<{FM(ldxWbckl^1HIp<n0aQM(cU6
zrok$j9XHvDwr=-9azu-ChIrXmqS?8h#)D{fTGBueO@F$9{Bjx`qUpN~+PllZACG~Z
zmgHwN`%f6i=Nr^J%0R!^AkGgP=yzxu-JdI|T^dnjT5I4BiKFU&XrS*S{b+H>GVrsR
zf!t_4sXX0Av+p!$??40lWd`~;8SLM=2J04$)*+r}VBcsR`h^DZbQt(^y@7rYgZ&jW
z=s*89Xjcx|NhOL*pBo%s7aPRcX5eR$Uaw8~^ruqvjvDF7^nmWCP4M@i<vL!OI_maq
z!tYT0Q1U~j8}#_lckSf~hEfm6TfmNIwvwmA&b_F&wp_{gp<cBv-X?<bq)W*UI}7#p
z+5~@JdkXBF4Lgo$N*^E&z#qpXC3h<j7sHO0tpsv?yG+BN?;fxC)e1y=$b<4^OKBzi
z?2LBRT%qLOC=lt;=O6Q_hFq<qw+ZhcC6MQOndCR6<#*5Bm<)Um``gg2nhK>S&$%+G
z-#ek-AwYciJiH%vJTM#}UZz(J?3@SxgK{9El%ha9N%c#AwC+lf3V*JJz8ed4x!(S$
zw<b&Jv%DGVt&<Z5O5Y*QiTd$K?X6Vu%V3|sx9-hQ@{aKT2<mk#Q1a8De-_6d{>$S{
zrj@8yo342IeR`R8K;P@tSs~A7GIfVPwbv?M##g3W4B9K-J(Ka_-!0?g<vGM5Fj6mG
z|88S3^y@k(`6mj*#poyY2}*Dl`7P6h9G``XpF_M%TVUTUCyJEh_%72yZtqOR=OAt_
z$o)7VxZX9W*PE~O2SPtc?UwDTMgR1wY|#OB>M+6M{$EUSmHNIeN|EJqJdpCp{$NGD
zjxkD~_sbi^O8sD}lJoKV74&_Rl>B_`NB%tn{+*8tu)pMcInlm%Uk5*HdMZ1gKwtJp
zS+7e@-YIp0+yi;!@pTXE1iC4Gw*t{0@?a+=;C8(L|Mef$qAWg+L|)#csb$mhW>n;t
zSLEf1yrE-9<`ou|7fmXiQBhPrcI4nGWz&kr<`+yUQhCvH@+QvC&nqsSmOrKRiXt&J
zf6A1yiDY_e(Nxbw&m56AZN`keX{9srW)xL;%F8Osil$Xiwx^<`yePjgZ|cm7qS<*=
zWU`p!$(vMEK^_$6&zwS7HZ{L=nkX(WDw0`b_n2Zcr|AXrWD_P%&YM^=Ij=asl=QgH
zqG^Q<b+VAY+3&IiOmY39f5ck<hP#ZQJWWh#M*W+92OwSU^Jw2@4*&kS%e<Dp&3x=v
z;d2tNjQW-D;fYyM^~nQ`syjq=kEL(@>-$UpHh&iV)W8o3`Zt-g6ZLU2DS@t^c7#H1
z(ii%7=FQ{e2b-P5rcvHn%G3KR`F?}^4Fh~np1wql@5v9N&$d@5=-;Le|EfHFAX*;Y
zQ^-c(nNUa{Go!V?;XBt^HoqzE+2}E)UqVpVm5-4<&$FSBevC%5*AUN<WKXu%=;sUc
zZ$~GQ;lOjD(9L9D6w2>i>))>)Wz*>6`Scz8T${EauF5<#fxl&qxXB+tNTc5z(2sEy
zw?Y{=jm)?yZ!YEO<5Ydz-%xR<ed8n@`OaVN^P!MF=Kf@Cshs@ocX({cMlRXdNj6r{
zZxQInX+tcceOo`v=7}>!4{cUrA=ye!qVJ)<7z*igm!Gpmam})AFr9C!F=g8}69r5c
z$Y4M96Ty~s5Y44}JbOZ+QrZ^s_cADI@;Yq=$#;@mAM<~@{vooTTy7!<hLQt?gc-8(
zAyF&Op%}@2_!ZgTOuucQ&)I(3zKm7N=&>41*3@^;vzpU;|MdF-`dq2uyXW!%yQjXd
z(0-D05fACkrEmK8kcIXBpnfAm#$z$%$?yI1oJFd*iPn;Q70Ji`{J7z=x#AL{`<sz?
zUbOLq?8~~XHj4Xeq0nWtt?!E>wJ$W1A11jzPmUI=hGTY^+!vGNz8FLMVhrz#(NXsW
z9Xk5=wY_A2;G3b4K5x-uN8gwq$={f#Pt4zGlAoBDhnSwy>972qk<IkIetphFJ}r@B
z(c(DzEs-h<9UTkkw=G<<|H<Qoj$qQD-%p`mD9EC+byW6V`fk5IKck#u#xZKog7_?I
zKwJ{VG4g1WZtKUJ{GE|k-U)^Dc~wJPqWx~d82arGU(_H_t7Jc@BwH!((r+-(w(Jv)
zzq3DbK~wW083Vm;ss9k^A0mBy4#oPeu$T?jV@5r$A?=~fDPZ-!*N~-eai1Fsd0#@l
z4;Zd`3Ne@(>G%G^^V)%t;}Iu$&#o1bjQwO563RX&(-}nJ*V`xQ(uUs?J(2j4kVS)R
z2<3EYY?uN~a~u86r_7=$pBOARq9mHu=-;j9Ed4kj=1d|d5e?^!lDuC<r=R-!i6<j>
z7?2Lx`rkT*NSsC+=P?t3so<%5)_?QAgtq17kes@<CX&gm|Be19+C5&W$@fXMuwps1
zZ6*ruBQm=i5p}O)QlBtMK5b_b$@{G`;gUJ(d-qC1hz_)II#Ci)F_FC2OH-0#Qm-&c
z_pt5o&8}eNaM+e8BP_EsZI1(=7AB`@v|;K*IlaSjJJPlsXD%U{7$%pSnC3@F|2%el
zL{<HWm-n(Fp9e&!{bF`_nC69H+j4Uh_<>>aIBXY&$#}DXHwZ^y`)opaX%c?VwLS7K
z1^ynuWv6b!^?MA(%KlG8!1+1WkpktIAk*bI;F(_#!QTj8j<+)10bY(zGWjr|%W+($
zw{T-ZjyE!S@6&YT^A(>T_#9Y`eD2$vFV8bFZJ5<uUuRzb-oQ($9r7F{ll2|N%k!8_
zhh9;<Jm<)C2>$Z;&F61E9)-7OqxS~*_u%=wi{kaa1F-o+WuN6WhZN6#TM<Yev$;P0
z9e|zkL`{j`8<6J~ndE#yCV9S;iGK%xZ_M#~1MD}yH&Ay_)&FO`H}G4(|Ig3OUHu!c
z|KIV6@_#SzAn<Eo{HH3v4X_KaKQIS44p<DF1*`(D1g-~e0`36r1s()`4UGQ`^#i*A
z`vY@;<ABA$S->jbO5l3nCg2X>Uf@CC*TDEX)DP?e><`QVy635WEW2Ml(FVGs-lt-J
zViAZ%;Qw|6_`J$d@Y04HBl!GTgY!9~pHEOIz~?8%>m$m$G^NPM=RXv$zdq#i<%)>%
z9+YP+^>9grTrW#+p^b3=dH%u3=V8{5Y`-7n>zla@n!vSu!i87^GHjp?SwELwD;q?K
zk=JV}samwe+Y&bsg_jSkRJn}rMO<WZ+?1BOrj&Cbu7AUNmBV-x4d!-pdFBJ8F6VaK
zOrEdtx-iyPRvWJGS-zVxq>Egh=X2HfsS>PbRQ@%}kbRHK^E`pq-SPUm`uQFVaX(l}
zyo?{04?Ljh8i5CN5Kaa4f7u_#1C$e9Uestxw?=_mW`-p$$M_NDg_n14)O2n4MpcOC
zwTf$qAM<szA#Jh$UX-7U^76S^c+$cd+CD-Q;i9g=<s*;32y=~m4>!F6^;;s0N3h{V
zV%xC@#3B%jKr8~W2pkguqxJuOJk)S<3Cy1)`)k(!*Ilg)FbbjR0(riY^NS5jbe2%l
zHyG0~u}I+?L?|ccQ?lG<l-rDQjCM@87<oP*&##gq*Znh}MR_tYN?n<F{r}E##Ydw4
zOp~%0x!mtv|8EqR@n|0-pZ8_G^4^%7ugJCi&s{YAoloP(H#uYH$-#|7)-DU){=#cg
z$4(^67_I-8_y4*O$!g@iw9&$3wm0)Nw_93yO}YP^C-b@zr4dC-^7)H=?r|D2dxj~U
zPTLNm-eEF+E+e0#^@UtM6S(f8^(~K_bk1W(jtp;Q>a(-+*_T}U$fu_cI@r&;^<er_
zBRfC0<+aB?Kd)-joHyp><$ZM8BYR4&ZY9@%pSGe$=eG0LydPFYqs?oHL)UI3mvPY=
zNffi4f)!0)&cDK+0dUcm4D=r*1&x)$P_7Y(X1^`%fN1hl4eaL|*g3~QzleN@W?#<x
zqsg-j>^x+ke+S7mqR6z~zz&I{>Q@`+e{3K(8^ldMV~W<UPYvWF4D8P`(7(z+{*r;*
zYhb?z>1#xhsg1$@ea^s6wLu*IZXnO03R5F1CBD?azI@gdEuI${=r7XsX`L`0K>U;R
zWICk#L+kziqWEu#mq}h<>gyE;DE=zQm+JOuJ!A(Z-zNt=N-sm7*KPKO{ppbVoVuD=
zi2>RNxg%T2`^u9FrTbvNrnBPZ`wTLLpzo3oRVmGHqA-E=@gU_kB^IaY`-|2CPga7v
zp`Xq54pj0E@Ke19LF>22DY=LAWLiviqOG6of_h!Jkgb6Izreox996z6;&WWqH_Ek!
z`ntv$73K7w>WEl-hy{?exKUpSsV`sD*CS3Wo9fB0kjo$^l^d>sEHApebXuX&LdW7V
zPti2Wlq(qv^DFWT@@Ev$+R3uXGd++hCa-wrw22j^Wz$4adHJ+5wH^}n(~?M4fAPdA
zWix1Hr!<B<T7IeL<&_I++Z3v+m_(kk89L6$Q|m2@q<<4D=6H(o3gpU7WumxrO3{ow
zPkuSMgbL(}Pby0*LF*TShF3H#f2yiz;#6`!(v3-_g~+8vq~uMZ-p0~tv@(^Jn#z@;
z(;B?Z%d5|z_HhQeSuZAE8pN`$aDP3Q>pLXO1?l=RLb;ms7@-W16q2(MZSY*KT39+w
zuld;QnR*lTQsLgrpq)%hWE*>PtdP7pR!H9H!ehOe;Zfe?<xMFqm?)RimQBphn>e$g
zthl&QWu?<fsTY*Z^iUt0BYT^!KVxEfe#OL+yn>m<#YN?`hs!P(RGC_&OO}g-RO>aP
ztXF@z)!@7o^$|lx49y&z=NdC~T*laJaD956QNPu@Z~bP&R-aySTBOhK;%Fu-hb8$p
zPXE^gsPf~H!?{vNYD<)VB{iiP-cSFNwaK}n`nPR#s#VS}<#00xV~QNQ%#?raixZp4
zJelNpC6kF(E0xi<oL9-zs<gDZoKfBa*@|T=X~c~e@;<KAN!Ih5hzGz+o6=`#Pv*~T
zS4w=GzTRHi<Ld;T-@9>MsloLC%QeWgWtuEjf1jKAolBKm4*wW(rJAUPT%O}_3ee@R
zh3Vszq056|a&EKCI20xiKVQ~`$#H96xASe7JXM#dKPf&<-;i^1oHBG?3zJi*>Qh>n
z9H$07&k-i)XXP>@JxtEAlX7R6Jp7)0ZkU{(=}UcAn7sbdjpXCQ<VjKy$sH!A4pN^=
z!sNDkDXl*Zlgs<toMwl~Td<JO8zz_Q_&8OE$+_>!JYSf+H7gUY3X_M=`~6{ZEFsWK
zZ3vTx-|MXjlcQ_u`kTY#yaq#N1j6LJRzu2nhRHDnkYZ6ACg-&aQvO<){8TB3rC0=F
z5r{<~7J*m<ViAZ%AQpjG1Y!}0MIaV||H=rs?^5fCH6Y&ukoQ<*;%j8SH^bN1vvD(m
zudTOFSNeSI$@fwC+LEvR`5Kh(r|>l<=kq<0*-5<`zXz~f`N{Vj4&9`9zE|=Z^7)>@
z)9|0~74Wrk{k;rrRHOF*YH^c>ui5z?248z`xLwu9^3As?p8c*<&98rVi|ga}05k;d
zm%axu9{T(q0Ph!m58(b=RsH{|_W&Y)F96LKTDDh@D?c}PpWb-=ziZ*Odj2NA@_z?#
zFEDW4B0b;r#R47US1Y|Xz%D@hEk339ozfczc`<MnunM>mxE|=fLzVYET&dgH3;98y
z<44rL2K58G0Q&=Tfa8G0Kwru=x?PS-Bx)U0|BI9}FZVa^bCxwmdH?i(IFa-J)BgXj
zguZKYy&u}UDfISM=<21=od)cuj?@2@X2txBMIaV|SOkt&1RP{o&Ncbj-R!xlrg}sM
z&sTXK$b5j*<=mO)n>-)p^>?hVtTsG9WchB&kk6sGJkOVFmZ%b}XH;IUw~}*XF3<BZ
zUf*bk|2&^&d;E-XDV3GyXD-jrGk3}iO6dyf|8jZ81C$e9Uf?+qf5?pAn`HxBj`5?2
z^88#?Tc%0`A|?+S8v@GtxU|Lo2bXJlPH?#nBFGI_tWddN2#yd%RH0{bdA@Gp61?A;
zZ`5`c5*|hUvG**(f)d+|MIaV||F0r|_5VEl(l|UwA5UxWf}|6z^?lf|nn14qPY_P}
zW_qrE?vi6F>qVji=PyP+Z^#N|IoE!5=P43-+-BtSogA+tP6;GwNdC)|pNs-$3|UT|
zXSG=>Uz;^ZFU05^qih6LDCBa|eOb<Xk1FTAN0nm?O2d>GxqY&2l00aCBxcD%lo+M1
zO!B^!+&1!)<KUCZl(;^APhkq;8HsOAQ2LBq-|u}-!N^bk-BLz`NpG9HcPID1d<Ws7
z69fO8`}B+Pd-goBx2VS_TRIPP*Gk`HJ7pQ8_Y}S)1NuEgQqu2zI_z0~&1|ZUJLr2l
zd$HIq#*3zv!n4gQnlDPqu-HYlvyZjAvwG~no|>&sZkNiyVvALn&9>F|wu>*`DFS^&
zz~XAA&9hb}YFb(+VbT5~?Bl)9W@V;{c&|Nuu+W@dG1MhSFLLTDxhV84?9OVZ_hHdt
zXXe%g{x{c-H(M<ZS5v#i>~p3Ht)1Q6){@gjnEZF!lhapNU2(aYPbb8Q7M8RQX1j3D
zqrSY@Qth;uEb-oik_2yc*1VE??V`EXBiU*%NY2o*e4@6n<8r6juK9&GOT?M2w(;&Y
z2~Cpnt<&>OrX*`nOKBx8acXf|g0Q%o#t)v?-zl^V@h_ppU1axecO2O6b&I5VmPIDF
z)2~rTtp#ybF~5zux7{l4>XcSsw^y_)&?wponmxnq-jHfByHb-aTD3WIazgxD8`?Sx
zEM}7|DJ<r6*XpdSB=dLkoh6#Z>(u($*WR_$JEBFhmaK_|^O8i9x%T!++gj550`p8J
zOOn0G&XGaOV`*MtZ><S0MZ-G9<m^4qYMJG=c6X#|wYl*o=dNZg#4fS?&9nMz^E+h(
zt40mqu8pF;ViD_1HQQTwa;`{7^Y#2|r#fLyFeQ1<cUmmlyl#(Oc#};{Ypq_Z)o(4(
zGCK)*v@I$T9lR#*jcZzzG;{TJ`Yq1Z*E+{KGvll&IbJcgd2(*2YU*6cmGp}hX5mgw
z_S&;kL~bWj@;u_5W}oHK+Xr_uC0frH&eTp$-&Xtd)K#r~ev`1eg>_w^x$rd;=HwP>
z?Y+XA=CpcEWQFP#!YSSrX({Q{@dCGe=^80aPgD!%h4z4_<Gl7$TG!Y+y=+@+UDT$!
zN%!hSV^(SQp$SQy2Bc*AI+-+|(=tEV?zD>%yW6`+EY`#bnpzjy&$yUg!13l@=1&#f
z=A}3@E7O~2bzHp4Yw9b~yq4q>g)4!+ZEDxV*m$d5>)gyK5<GE3$rE99Hg{P9<}sF}
z3}Nz*w=Z;-Ikk)w@4&QRNAbSq++#91om!@Sh)5c-?1cEZgf0)69rL^^#VX4fGdW__
z(wx<)zN!fwMz^RIYb{;ob+IhatY3Ryo^M*@D0FUp#s9KKH#e=FO4n{}Wp`!G>th~c
zvid|b_inE>fo@nWu*dlV<_fDYrFbbge(&HocMEaXfEtU{?_53K>&{%!d7xu&?b?+1
z8&bAZA6R;uxNwOz!JO;6A;sCAj#$%exfY+d<wj4kQyk76)m&7|p~K?zwzO6+wOgip
zR%g2lp03GHZlal}24TveQv>yW5x-Kja@I^5t7SQbX;{CEX2P0L<8ua@tub4c<s@0G
zRjsvDw_TX+fmO-FGQ1U@o4cpv-$<QQTWrqCH@9?-(UyPfozt{~=G^LkQ1iCP6e|i{
zmbAdG?7?vszuA{*@y`=!m6r7FCW~7Hnm_&8@}QT-qFjsDPCt{9l9cRoi$0ariv+pT
z@;T?~X1>>(&NJt>XfeK-FYWz!k?FOFT({`xeAyf{iI$hrptrZX<z(wu)CV^TdnfCL
zznXH>ygs+pG*oznStN+gS(@3b{~(S%RlXe|2Aj5-O(L~L;NfTEyzRZN>~6)C?ye=&
z=?1l&=M~zT%;k29hqp2%BVBB^-kv}a+9~dKED}vL=SHhP$vw7j(4HY`MB7SJ!0Nqa
zSypbG!&}hN>$RG!W5flTNN6pnRVI6y2vlyR%j{;YTxL-{SkzVz%LsV-cC#NI9JgHb
zj_b7GGBL*PHsw0K8ubBZwPpSg>xC9!o!zWu6Jfo<G4|~A%oBGE`Z7`5KznMf@b&4_
z;)A5sHZim^$t1knlCqw$n6}QJFV75SYtreZ+t$A7?LM}vC1CwWwO0hHpVI0?Q!UA#
zJj^-x{gi?B0V#t6&bTy_=+=3l-#XOncGC-F*2z_q?bohNIdg;7$6C0i<6q5=l4_?|
zSSc(Sju!qbI<1(s7E4!GdhMCyhPT>MkQ`UEJutu76|0lI_pcmc5*E?Y=4~(5_7q)P
zHchCon8zk8?$9Dh9H1kjODC~4KYPI%`%@NEM-i7H%=7Q79AS588S%MhORr8&XRcW+
zw5JXe^VgJUJB6u*m}eO^e?qpCn&r2*7U#E2GFjt0`Gt7OB(lB267T$1T5l2GKGnI|
zS(%w@wtI8Di-(9&rzNCP*X>cRSxo+URy%nWx2(E{*_2fo*KExY_W@@Mi~U|pfzxV9
z=}!%@h-Bx01k2FBXep~rcI&VW!a1G#j6eQBXr8_GHO`if?m3#RckatxZM8ddTL-5&
zr&Fpw=oDwfx9H&13auuW)ovFpybEp-?X<B?#A;K^Z0}32R0>T?p$S*qFp*0A`uZIi
zrpjb`iSK69mDN+d^F=eO)@PNyjm>g${yP(&>fgz2$vndmx3yFCJh5D}P*+}cf={HF
z#$;NBE7|0=#($J7?CrhIKy#<Xo@Qz#0>T?IztVf8t(7n@HjBFDE!&Eu^z(%`_t0z0
zoSG%Qnc3+Wn_NfZk+s^C)YGi3ZZn_yyfwJO;dRrignd)Q22;OWiUA!VLdfTwaz4<4
zNDdZq4X<3UE1wU_XP$DlDQ9vT5gkOLGf`KfZbaRQdJu8yMH>z<y@CA*<vo>wL~<sW
zK_q)pHc<|dbYvKji%3o>#}ZvYbRm&U<B8;}PL~qp6BQGc5Xp2I(Nv-`qUl64h%P6(
zlISX;c|;3{s)!a6Ehdt;ndGgW<wP>AAi9a@7NXmT{zfFzYN9)d))K8Fx{K&uqK!lk
z68(edA)-f!wh%o=^f=LzL^5q9dWvWV(JrFhM9&k|61`0HD$zco*NNUFlIbm?w~5{%
zIzaRu(Fa5_9a11ZB>afzFwqx8UlJW5`j+TBq92HUB$6paWHQkaOeB+qunAF9BKaPc
zOo@c@H#O`;Es5F?wI!0j+tYzarjCT2h~zs|U5UC8$#3zVP9)zOJA<en(V0YN5e+07
zM3hOCMKp{^{w|Ju=l4RQ2}GPOq3r_jg@naK@*TR#L{o{%i7JR@5nWC+o9GInD~V+C
z63!#4B)X1hF_C=d?XN^P60IV-ndnxc+lc%`Ylzkn{heq7(LF@+J+>O62Z<gc+C;R4
z=y4+XTM{(<=xICQE~4j%b`#YS?IC)V=ry7@h~6Z6i)cSlkmy~a_lQ0q`jF^jqQgX=
z63O&A;a5c85Pe7VJ<(AjnP{#=lWv;-6Ez`<C!%3O$ltk1B(f37-wJ9))S5^p`7Z29
zL?;uSO4O046H#ZPE<`e=5uQeLI#C~@zC`_r&LkQ@bQY2Pt(!qaPNGbr!9>|axkNGz
zBOHls$)88tV~FH$8OZmo<-2tk5nV!bDN!C#0ntRFLZV4TB}9{nrV`2IAuK1loajm-
z`5vQ}XdY1|Q5Df5qU(uh{1eNGRuSDwbQ_VMXf4q?q76j%5j{Zk528&(TZkSblJ5_0
zBic^%4AJvMGQCK+2ivazUn6{jXg_$R8T%WHKr8~W2*e@~i$FsJ+^_yqUzZX{d{jqI
zr>oUEmHThhv8G005a`^fc>g~X`W{y3-UK;NJfe8dUWN9}3SGb&pteQHgZr20^*O-1
zfsTK|4$upGu6;`GcvPYNF@-*05cxIC1MQD1Jtxou^aE?2Q2D_)wLUJeUGdxwuAl8U
zL{e?JkE(TS{@drN_<gDP0MM1B>}p>rdF>H}z7X&`h3;<^x?oTITk$o`;TO;cWIaFl
zNWXq5AIw(u)J~iq)qgEZ<vWKf6r&XWod3ZwO7D34?|2!0{Y#-1R2Uh5$1Wx3_`8JS
zeTctnGx9en6rZbgh^~-AFR&J<p*+XG7W*OiEBD85_22jCFZ17ycKfy}eft)LHLVnS
zA^)xZ`z}^?{O2eX;}yDd6nc<<O#it=<#YTa`~R=)Kfl+1H~juz^Pl^Vr$WUu;AtFx
zK7ReP5$~rJY9-*&|Gnt{_REoPRcO?IoY4Qh{U_3YF<sTm{pWc3A3RR}`}2RB|2+Qu
z+3|NTfWPAv{@(sCVDHa5ertX3{{(fO6hI#TgDs<<e;vt69y$JaVei+Te}C)vBU-_~
zRfzxZ^q=Pif13Z8XgF?x{{|Be&&@xbZ&2*Gi+TJJm_N98D?hpa@cfPE4_=%<1CJ;@
z?ms+!)<(=9{;ct{-v1YV+JEOZm0$NvbpJK9kLN!|{{PYAC;R_%^EV&PU%z+$^GBV(
z`1p1ILF3P#e*E(IbNW(!{{MT&&*SZXaD}>lH5wQK@8WvyfI{~>3bpqX3gkN=XFD46
zz`)x|pY;MS-=xRK3ptmI#A&5UFH&yA(|cwic`JP!6awF?dIO)sUzUFy!GF`3H?l9@
zS9-&MZXj;oi=Edge*<4Cy}&05^#^cdR6g&LiS<X7DE)B-3M+u$U#j?a!xWAguFx|?
z;Yq~`rxqz3i+r~0c}2<F?^8G%ycbvv^Z{1^{lE>t8sH!8XXJCkJ}NBw?*+=uzToBa
zM49A6T$#>PP`_?_0QjLz)P{TEN_E`jDGUq+{;2i?$EN~uL2*OyqMvBMz(>lC8z_MO
z50%^r3?5Rv7pMUPAD}$Ydr<KK^rJ{eeoOzCHV_Um4*lX{;G7YvKQMBCjjWgX-&((X
zHY^j@^J}pH{{B`xvOu--*Org;^VgRDz4|ld!*)tfK5<0fg^~PDjGIcU;jhtt{H^;h
z9;RQL)VRNJe0cw|2Y+<^cffEV`aubB3eW?r0L})^1$u$k0dD}V1p0w@0c(I;fR6*W
z0-px%0zMDi1AG<uI&eSm0Pua_hrrK){{{*imux>Cd?L^eJQ3Id*bUeRcs6hlFatOa
zm<KEY^8Wm#$orf1H{<-U6ZjhN5b#@|^;Ol6Qh;f|ETA9baR7J-=tjQ%HC28t_{G56
zfcF9)1MUXC1w0Hq3QXFo?6n8>0H&j!@xXzQj|H;*Dd4@p70jdD2H;L$9k7P&08?I9
zet3YZfC1obh{Mle1@`Azk2&;H+$`|3fmcNEy#MAxUWfYlI6mI}^mxblcpt}e(T|Q-
zKjh==w~pU&Fd*0N$n=te`uQnX@muXTg@G05Kir>w4!M8*-umS_Hkr~e-~PFJIbW6f
z<@1tAf4My4Z;c<%&+h!0_#LnMd1(ue4@UL;*7n&Ep!?9i@z_t*z+I5<1HKRZ63F|@
zG*XR^%tvAq<TnFW|4q&Jxjnzt|5m(%|HJ6MQ?<u=N8@&}TXH|jW5Iiu%69`fpO#JQ
z>F_8O$Kcw=*ZVw9e2#pvu5mv1lkbqPMdb7KR0#Rbh<x^g_Z#z!FS(RoZvejlCXH6|
z_Q1ZtA;5g#Y~Ui`-+<auRemG*oxnGM2Z2X{$zznh20RNm6gUA`0-OW937Cue?g9T#
z;4a_+;HN;#`KrEFz!H={75q8C3xSsbuL7<F`cRJJc{k)w178Dv1#CVRaRqh<4hFh`
zR{|FSZv}1wZU;UOd>6=md<NcffvPVB*d3S-91Sc3UJG0aychTw@LAwN;1Qs8oT}#}
z;AX_7EBH)c9x#Y}t521409`;Ya04(1oQ3iWf%avp{3`Gd109g>0KXsj70`B}D%S<*
zf_@fwFZheVPX_uS-wB)#`K`b@$WxXp`%a(-_%QUI1?~ra1pFS@cDyQo1~40VDewy5
zD&T#<Cx9;j-vxdJY&t>NO$GJ=4gpRCP6Muj-vJ=^uSl$f{`J5cfwutf2W|)M2JQuZ
z2K*YB^cUslsX#tX`+?5@jsT7a76HqEUSKuwHsD6!4&d9sZ-C7&QuTKR_624D&jUur
zA;w3KKpx^X5jY8W4R8u@I&c>73gFei1;A?H65w*+9^g&D+kkfhHvsPgJ_vjS_!w|2
z@M++)z!!mgfo}p201pB`0!EKUOd1)1nyb|FyV|)5gI6l_0|UUCtDy%z($1gF2mh(&
zYdjBgNAThSm0$bK4eGx5&p4Qjc{HnaoUO(oexAZ|#@5J}&s|vdYtRq>ykGt{|8Blq
z`FE~CrKd&scP@DPeqmI9vyso{+~9wN{#5WhFD{SJzYaWoFHyI@7(CCj$3LX}W&6Ju
zS6!v-Jp|ked;@6R{L}U2PbsC}1)5Rs#%!Bf$68cTQdHg`79Pm;AA70SruR8mKBFYR
zkbdTclN<7qvKbX&X2f{N8-6%Om2g3>|N0I6o($W`ZIEAHRzA7jloN7K{)`!y7qT|J
zd&ctevWl{aWmC9-gXJ>|rj$;cTr>wo(jYJQOl&YhTN>muippn|PHYHJ3d<WeSJ)wM
zWQ3TyUF!Rc0ibwR@pfPxFtQx;!97af4Ris0Krhe(<Z_W%^P#fC<Af7Db<?QF0hhNQ
zQRRYPDs+9N&<_;AI^=UX#z5RMecW*$g&oF_;_JRwsDbx@uLU|xs(g)5$aWb6P9+Zx
zQs@T;fOSCExzWpehAVksm_j%BNVG%F^)mWLs&XD+E$leJ*MT=G@3=sfcZ^f$1@9iO
zcqe$SFA`msDm^#Q=T>|zcrP%r9P@Q$sCOFb1G<4eU=YaV7(H`TxthxrdVoHlhWr3{
zF30G5M3oC}Qs@OAi4MrQ9HVcGD(42)!Hx#rj&erj>z+{MT|hTb+XjEY`+;09qvNpB
z6F@)E4b(nX`3|55Xb08;{Xn+MSo4J{AN*Wl&6f&2Ko`)#a_DK`oha{xob58YvzF_}
zrw<qeI<uAB53B`pImX}!=(!ZSFHq<j1stcab_{TsLN5PMt~x(C^VEA2EVodS+uM;}
zJ286x+sGG}MbEE8zHeIe{FXyi{k8KW^G*6A&{X8tEm!#t_-Tq){$(QHy|!_AvsvYj
z;qv!1&bL_PTb}d|GxCEK5%x{0d?lBk9a&z!=}YgnBHy{GaX#0-1^JFA8s~HU0sBvX
z-&fXgJd^iEH9rm9qR@FO_&XJfn-zLjfrm3*zv6>H_iDv!YZUr{0(JsNvCabPad>?W
z-yc5S_)B9*r{im+?ojNvJMvZktc%$Hk@n=lB1v`%KF3Gu{1Tq)=X%+X$o?5=i|b`S
zBJ+)Scbz)_2Y|JhM|e7_agN)v=n`c?e!D=XJHRI?R%r4iXi9CtJMjG~_4O=zhZ=Hy
zP@(PgoLls9@5qlK`8ffxx#e7)=jSrD-LBGkUI%e#l*&(lowwx&3MeHi5Gg>O=Zk>q
zhkTvjLdD6@<L755f}eeoIzCyRc898;{dUb&ip`+U_3?Mv>lUd<@psuzhTbVaejY>D
zF?xHyy+P^I{?>VpbNzSOt5yB{UG`TwaIp7B!3J_bCclDE09XUu4BQE<1s(#Dzk2!>
zs8uHo-cKKtsB5FHV_thj?FSF?$G1~*E8g2Xv_s{)-5tK%A&)%BE!&lR70P*F&)Qa%
zcfyYQQdLfa+~Za2&)<T69qRXgtn9r9d+rNWxw=D2?uA^y|8L<}9ncH=)@PKycB1kl
zi1MrIlw2TxGx!wfZ9x6nZl&i#eGc%4V5b`ORJT_4eLy?xl%U=I8fB-N{YKory-vlS
z`$rIb5OMexaT)lOD(_5rTR+a~wkbaCD}@5}Y2f|be$>O?-(R~`=?6L9sLzW1=R`e0
z#4!!+bt3*6+B2Km!Hxs@J9c%0)c*306M8?<On=q=c27xY^8@vpK-cSZ-k-csN5^fd
zABepxbYAPA^1VrlFV@uY?(3v@@eg2oRenA4Hvv7ns&qTPCzYQ61;x9{;ZGN3U!1D2
zW~0(`?o@m|wpI3B`!GM-rP|?aq2zV1qFfhc*HfeT$o3uVp#1m1zWY99&t3%m6V&<E
zfqK}!f4B0xW~z#tqoeXK*iE4q?Gp!-UmWL%-O`A?h6Y_HNm3h06(h&vjRRWW{IAE>
zwRwJR#iUb4n+Dyz$Tf<-E0=#_yz5Bo>60saENpw(12<i^u>W_tZ;q{8m>j<+bmncL
z>mRmEKQLkFqGc~GSbS!)>1&@EbJw75OH5Clcj^<<-+Ze5sD}o8H)`#-8@<<$NPFs&
zw)=-adENWY2X~z`{L>FVxPSiOjL8?B|Hji#e97Z?6xVc->~?bO<?;7eJwAgp#^`p9
ztq5l)eOz>+0xQOIKHqS?4Hh(r_{<T(dy(w~M)0*T%x#l58Klp$y|N_|3FOzT{*=V^
zMXOBuD1DOWdWrJfDP?kQaUPMAs39(mw<$|c(!Tr#ibR=M&hw|WxDJtaDEk_|?qR)1
zWPL8r<@kCiQlEL&e<h;6`s*ZCj6l}s@?0NZKSik5v#f8%e3i@bbydA$#0Kkgc`la<
zI}z&jY#-!DmdpH^a;(qgxjw!=tFItpqrjj(zHW<9uV=YFF3<Jx^<2GT#Kw&V^{x4Z
z?c?%XA7B4PR8!9;U*^)!E6LLo_Sk)%&i8l;xEHunp`%(|k9ha1^RE8@&e!!88*kL&
z{JcE^<#B&15g-3foc{w6>r)zAi*Y{}0r!%h-XA|#{{MP@Wc;eN`@a8A#6&-dgq=rf
zF+Y!#2cP|I``f?#x0U-(#NxlH&dB~@uf#n6Kjfq*p|?J-t10Lb9dW%=0}Mo9EqKNt
zc*Z*LjN*Cp8=yS`HSmlM@QhCIj4tquZt#pA@QhyYj6U#;e(;Pn;28tp8Ee5a2EjAd
zfoBvip#4C51Zv<J9pD+A;2B-u8QtI+J>VI=;2C}38EY?5{eZCs`HTVZjJ4nygWwtK
zz%z;$(SD#k0yXfA4)Bal@Qg0-jBfCZ9`KA_@Qgn2jDGNpHQ*To;2CSdGX}vk)`4df
zwP-)k9)TKoMhAFCCwN8|ct$sPMh|#KFL*{Dct$^X#v1U90q~5q;2DG98SB6^ialsQ
z&>n#rct!_!Mkjbi7kEZDct#I+MlX0q=lQC?Gy0LwSOcCh0G_cHJYx_%V;y)#@e<k(
zv`3%@p3woG(FvZ>1)k9jp3wuI(F>l@2cFRnp0NfzV*os3EqKNtc*Z*Lj3U3Qex6~p
zzpUcHsDWp6fM;}i)c*3cS9V;;XLN&S^nhoqJ^ysQenubi8U5fHYrr!Gz%$l@XAFX8
ztOL&|a37P=4xUj1&*%Wp=mgK`0?+6M&*%Zq=mpPc5oWC=%{`oShoZWg5a(;2j>zjk
zJ$dlP4A8e#&0CyrHlDxq$NT4uHRlNZ{^m>|n~OxQN8Ue?>DwbXP83^zUnICm$+aid
zb)+^}-IsFYD69e24Oe{a2!)Z)|HOGp?gj>dK9`b<(Z~n(!1ZhWbuF%QJ?ASu7tjH;
z1M9}9e11OYgS-~<8lVs80lI(=Ai{7Ac=tArQ>4Omxc-RSr}Ta;T2L<1ZY1x99`C=%
z>x>$M>kluUH$>X|Ii9aSD*vqOD_kdWKaFf(z@UABoj<*QjM~ZV<NeF=74V~`m%4v$
zR6ZvEzYziMkBq;z)&5M3cLoK`cu+7B53;$QwezAEaP8}<GH}~}mHcr4N(W3k^CUV}
zw$~p$AGq~t{R#M!lXs-)`4PO7MNN)|-;1uVixgdN{<h;RY8^4*e}4on`00Kd*#Pdh
zybR&!1apt_bw@sn)Qj=42*e@~i$E*_u?WN>5Q{)80<j3hBJiJ!fcHOFb*$!C1Y!}0
zMIaV|{|_T@bVBL{d)FTM?C69w^PkzV?NPoz^hY;U-fxO#J5nZjsmt#V{n36LWAEqg
zpVg>!LC2Jf31Sh5MIaV|SOj7bh(#b4f&cOdxX16;-$!9LBj;T#k6|nVu?WN>5Q{)8
z0<j3hA`pu}ECR6z#3B%jKr8~W2*e@~i$E*_u?WN>@MlG!Zcu0Sp6qe1(BDthqxJf8
z>O0QOiJ6Z@;QvGfBHus#Khfw|BaeFoj!q~z?dB=-X<gL2og`LwQSa~nY3q`Xa|{35
zp8p+hf8;oEoI`qC%~O9o+MnLjJUSsWw8S~vesse89~7So`8Htczm<F&u;6>en}L}u
z2d)8Teh0l=g#|$KSjDfIt#BLTMT$@LD13L4!h)d+=Z{pl4OlQt@tGrl!xfq>XX@?R
zmY{GAc(V!ePoQ51{UESL;r#yK&rrC{r10Hd3R7=UxW=b2bFRVyC-To#xNQ*f;}y;a
zwtT~*`}eL{$=9SSzsy4vX5QXg&o3yJJon?0SAJ~tyuIYR*n8Q>^#F-5&fnjN5Xab9
z1Y!}0MIaV|KRE)|WRJ)(tDoA@YmO6x^!EXL^5eynocdoOf?N3vDrM=SDgA9PPSSOM
zYVN&wvnt0Zgr-}RQu)ai-(xD@_gDvA&X|G^VQBn843v}S4_U76aaFGFaaFD!l~~jP
z@p%~i-34iCC*+aXNv~Z9My}TNenp|&4&JH?M&e`mz%wJ48%^c#Ng7!GY2dHWk3oG#
zaq*x&Ml?rnpZx6=nLT9G1>&<oV(O2*`>Z{A<jBB#P4|9ObVG~n(t~^=`N<nupZ;4f
zsm_9=IJ3wWBExEF($XS-mnq&VET%RIq6I}eL#(K*v|7a!k*w8JdPU6{&SrImUUOy@
zS+LlJNlO;=Q%@<PV}jXwf|evKL6gwvUv7t%C$Cy2PVe1^erZbo9KW}NIl*fdR&9td
zjqXI>qMuJDC`o_h@7lDJ45ju&vio%)N+s$-#HlN7bR+7{0>aaX<X3%q6ZIjIsjq_i
zzy0dB8aDbjY&P0Dleht4lCx=hAkjHtvO(C+Aj~AnBBB_I9HOB_!-+-`$uvrV{#Sl)
ze>Bk;qOnBuQ!Zj0kxUba{z7ySk^K0O{y`!_H_=3*Vj`JJh)VS>Auc1FOf-dPDrD1u
z67(OxA@o2tov@r}CebV+`HPl2{{HN!Q?B}U`qYh|w7;lo$?LaRueUz(>VeN^O@FDU
zZ^<NW`2H_mdhh;LJ-#W3Gc_xnrF}6fY2ytS^fd3La*3b0cAtLLOQ|1G{*}*+eQEU9
zP3Ltue_<ETxG!FOxpU2__rGd39hx#nOWWS6N5|(a>$?%A-ZkiiZZmG|wr`i!@l>wa
z)cUg5K6gF4@b3#|nM~HfN5$vbeH~7}eR|!=_k7lK_0`v(U|m1uZ>h8AW$mvT)w|h+
z?MIEcadgj5zUaO^-<IEf(j%>UTs@$2-xJwqzEDcr<HZRh|4MdJ$j<2EW&Rm0XK9yr
z+dkRaceMYG$3H)K$KHvS0&8K_J-b`=cs(nA<DA4Fa>`5F^te0swa3=o`RYq2eA0Pu
z*~3o_JKDsl-8*lB<K!LZYbQ_7>c4NzH>rnRUH`S=(N;aSlwI&m?SMAU2Pyw9;_oK<
z?j>QGvEk(&9V_-{9*8SB@5YUn1e>4s{NSs07FRE@HQl&yZKXE-{)yRnfy=6^x^}#_
zO`-eSCH*cLGvvKihd(>*wq4sPmQVk9!baL1QPWB~u;q6UBU6fjT>iRRG<|v8Ymr8X
zpJbpv)xiEjDih6qg@OK?2Kplm^hX%@lV%_{k$sIQGQC2AXnwXZsP|d}`^yaUuQHJL
zpnA1PXNYfYU|)_Ka;%F?ACrCy@%b_LNStQbuI56#tV+mZStdK=j{6i}r$B5Vl=_++
zEGb<?c4hKV;H2E;)>+XNKQMM7<WBu7XhN)1*<u*vbr&l^F7YybuE(LdaHlH42qhLr
zSpOa+fH|?ppk1}FQ`14&;km`%^td$_&XG#~7x-U`_WIsZ_N!6va>zZeH?H?mG8iqM
zavYX%@MubZN4&!B@IN32WlB{F#M=h`pMrX|`<3De(0>(n9IKR|KjiDUy{D*l@p<JN
z%9sAsj#r9%p)bz~Qtp%|LQ4GmHUW-*MBF+fZvI}%{@J7_)0qZwNP?d=@?=D*iz+6v
z4D@SANBZMEU01L7CjtHh?aDxFl`Z6CO6u2TDES=7Pl4RtL&@h;9A&zfSgG&!>a4gM
z@|lRU8-d||H5GO|naa)!Xm3x*gT0kJ4R$Vu+y{TKo5huo>m8n|E{Fcf@Fyrw)Rgup
z5FewxPV|2{hmk2Ae%7Ktbnm3(*FwMcMCDI1{FJ|%DckG1SLp>6i0y<@?nj*YxVQuU
zIJ`=+BlN%HdJ%tgXK_B-<w{reo(=u4NgnOESO)vrRZ1}lex41v3*!Kf4{-+l<Po$t
zaH^_zxdKrEf1Ei=@C58<LoUu#^5M{bk5I<Jj`8Ge_<u6&Yb}%_%!qwlFOG|R=y!(P
z+fwPD0{Lv%_o8Fcd{s}w5l=71+x5`Dg;4hg<0@Zoj5dhFF4z~9%77zT;f-jocZBkX
zCjtS~t6_XAM!jyx9r6T6$pJr4V>$X8@0UrWqxUO}8$nr!(*1}-9mby>N-TzPe3F$v
z9U;FMav%E7Qpj&$JMEPIKZuv<6ZjKAKjip)2mKlxFFe<kmy3El7pU@|DiC&qc*^;!
z)OVyQ#US)EVBdrNn+W+B$ZOA6etHnMlMU>5H;@;z|A-GCzZ2nq5ChUARgLHeeFyp{
zy0v(Q{Y3v906))#+ztEudzkOC9mI{?)6={3SKq&56n`ZS_&X8*+Sir*TlmwLkFQBK
z`vSAMEWu_SmN<_7Jf*I6X46D*ZCNO!AE(1?_9}CREoDL6Fq<ZIQ!;G!3|YcFkC<GY
z$w@R%wOOB{4kYK8o9T~bdML#HN!f1shh*GF$p-09k1##C8^0o18p-l$TU;ppD<JoV
zBo47zhdxD>EimWWWIvMSwH9>k6INcz`jV`KWIP5-SuV*cNXGLL8PAJJRzotq?-fhC
zQ>dPtM0v`cO?h8Y-WRkjGHv!H=1g14;<yZ(R%OYsr7dVO%;q##+Hzes*BD!FhRvCk
zU`wN1iV|sON_QtV7Dz)v$ogL)+pEgyH$kO<v~MsEx1}tJ%d%;UEm^j-swP=B$AWnC
zFtRw<rcu4hVpihd2uVgF-HD-l7Bs;<MD{9VUykPMaPpP*S|k64sj_Y=+hb-Zq(2~O
zD4Sj1&LK8uToY47;jF|8s_Hf>8=MsiJt+IsaGQOpjL{8oRCb9a)0VbawxcRO!<N3F
zDMc$ThpeRM*c=qT%&=zVCT7{v2OlHOcxuP)=U;enu+6!kX;u8<CQB?g#4R=J?M-e)
z{eyncva_tW!Dose_6^lL=NONv50JWx%C?^q3hDC;Dhr><A^4n8?{k*T9akWICQ*dX
zIf>HNZDh-RWhkW2seam4j?FzcswLVDx?hLL*4V2<A$?x*)3(Sjb753tS&8~F(U;CC
z>E4E8Z!Y<oOR~Ns+b{b_NqxH;_EmVhUFPDb_A?T%i7FqG7&plzkA1nvhT2?>_LvZb
zROL>pf5rSzNI#EXSzmuc+#CAJ5Sz>VXw-Ug5-*9?RfedF_4`H+nrO~lK>LL16eHy^
zJyajs#*=#v61gb<Hp<t}TiG`IVmYo=$>Vx~#XLccY;^Qy^ZCL_Wh$x6w^YW8{uW-Q
zw7y7g;v`$zFq@-7KhCDt%QF(?F}jm%Y_AN37Bsd&eU|F2u2;@UET`?k`UpZtH;qF?
zmNr79RE0t=9N)+E+3?so%?BuU#~x-`iC2*P*yAiS(Yz>Xu(R|&?xng9Ul$5pPUjtY
ztQ}K#qka=-iGI9fC8kjE-DFF<J`~dD*b{Bqc{b}Pn?`M;yhD^%NO}6SYr5QTQ)$0t
z>anw&Nb#dQeJ)Qv95WVY(dcp_)!b-+8>;*7q_P(;4Tbc1<<I(Wel=>0Xivy#?qaf4
zyF3)quLY>@+#tuhC2`b;7t3=(RTGNEg7~-;=}F^(+f1=<B+5=S=h)I|R6JiFUQ`SY
zll{6Cq0mOEOAN6EP3E-}gN($n#9ApxD{c&h^y}Vf_4~o#_>GH;z8`WD<1BG}pz7m}
zo9y0pb12losBbmezvj^tAL?xp!y%131IfpG$(HxlP)MIs8OLXIWPFD6!6w@sOR)&V
zBJh7U0&b`JU3Lw~fv}HLatl4_qNh{Bc?t7PXwG8|<Ku-!{~Gcwly4Wc#Y*2w+cH0#
z_X=+quMsCqoG&@ENmv?qvx#^+^X#AV;m7LN7130#{yvyiqvZaue0i_o1QhfmU)!Yg
zOeP`Zr8wu`3i&<2j*lq4P2jfz_x)4JUjY9u@G~HnBhz|%(4x=={xR@L;GMW3LU)+;
z<Q%6;wgu16X-@(#ug7F+r$DU2^IX0L<NPB(hRUQ$T?e&4`1<gWL-Dw56~4}j*Pl_6
z{z@qFsY1>fWRlYoJq6#=<1bHRGOa>GY1*kLzE4W0SUqhRsO+<R!vMvzpKQ1p^tryH
zA?3dZhJFCgar_bdcm#@;H|p)>xYvQdy1%OLJMbJQRCnA#%l7K=;^*e}vpRga;~F8n
z<Ajd01}XolfGdIPft!FkfO~-lfnNjT&sBEX0J{MD19O1mfW^RBKtJ{?2jq<2XY|SM
z+q<wY{hR10>gzbOuzyj0|9<I3y?amW+pDmsV5Yp)B{D0jl0>f=B{M3@EAk6OuW4l!
zMZG3Xo7t;iX6cl|o~4CC=SuQtl!#u1bEeIhGgV<lxyqSUR6e7$Y+55G52cWoN2=vT
zQ}ShbY<Z?sh+d`BO6f0c_L@{i8x=*f>2EPfNvW(bzan4sDk{k<F3+D@lvh$nMwnAY
z^YY8f^XI4%ye&(~y2ya;X8zRDiR4;Yg{+1oy3v9eGjw<J^e9Xzoi>@H_S;b&Z}J%7
z<H~5=1lNFJgjpenk*}$bbNM<P2#kIlz<eCnAAhirMksyY@;;0MjN>sN$a<w|F3-oS
zyyla(x%?q{Gl&u+k1I-g?6)w}DMKE^T>fghPLYXGzduAg`_Foe*_0>82<Ca*U_69)
zNdM&+!v52yp5)wLjwxK8#}!5w^dj32vA%9V=5hHA7?&8?arU3<=lIDr1+si53~s<c
z#mM8VEH8(|$nuwyOg53nbv~~!@_0-tdXja8r*dM#%e!%2V$={nE`!}mXL*(6V~g<e
z-cyyFk^O~wz5gt)iYU+Pq8MxXH+F#QWqXSw%JX@Yk<TxzAKCupD9`)Pjq@<0S9Y+l
zq=jwEP5FF6ddTIC)-Oq2nIh}oK(g@ie)+(O660a+D7v&^|AQ3|(uVBU+`o9<!0Veh
z&aBTo;}eu2=R#bb=M}54ZjAMe%0EjP(qk^~$GpUk@-EdJE-tkCEw!<aK=xfO&*#hf
z^=By0abthDAIzh&QiaP0pX@*!-8X$w71VUD;SZN*3{qBjc@a?MML?DJHx$*kxE$k$
z5#{|T??-voV>6NbXOtmj?EeOo-(XN)dBxV~9y)DC%49B=@7tiNZ*d%1EX#7;ER*;a
z>W_;s9>IneiEYOs5Q{)80<j3hA`puJB4D)s-%+3xoX~Ui;2ES4a{YfZ`kN-Y>T@G`
z&St$x<n;!OJin0VbXm@Y^?Z@Y_d^(YJ|)lZxCErFW0U=_Y6ie4gr+M=shr38v5wz=
zYX@D<=)#SlV6HBvKbpQFTP@3#+%7FhROJ|X-3=qRPqs~xT>m9gBxWhGzHVG@N?n=w
zeFG2T5{d7j0!A+Pd*3%Oii;Z+Fsdjhf91V3xgX{G25CF)e%tflE6q<m;hcnaN%yte
z+$L+OEGqY#EF*PflKKdV&c9EY@_C>}BroOLb~JA^S?y-0GymH-tF^UPW(e>9WA94f
zqN=|C0|8MGMsqFAamQ2`R&f*?6mT?D0yR?$kX>0uHqk7{HOVa6%vRCLY^O!D<yW|5
zSXNlMv{RXB)=#uFO^yFK_ntEhPe!d@zopOrUVVJ;Irp4%@BO~}?z`{Jn>W?Ft_Em#
zc#A1U^EO072zOsMyU!hN`zqU1T2>h!wYP1(a8Rx#>()u_wY!b>-d1};fLlA?9V03$
z+IjmL-KV5SKbGcUY2CWDW*_1furp@yg6ekO0lqO9{c3M}H;)muRguxy)(xFanmxtc
z&{Jy_ZE*9{5(1uRH!48W7J96*z|G{k37Wgb-QBR+l-bb^-n~8AyK7zTk5qWU2btam
zqnlj|@QUfP><VwAW=N^9SAh-OFxIZ{HLP9mfScPQXuRE^RY0h=clQqL6VS@i1Bw^A
z$E;cJ?JhnW1&`}tOMwm3bP0~XX^x!yY@Njn$cUR?fH|<?cLVfkCMoc`9#g)ACV#~Z
zOVZ3D#Ov@P*ot8*fvpraytc<wCPDng&u{0Fd;{P-%HISCw-=jD_GY!3wxAjPLeMR0
zM!y(zOJKw6^_yW^1{<eaI77JHNyh7QPPakkb`}8dpgdlKJEyxrSm~s=7xJq}zK;;E
zyB~n<LD(K5-NS&ZDPKvr25>Fq86M#b;G<4?=j>x3tb^@wQmzMl0=B0}W_TJh^eDJc
zJxc{RDmmdbI;Iz4+aPj{e_n>{CfHtq?N!)bgUva;?kt=q{|_jurt-~Bc3UXF)k)`^
zeG7ydXGJsl9ZovD-gi##g0K^|_h9?Lso+DWJZC?q%x+c?;yr+Uuzd>K=dgVV8>X*d
ztAp)p*!1ZeP<%_u{Z577QT_nn_fC4u{s7xy*nWiVC)hCkQ-b3!UXNp{mmvNgrTouM
zI(_yRP#lNtS1LH^WN?b|zd7kRdm1ulU^`2S-vJvaFTl-NfSHbDye{|9m9eFlPLI#k
zv?864u8e&c@%eODsSTC+>1_N-rbjQeOF*W_DjlIZfUcv?4-4t>Y^^htb%Cub+2C{F
z&WRqM)#&kNtp}9jBbvQn6VItR8$+I-tJkTUhyhM@;P(*Dmj*!jAlT?Je_<Nr!p8*q
zVAw*MDGznhg+V@?WW3GE&#YqbAt0M!!{@-8OT$1P1>0~czmm{G7)>~W5Mzv_d@SK8
zKr3vcNyhJ<V8Um}=rL)jpuf0e8c#OP1$>Y1>SoFkK{tW)6LsZytMwY#+<g{SC2Ux4
zXaRU4<muJlJ*;)_+_mV(j74?k@mZHe-aCBp&W1hL_uF#Det0Wv9zDL=TRyGLUjJEM
zZ#)tDLd%#(H$E}`vxnxTt~%1LSBuom+uk_+X7t#H<_#WofB)G#_j$f{>lfAUeYkAk
z^XtZ>k3aEQkG%We+<Rl7dr;1?sD|$o@0@Y?rGh7~zhu(d<@^8Hb=Rfubjr%uH2T|*
zgPuyd{?l)Vo?bog-kW+S&&(K9aCFtR&-DM~!Go{Nd#{uEspB8~wDX5HLnaSidDR1H
zA0<YQ-FWu4+t%GR=aTW;3YQ&z3$`7*x81s~V3%k2Z2PB?8;+QUz?L<-VEL}pV;wg9
zvZ$uj$U|LEbtxKw+iBSDfi~ytFY*7WmvQ7&+e=5>dYyVSb?u#Ak4!7L?(IW4m2Io{
zcuoxY{r#^$y`yih0jm#09*i;EeqW^9oVg!+_UqYU{EAWE^nZBZb!%VSZQr7u3I6Kz
zz(?n#+TZC=zx{(#ZOnV}e++u+z)!GUeWJhNnWEP|dg#<p!%zJS)(`C)_R<5p!@pd=
z^Vj=Mn4S*#6}A?M+r2g%nsRia*Nh({t-o#XG$y?GgWK`<)=oK6ao6|BFW>u4!~@sW
z=lVbKT+QAk1-njtz50o5J@2f&H0j7AubA?8haaqYZP+2$>hAmS)xk4%{hT_r*Th@8
z=e{uN$0;vHwfxv~`^lfTmIS`|Yz^QwpLSVwzkj6L=H*?tkD5H#e?jgw!?*tSVEqSB
zcI3+!ulS(;^oFlK7!~)^vy0!m?D*uUJD>5G{%~sWhS!F;Z|kOY%KOzm)^GNAE9Z6Z
zU6gY8lC@*HrTb4A(&NY{JHJ>`7JAxv`OBLgJ96{f#G3u@1pn~X`iaTqZ+~9%(~AX9
zev&xgi9sKpvUkc@)w|W&Wd)y&OMd>ns*=2nd#}0VgP`ueO*onV<ASwyLyz44i~rn%
zlWy}o@xTz<v<LQVUG$=NtIlU0?)q_9hsRIfQ8TsUfp0$;Tl&#;{|Iv{8U0|;v<|P7
zTnF322X@shg01AWnAZ<SzyA9dFKm0P&kwJxY0>kE1Ft+e((Sf47SG1*=K5=E?&<sL
zJ3EqRU(^4Y;d@W+DV%dhmjiW0zb&||B&%wY;mxOqEDE|jVD{bz=5^ih!nTYy$(PL7
zJm~2wPk%7?>(;-$Qk`=5{O|XkSa#&&{=@fP*JbYa-=?=8`Ge8$X13q9*_$m7B!>=j
zn|l9H|EHeZdt=Ugd*69g!<YHYy5^Cu+J5omKl(iK=z#0ro_plzS94c<vuDe@y*7Vy
zWBjz@L1%0~{^Qy;ckj+<_58l5+!IGr7X7|wz~<%M^KCy4>G6$!|EY6Vhie1FuB&b}
z+`ngNC$}@Zznb(>i;Z9X{Ok=qZtmGJX~ED_HwU*pvG4gg_l7PFDF5!oy`C$2R2W|C
zP&5KJ=lR1qf7}<Jvp+X|3-d&C`j#&AcS4Bf>~YR-PJd;a=JjB%8P9EwHrBm)``?<;
z<9fEaWN}gsgo^N^&wHG7_}N->=S(&kz?q9c=XGau*DV<04cNkUv3T9BQ=eC?by2zz
z^qpOlHK)Fnl=?MibGE44?NnfP%5(N&*V1Uu*A(pA9Qrk*)USo~CVc&ZYaQp*!&x{_
zzAcpDHHq^&;ymW(6b%CnuGw+@#7R6e&H^A_lVBPI8?Rl3;yi!ehq4qZ$Lj%=6a$I@
z#eiZ!F`yVw4E(P$V2_u-4_Q0tOYyv}g=Ax}lq(iA{r$)klB)=-SzajZstFB*MnXGb
z1z|m*wn+L_ekQSkuyVJQ>$g%|Ld%Czu6kQy!cK_>@=N(d$`-<kPo-=pStGgfMyao!
zE3uwr<9zZjC%>B|R^CqdxWu~s5{(BWS{fwQ9+X)1g2eilCE7PjtoTZzsZL@wq5X`M
z>j>B1Dc>Vn`@7WFr%5zUlc@3jq5Z|n*>e1H03??0BAM6f>q&;oOp(eaNrcNkk&5ZY
zSr?M|^UKRfHV|G)SVf<6b|u-H(1XxeB*zgymv28NpQHDr^2yz0f4oTU+#t(alN{Mz
zW_-Auumz!3A$fw!PLb>_C6b#)qIh-`OHK}#^|3x7Qp()!_0*77GK}MTji1Z6#K{7F
zE}z?VIoUHmr{S4eR0WWppWCk<B^CVKKDV2n+ph?f?cwM4qsaf?1@M(!k>KmjB2B(t
zKBu2@jl|09ZV~gsek$j7q@>7le$JoudgRZsjp0q7|6dm;`?-;DJ7Gn^3Q^wyl1~tN
zjgs~q3HuU85XKQsBFrQ#C0t6ll5id2M#Alcdk7B@o*?wHQvHN|2_p#O2qzI{5|$D!
zC0t3kj&LL4cEUY`2MA9PdX1*~3HuU85XKQsBFrQ#C0t6ll5id2M#Alcdk7B@o*=C5
zbi3Fu9mmLa^d*cSj3b;xm`PYlxRh`u;X1;Ngxd-C5GEAL?}?ls*=CS(M|fErevhOt
zVFY0u;UvOL!cxMegewVooL162%kyOw$&A$`Gtyz;_=pPUA)bpEIkrN@fMP%~pcqgL
zC<YV*iUGxdVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r6a$I@#eiZ!F`yVw3@8Q^1BwB~
zfMP%~pcqgLC<YV*iUGxdVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r6a$I@#eiZ!F`yVw
z3@8Tvo(x!??IM03lktDG7gtilPIi{RziHtYpET28ez!t1Mdzx{iDN(4MrjlSiUGxd
zVn8vV7*Gr-1{4E|f&WPcT>buE1+DWf)M~>${3b+A!{7gF34brsE*0;w<ab3j*TLSo
z(vOjUtpKk%@KuiXQdw><?J3+D`S;E1`9+qRQIw-K`eolF{jzV8evCG{m|^7iM&kP}
zajT(nJ@$q&OpIuYiGPnTCR)~~$6R_JB_sR&tG~zRiWfV*_>z&jBl?E#)Wmn(;cxAw
ze|6c)b^(z=UoTzU`{8}sQ{CQw@?-1|zMmf%<9Fe8hdgqG`w-u`iSPBqcZpKBv~G~c
zcadVX_%2O+Jt_V^1;39JvLazS@b@A3_Y;he=XZP}Cp}>61e<f>cbMW(>i`?JjT5rX
z??mNymvR{n34EWYKPv#y7u$e!VSkJs(OPGt7FO<V+~(QR?y*lZ8#VVH9*d&4J!g0{
z0DqcEaDevQSL{3Y3+u-C*k5cL_6ysKbz^&su+?1Vx%#W03U1#!<F_>_@H;r4!0(Xk
z-+cK&{5_KE@1LJ_bN$MB+x_aITJL$Sca))ILeAhZ8}8a=`)R@xkBxX~>DE@qIvzOQ
zEB*Ch)-NZQJ>+5O^v@0dShDb*n`=MQrfnIz<Z#!S_F*r)3cp2i<ilI$#hLS#4tr)u
zo6ld_yL#rxfYs~oZeM@cEp8b!quGS*ZwGtMbU*gA@psuDCw<DRQa?hJYT7Z<+wjMs
zAYq>|b%Y(!Ry*o=pQ6;lZz8$S&jACh*Fe9ozLp$(UD)q}>d?NL?4!vZe@Ck)=#3j?
zd%{ScM*6y!rT%8p-vT1WvEz?cK}sh*9_MXgWLoa%DKr73&jxG@|0dJ?fnx^K-LPSt
zdWth!Xyy1ty(Lb@-%9!<vbU5;{cogi4I;L`ZiCcEk{-iiy_VUsJzVd6&|~~s`~et9
z6M!#FPm|s<N9tS9f%`g?p?$)gQa?plLu|57m?!mpNsm8!iuP5s-{VN%oAj0R<7rf_
zIF{PNC`hr(c9zk>b^`318xP-7d+P6#_40dhhmc<U7#QT2QN1ewk)NuUq&>C+6aHox
zwlf8PpbXOG)ZZi!k+&4$Z4=qQ4>m$i`w@?4OfxA?#RF2V;)6W7usAn8IV-nddTOqg
zoSc<WoSbGWNzTkjEiNg{DAJPCbCWZ3^3#*kiwcr63ks79rq9SoD;D~^)R`H{Ma8LU
zGm|sQGSW(lQ>W)<h?0!5v<zEuPC<ThS|LONWpZxL^fWfkPZw5csl{p8;7YbyUTSV`
zL7FH^&k%WVO)k!rttiUL%1_O8GEYt}F3HC#MXO6i`$a7oQd<N~D#WTGf-p%<PcIbF
zv6rZXwioB5CFkU47HDaWl{;FLoLQJ5JCvR=y(CN6OF6YLt5k$?ZYBiE0cM;SO)f0Z
zfB;9gL{WN9a#2R1s1(`FODibJmxMY86VbWPg@pyO<&8~~#NsZ=OT|i)MH7V=@hY5h
z(+B0|<d>8UDvOL56cH9!R1g@Vjfx*VVq|hiU`SxNBOmI>g;6dcarC6<39&F>Jn3Hx
zKo8j9=Vl#$u}n&AB{gV^dil={jv)MZZ0L<G@jQs<jIU|TBDh0Y$)!21xD1agOb9KY
z^pc#MHdxjhHk@m5o^ZDD0GswsHaNec4ccPDIM|*gAEb#oP=|9l+Mq3`nzEiEi+Yjp
zhY%sv>hwG>?bGS2s)V^#r_)#9feh)OPH(^;^@CKe(;H_<{Yjm^K2qwnIZfMXctPsz
zI{gjsW4xF)>GZ~-QeUsruONHl+@|rLAbo&NUrqYSI{j|ar|9(Gk$#m<e~R?AI=x>9
z*-q~pnzm;E=_7S|3+XL7eFEuiI(;VT%XE6{P+4!4PG3g)gF1aB>5cQ6wzG!xlXd!f
z(pTv8v#CARI{gaL*XZ;Qkp84j{{rc?8=JOgE9p%-{a2)q)ag%<-lo$V@Zf<|snZ9L
ze!WiLd6=}X)9KA*-&wcrNF=>gr_Uh$zWK6WcpBs12Eg{zIqCVG_^9tp=WSfm^QjN@
zMkhV4?KuTF>1E4hiOEUtygrX~((`Wtpohgt@4Vi(I_VvA4cIq?FPzm=Yv{8W{`z;f
zKn~?>40WguU*_FsF3NbYAF35{A<EdqL)Ai_fikWV4pj+xI?A{VI8-U*DJbLR&7le*
zC!&m$91`!=@AIjxr<Pd(vPjlNQLC`2N!I1vaP4C)@0J7LzXX3=)w<#a%Ny7f>*nu0
z4laOjVEzQegOa2xt>vB(=&{0Pu)b{(owSx4io07^w0X9ZJLtpRescqes)Y&o)osv)
zgMRBlvD|YnG$DA0bxE}~ZCCMS)<v5Fu>Hk8*3I8|G;Fq($8_|zF1K_({DpPMM<U+!
zGT!s2L_CC7Ycbv%kGp9HKY`z(J@}FliXdS5_L82~72Q@qmDchv-vlEqxcXow`~>ft
z^C5rG0)LuI^~2@w0mxYv=4+ChvsF69fMP%~pcqgLC<YV*iUGyI|0)A1yI&Q*PgLzA
zpNr#m4*YPu&=1^6SR;|k84Z@+Bfpr)5?t@>l<^uhe=KKzN<L3m^^!!E&x!J2za4LA
zm$hHh5N90b{rkXk%9TPf@LyoS^pX61KgN6VWFu1OIax+=Ee{vW`X<))(K?>RD^Zmc
z1HeE9jhp|!Wbhpq$-wK=hQHuB-ufS>!`@dNm;Z6n{~uOpcvP-?Ty@>~84t8+dUU{j
z$tTb6?<%g+D+Uw;iUGxdVn8vV7*Gr-2L3-X;OhB*W4UC*LZq0kYkc|xpZ{;AS>V~U
zUSd7Y&s;XgU)4=g%c#-!3UD1SzBgbeFC9)iDp&f9&VHgkJ3h()Nr$#yq#vWRA3pS9
zn<HCmD{qPc14J~fitnsaJFyn@tA0q@S3lDf>*<1ok)LfrE8HTfnR@IEWtbSz788Hp
zpv+%t^q5W;M2zhBuYTXa6)zQiWxE)uyP|LSd?G%0gWorJ)8n6R!=LNeu-NT~wS%tc
zcyE`_?~FjtMX+5D8`6o-2~~XF!_B?6?LzzHCoM}VlE#3)dqAf5Ez5)FK58<!_xupd
z?5)DAlfVAB@`_6Wmh_l5d5fQKr=Hz3TPMQ`leW-oOdPaz*?^wyTK8QWX>_~a)MJcy
zNBbJXqKpyW3=AFR-m{(El4`VgT6?#Br9<1+J$wuvmQ4e_-1j7C_LNOA*p_72Qec}7
zTRLnRuw}xADGN4y`e+7hxv<ggOf4VsHrOx~O3+FG@%eCk<{i@wu+4+*M%eI{rX9A0
zuq}ZN(^3f<KC^xcZ1@ax1#EZ2b`NZrR>F2KY^zX!bRTT@!-kJBKM32yu&se@Eo_g$
z#_4g$tcUGsQal6r9Bfsvy+FDb0bim#!bZw(0(=Fw*I>iv%HM=-3v63q+XmZPux*F!
zZP+mFfbCt_cEX0wU+;nq)5oywhHWow`(XPNHcp>I=1bV>Nbxn`x0K%xc!2VU0Dpk(
zN7#<QRu3C~Hv!Xe3EC;Z)3BW(xd9OdrW<VTuwiNe2oE`F-mtZT4WGyNfsGz>*9?#^
zf78E*&w>1!J8$&gyZP>%t%lduuPb`2H1CmXH}3mjz?rvx9eTR^Q|nf*^ffJA9Ccf9
zP5SqbuV3(F#+9LOm|F%-*m?NLCEiPVEu8<^rxQwlEE)Q2`Y#zPl3x9wws2@j_|5k6
zCp-42O@HkDZ$2A;P3?%cz1MyD!P$h84<|nI(7v}0e14$){FinO*zr@(IcqL^<jEby
zHLu+E@{nmqu1u=$IdJaV^Pa4`=egpVetS!Lr$1YI@01rN?EK^(OG`ItZ{ONEGI>qV
zq}OhFVN+nd->{EzJ-gg<s`lYKI(ff5aBp1OSsAD2e%HRsLw(X$fBAUwj+d{RH-1U*
zRafs>Tr(@^XVdPbYu_I`Zp_#APCs1QtMI+|;+N(>>1BC-N}sg@zw^1}Y)MLs+Wbv>
zPmNnKVtLuq?fO-Ie>!FS4G#|W_%WuS#>=p~%}v*K9<?a{(@&p2e%rf=ALZl+?f2?_
zwW%bgBBI-ED+YCX=!2GTz3H>^<${wNhh05-$n1UBN45@sFKqIj1?yftKHLzy?LFTc
z?%VKI{wr^^3x4jGXFP(QIFmi&)w@Sd__WnkPlm7AA9J|x=Pgbr-7;_Q-Y;k8J@wj4
zZU?UMy*_H*{NQ0fnSS~3&9$@QBW`}>#F*zN)Lwn<b-On_Hr=Py9sm4bxmU})U3(7B
zx@OPt-Nh}APC3{uZD3-5+v}k>r#(NbZAROVAMpLgxaW&qPx|-<O#XaMW`X_5%nfc?
zMTdSkS$y}wNA7<9uA@~)JKpx))IB}=zEiv;qu@6C`s0DS+w?nL|3mjZzubTOYlGkT
zc+hukm%O`Y&B5Ou2s%CJhdVB>-oJ8oP0}HcW!K(iELrvFj!_?VUGUVlOk2(GV}9GQ
zd`MLO4@-R~KmF3skY7UIU31UnwYj;G*Y25PA9QN-<max^20yo8THoEP%HMrx%sq2z
z_vCFXKajf5e&gdWC+BKE9T||>aq6CtS0C&!aA1dtEi--A%<p>d@D8tEv-iqJzPC&Y
z?=*c!MOo^{ZU*gX;Hvwp3D&7ze!t35f9~I&ZSeaP=ca8Q&FgVaYtH^3aJ)39p9^0K
zYfgX6h5bMm`rF~;&|Dn+8cTC}1GLk4Znrce#A#fqG#WhZ*?gQ|UGOl<MSEUw(O<KR
zdS|&eUvGqq{W8ym{djn|w>ci3aKX>_(Dml*Q(e?M*M&WP&!)LJi(TkvxZo|+h5jiQ
z{T=6`oy%Rcv!9Fj%UtMRcd_5cxv;PIZJviME_mqeVt;+;BF--@n~y)pMSB*z7|)|z
z)ca2ty#4NihdW&Cmn;`?u5!V{X&3!UcX1pYbI~3MuNh&}CtMRWH$ES9(Vp*I^vmLc
z=XYJSXOfHY`Gt#maUIuOI~Ti%@9rXgTd(Hh%yTgw>Rsqhx)_IhU5uZ47y2w0{q5?)
zeu4|0A8`?9rHgU$iHrSs30#0R$MYH&ac*&;KkR~^Di`fM<f6X=VZR$;)2CNl#K-H5
z=Gq?z`kwkAAlqEDe}N00-_f<-dyJ1}f;a(J!}`7MY{OKnBhJrotGm^&^P(0pe#V5>
zew}0d3?ux!tv=~~oumB>v3}mdXBzluUthcrp0g{!XCV0SIU27a9|c{clMYlO{_CK(
zaD1qDse3fGX_TLF8ng)%(SF{eeX=<#eCoj`7ksY5Ai`%P`OI?ojPQZJNq22BnDtJC
z@1=sz3+OY#&#=OMgkR@dvF}SgV*COYdB*seidy>xMEe<`0bl@JjPfbfX<~dP_yvHE
z!$e#^+9CFj5WBXS*dsCaC_hu0E>f(|wK`37GaC2NO<SaizSKi~@!y<n*x;xy7W)EC
zKVM(woKxR5y1v9iym<Jsa}~t4!m)=u-R=&tZ)-8a&$z;4gkQj|p3#1$rC!l~k&9Z!
z_*J^KxW=uuUnJ<F{Q|%SIF;1J!H^Z#-w$HE{bw7V!Fw9Ic(E7{c&B)y{HlH3zi&2N
ztWS%s9)2~xF@CkaBmDNkR_8kf%;{YO6W}`W=d%rX?W|e-65N-z_VXU$Bfihy70T?t
zoNcJWva9^++}u|~VYE+Y5F??i{qeI6w_=&q&*EMHMWdzP1SneuWrAbj_o(nQf~gJ4
z2A(+E@GSbp;+P%fSL^M5V{0*5<3#Mc!DbWK=>75dsDiR-zn*RAg8q=OE!#ks4Z3$w
zA?kZY)Mo*)4$8WoJlo)h`vi{L+i>izz<qM72k^cW##gmli;1u&aQHg*0Wctw1)%pt
z_#)7$vkkplfFO?j7{AU-ar`gxxIrE@Fyf<qq9Jb;qd(ebs*@huoeeQQK6AFA1LQHG
zZXW3BKnGK!N{Ru+fMVePJp;9;x{CM5RuI<xdbyB)Bs@WQn$Y8<EcYSQ6KEiP7s7@v
z-9<dM>rMKxHyVY$Jr&?rVKI&SMaEx!LL&PyM(E<|{ZEtMD6(Te#&ItEQ^?@^-(<bh
zT=-{^esovaFRmZ&(PNrJ<qKW-uOL11Vi8`A_wwV2=SN-mZ_xR}<%LLPgd?f{Z@KW_
ztMi9txkyGrjq3m2h5s>~KleX@{QDlK{dTJP{nwK8-2VWsm#_nsU+%)czs`RU`7saf
zRR0LFJ4@q<s~N5Hhs#%yaD9MjF7bDbi~6&5{>-0Ax@%=rK2K*y#o9vB^Z4if<GKpd
z-BkXd3;%UGJ^M#WcWnoiZzLqO1HRSyTd0Jw+Jzqn_|%2Jl?;BlfPaI|A1?nyvJ>9L
z<A-ED_|fsn<A?cQMut@v@bBruzk>Wfynw&ih5st@KXL*8@h<#%Ki)^<HHGYk)A7hW
z%+UG6WrawUghf=oknreiInPqDc9+hd`Cm^4k5KtjgzU$-LFaFwf=z^PQTa|6{<~fH
zSCRjI7k*T%)l<2C{;ehh@6&SLX-~+0j6F%u<Hte;HH3qye7Fn$Q7-&z$$u)9XSwh%
z(fRZE-ADen(EeUXc0cLnulsfW-sE?Z@DcKV*+u;yxbVl%D>1!J{y&nPe*B$s;a?}b
z;TeiEl79nXH$p1b`jX82^ZDZ-84RQHkvc#2kJtIb?Oc)S2`9Voqhc+c%6a~U+mRwo
zCbUuc62iB2$KU;==l1h;D}Ju~SEPDBxvp^cmv{+bFTxN)3t<vrHsM^tTL@jXY5zbO
z|1jY(!c&9}zd_QDWiOK35_To*M;JmFML3#pB4H{a$IBzRnD7Q&`2v!c5#CO?l5h>-
z(}XV*ZXw)3SWCE<@Jqt)2)R9sKa!r;2O7zIyd5W*?<JfenUCidfs!{KAFWB|`zYN0
zOR2)PRDK!B+@G!_bH8}L?@IP?TVAAIB>Rzk1<CYyjuuR^fyyIE=KG!&lJT9Hn3#9I
z$ICqNJ9?PkPSRZC{nMFb-cOg4%=@7`$-F;|B=df0M>3C#9whVrx`bpNFFb$nxZ(Rf
zR_YJlhQh?-oDWkT|9FcD6OVuR_(LQ<4tRcG`(9K&j^g(wc{0g;NWPBbz9grT+>hi8
zk^@N2Be_4xMI@8x03W*UV8ueR<9?Ns%>Aw)nfrM!$=uI}N#=e&MKX{3Dw27g<#Er)
z0grn=4tU)2alqrAj{_dJJdSzX@;K&k%j1~GEsx_js69N6dED|iev8U^9KA;}kDJ{j
z^SJq(WF9x)lFY}!A(DBVA0?T``6-h5xb_H=$2}j{{v`8peJRO&T=yZFkE=l>^KlbS
zG9NclBpYb}#FEU%$yk#4xJ@FN$1RUj9=AMBdED|i<#Eg7l*cWPQy#aIDL#+eERvb;
znItpc1tfF*#U%51ol7#0$0a25_+CyjkLNo`=J9+l$=u%6By;~BA(_YT(<Jlw+ek8x
zziN_s{JldmkH3#e=JEF>$$UI~Pck15^(6E0aEfFeKklZc<Hv_&9zX3#=JC^wWImqy
zlFa*OAjv$QgLU#?l6kyDk<9$Wkj(vyCz<<~L^Ahp3d!8RRFb)U*(5XH1tc@yr6e=o
zH<HYJFD9AC^Q|QF{B)nLd@ad5KA+Q-zeF<6r~e?C$L|)M{4U9Se0)GM&zF03^5;7L
zZ%F3(<N(Qh{Qpx|eu8A4Z+<74=LfG~IbL{v@FSV$hmIuke0Vv@JYV!Bnfn_^G9Rzu
zB=h_@jAZ72B*}byjUk!&xte4?-Y1jH{Y@vC`LL19{hdcL_jeh|+~2!N=KfZa%>8|u
zWbW@KlDWUzNap^2Kr;9DQ<AyA-;vDyttXlLdzxhKFOMrmp2v7S!Sb>ad7aMkVx4@O
zPF_Va&zroyVfj&Ad8JN%QYSyJlV2s7=UHAKvAkVZzC|bR)X95v@@FLTyv^$^mcQ4P
zf2)&!BAJgnUZ1n9KMqfjJ<n@=9mDePx^n$_!n;)NAD;L4`iEtMu3Uc}xRlCyp5*H$
zmV4^T_2-R&RL=7%UvIG-rYqN<XNFTb&%=D3#&VplTz_6kpmLtK`TCCK$+~hrukyUk
z*MTgjkv+?N-sO3puNPU)CwrFpyv*kXzOH0>4%xG;KW{Ilaz3x!s*_jgWIoUHd6ch@
zSbl`;`8>+kODsP{<vh>0x?bUV(be@8&yz1wJf0{2Q`cL(PT}iUmj6NRVOhVfd7H|4
zUCh_*EPtRY*RO*<p>kda@%1swU+c>C>!!n0&g&+=o@V)HUAcapb(YHcJfL4!d(wSY
zE^noi{YmEYf_@#|MOWTSCkK+u=N<jJJ&elvJfvU8o2i`Fh5B{8h01x|tY7z!qH<nG
zjVGDUYka?iWmjliF7FST?v|KBvW@lkNIjniQ*M=V#qAPpBv%sFuatT|FPT;}wXY{x
zyG`n~`y}#tvVK)l`#O$G$o3rHLUFh~Mrv0AVa2^N9-qhlX!~sy(!Yw@X}d$pwf9Rj
zJk&IPHI-N2Nq%=ptRt)<KDj+c;+^@gqW-6lzLI3-U)f7O6+iNbR^vg9hwbwl-+S|S
z9}oKTR0Yk0RfHOyU-*2Xx38u7vYzZyE?{3r_8QI0l^3wDCVK<fS6#@S%8g`iAb+0!
z_3clf^Gyn&h5Yz@r?<ZlKYZTEr3(X==aJ0o{iP)HdiNue`FcdN%bH28#p4&3Ao2Q=
zzc0<}KfaFR^<)`+pPJWqTz>rbh60R}QYfE$@<ahr${kYX>&KeeQs(QkgVYed-r{w)
zgD0bWKRu6=1hVJ#KHs0>^?S<KGCu3IwNmDGrd%o8#n+cy-|r2w{kC9P4v$QU{&2nZ
zBwJU@c&A8yfDNSEU;el5W?%6h(t1Lzh2(=Ps_P(SW6P#`W2{}g=hV1N`t$eMHIuZn
zPtwG5ViEPSKXnn(zIH@6p-(7ZBK&JxekpJ%*{x)Ilhkh{IpuEYXUMo)*eC3`OW+AA
zH&nC|{pfg9*4LLXf-sJ75@9BxvB%A_ev(%bR(nwWByT6&LwJDD*jM^jjF#t@`m1gc
z@%kQ<?XZlI`Z$s&5hk>f@wM($Kj~Kzt|Qz?xSenhp{+*x8FFNO)s>Q;jz3fX2_p#O
z2qzI{5|$D!C9Jql##u-5M#Alcdk8Cs+$Q>Is*?NB>layXN5Yh6?iA$_B%5Z^{v)0;
zNnc91l+chW$H_X96ZXnD+ezL-c!2N(q1SO)zp=NhC#B^ISwHFH2rD1|Lg+I|w%KL-
zmy*1aa2=sx^CDrto#Z`)rU#{;ZQ$*q-0Os_zkac-ziKMgPkN)99Iul|&Lk`)TuQi-
za2?@B!tI262oDgRAoTiG)^DTz(wF22!s^wsU6V-8BrGK~(Q&wv<aLA_3AYpOA*?(k
z`+tIDuamU@3HuU85ZcGd<6siWmAA|NP)hPrLXD1>btG>j+)lWMuzI*0e<w&bQT%1S
z<@{snCi~0t*nbM~yvgUH|5TlSmwIe#<*LGOVDl|TwYye+r@Q!dG5Ai+`_leCKH2Ns
zq`mtqY;XKn+OO5wu_*1|hhNN*XX{$B4|cK0lJ@qm+1~JpjQ{X?+jH+xwy*k9+Q00}
zEoh1|ep2@q)E?VmY5$sIanhW9mjP^VIwI})oiI(U;rm0NaMwOHv%QVpg>%Ds;}5xt
z?d$0sIJ}l>KK|@Pw%3l!_WbI?LAK|^REIsZ4%1ZWrVmf;pO(w^DQ?pK#q;j3kOgdS
z^pf^Hd2yzzLAK|`d)YogeECVf6GUe!L}}lBjl<qw#?Lr!d&*y6`|3-i{dJDTW7Aq>
z{8c;I-gud`_va72nwk?o?T)d%DNx#9D?w+?<Dp#x+gF&R;S9dnuJe%X>DkqT#-|o7
z?Q_mMeohZ$ds~dO-+tcx`{rP_x5P>NdAfG6DBE+^!uHi<KZ6Z)DDBr++1@lt#y@-B
z_MDyWupce$H=K9>&a<(7{a9%~N7oJ(WqUT3vAu1av@d4^9ZLJv3)x;9FYR|p&{^~T
z^|_7h>#mlDeta>d^N{iT!Yd5q0aBkQ?PJfoe>2}=dkbCL@wJXFytX(@zPk$Fp0ggt
zyl+de8~~rZ)KQP?_){V9bRVCSuJ!ERaR}sFc(e?H4|rO5dtT`aA%20I=Ph=1O2QKF
zmRDejn>T(@y@h+r?r>q&!p-X@kUa1=UOoDBnu2!jo+IHB&f`j`)}vq7R8V<(c;>**
zg?ju7mx3PsMLixXAeP6iT`W-S(HD*~j}@0;-pjk?D*)c!@50|!)$j+ZSOQhFLWRfs
zK$pi`ov@{?eL%a<^SJi{pw8#v0Z`3}mbXEH=Cc5-JWmU6Zyy8n+VhMM3_kN=e)K#m
zgw8&APW1d;2u7a-=#6KC5CVL?;ZoF7a}$!u_kNVzq!j798XD{AE+vaEjyq2eDOr81
z;4^ejPbnq%dSHAnDNXkM9UJBC*6|(iNb%i(ey!cw<8QuY`#y(l^mP->xA|saBMnk2
z_U#I!dA4(Fvj%!r=KCJ{b#!ZowAy{Y2BJMXx%p><w9t1K*5298dk@$w@-2a(=h@ZG
z{{TqKe2o~TkDLE9U|Heo5BC&22f4LEs#p1TL@7{8mA>6j3X;-#-`*&hq*UcQ0HqMO
z)}z30lkXEfK?;@atoD5oTOID^{Ug-3P4n=Fz2nug%eP?JrY{Ty&-?vaPl2PkvuC9@
zQsDDWZ&0ogbZ9=Y7<jE<!`tV+t3g;N$S}Z2xCgAgy)lp<jsfhCKMqLLt^h%3&#?9q
z&5#4Dz_yoQbFOsjh6QauLuM=_tOZZcXeoGme+W{m02sdQK7}3B^5!nMN7{wJXlS`i
z=7#~vEz4zo4j6i0>U$eF`2|8gAGS-hR%z(p2Ofa%zU)#|wJU=^-j@rzcD+&GUFhMk
zGQ(j7hf*IH{-WU<z*2OlJ9MWSgqEmj-@@SWhChsTBM7ZJUs3=SdPD<_t-7>di1|~n
zKU-be{vOC{Ho5*!nJ?Gh)!if-!zCJKOXT|l=z^&%jPz2}YRRwqa;dK(q+-oRvYoJ=
z%ezxOgcYQ(<Z?m-VI9@OaW90m97ktZr_&orwh)>K6A0@svR)JQ!$_#B_8c+cBJIv`
z)F~`~Q5;>^|C*S>`-8@(k-jh4O5-Pynl{a*8;`bGa$H&BB;vVl5o~Z+i1!fM3hBC^
zaB-28F={sp{%t<-eZESrkK5N*)}z(S>*auAlA8~s@zP{+sI=P@($wDnjCAzROK&zp
z)ySsyXar}Bi%to4*<Wr4^Htr#tI6*hU35U5A>S|N_eR1@)Ff3ff28HM!|$2fX5o)^
zhs%PX?1H?EAY0*_xzp#&2r4Q}3$hgy6=%UOwFedF+29Ar;U~>=bAw8X3WMOM)KhbV
zY*|IKa)Tg@rVk|oVo4DEG<b4E82wIoR(=U8ZN=H}!{zB%waK}q^`DFTMD^6(>=nNY
z*<dFA>Aoq?m+ZP~NYnjOMt<41N&m@&CrRI#<N(4|gzE`65k?YP30IN-dctLd+>Q%j
z8O71tEz{{YQQUomHG~Is^&Qmt>Em|3UH0Fba39&%5T4ZeN0OaB&L5Qn{-QX4wBCQ0
zy*Ke>q48--l=tOY!8>`IOBK5D$j7^JlpLq9F4ew+gAiv|SRH5;$K<%L_*vqnQxY-u
z`R2!pP}z=Zu8-Ph|5)yKt>Q~@{9Nq(SQaVUw{LLM<H;S?TbO=pUn?`ZzxC&t9|OSF
z#ig?&imj#W2hWfE`^N#EH;VQ<4F6=N@_`=0GleSPdN^Da^G6Ej`O$f%YIZ&oQ=hXH
zN7b}u?NK=wXTD%~S(W@<yn3p?{tKG_hD-i=o^#B96nYtzb4~15`=#`+CggHE$rY^s
zO4=C+jf9m{UPWjiG_oHf2fhH-xv;O-=}jaj5LyURbaof0PD`PFBoJB%O@w-Xy9>SE
z?n1KdFN$-adj3=8Hr@|3J}a)2<FuuV^`Koh9(lg2w>HjqsnB>{4{AMSJUijjy&8WX
z@_h5%O%c+ritD5HRn^gaS@-%Mn(wlsWczA{H9z0=)taC0OfgOEali1KaWVU7SwGni
zo`?SG`L5!1Ip5i6AqW@vO;RQEM_O*X)667Fnw`IJzN7i8*>Y5-;rtc>6aPH1KWeEy
z`)l&|JL_{Ke>|^o|GAOby7LM9Szed^Cc;{dN2rnBMrbFjAT$sf2`kx;Dz4DU+#Xk$
zKymctDLQ?XE^f6>X8&rPAL|+GsJxc2im=j^-#`Af-%I6s>s`fXgTEQp5&sq%pH%~I
z663VB#^1zg+MyKPc;xx5dW;;$Mlk;wj<p6rxIL!Tw3Fk$=2D5dS4e~+v7Qqz1A;#&
zJEv!t$@NYJ*GKKEyOicf`~81tezT5{?OPw+biU&C+`|Srzv<hPa)J5nLhHGzS&jY0
z{_*k6dgD?#E;&vGkwD8#2PLfM`H2gRrNc$G4(~2Q(tnn0XZ~cdV{UeunM6sm^B2x<
zJb&q~H@P1a7G9i5`=9RL>C1Wj#(JLj*#AQ1uI5KqdRO(h((^dh_d_3F-|j!_$Nj!o
z-0LInk6e<IBc3?B#aqhvkiO*79PzP=D}7B%Y5yMCx7WqlO8SmE{U*}ibH_?Cv^f4V
zq`zCIUqkxhE=}8WvHfH_kF?g}_jym0JgdKj!zO<S`_prB%-`n^%llAA37J1v`kiFg
zo#HTmY<~&qx!<h!COz}S`m+>o3Gv7JKh!BFV*_3PazD7JE9pyVe6T)%^o6vaSRY3E
zBAwnsdhQ?F$CIA<VErW0-$DIjeJbhi)ai3cze=Z{P5LDtq>1s(arFAHL*#X{o(I<J
z+sS%853JY6KkOnt>z7bFc|UU;Ztow(XT#-ruKK%vO-==`kUr$y@8E-##<ZREkvjcH
zq~D^8^Cjs+-)kD@FzLf|`jc$`ep7o-x(*4`>D!S$T&M3&`rz88aR!n;M5iCZ_FJ0T
z$B}-Au3v?u=W%*JwL6pae0;Ee8tHlbvi@q)^ZsOgBjbN8H|J)#p5gl0dkN|F$LAfS
z=lP25A0WN{xP6@T`s4OR(lZ|%=S|XczgfS7^d0Ct!1|9!&*ul$e?j_AI{WWQ-&v<W
zO8PE3{b|xy=#GCcS_kUqk+!7Q&m&z)f0r&!KhpF5<o1V<p3iTrk0Sj_T^uXv@73v(
zNUxv2t|$FMoqZ1J7wPoHq+hJl&nG>fFSwnzkbbGo{vOibtkYL=9Nm1X*H5ASp&y5=
z=lU-e4;+;DVfEwx80q!n{0!;!<J0SqjH5pvTx>g(onk;SpcqgLC<YV*iUDTejaW_G
zZ(x}}k1(dn`w;xOLDh6A^L@7URL<YW^P~F~j_)5*Imf+N{B2Lv3!dYx+RvHr6d<HE
z4ROX{-oN7`xFRj<E8ENMV!bnd=Ha20_m_6uUi}BqHFSTI%hM?UJWuGjSBR?la2KWp
zY^cX%36OD7#v4*@{x*1E&_na-xo>#G>E+_<kIomaq8#iR23tBboc<^Xo*fm*PIq?@
zE!CuK28xO@LWEjALj&2=(z5X2kmO)3Cp;29UxAbpL7D8-qS+a_x$v}LMnQhEEjvdP
zpZCv$pULkZ>Wyk|qWi@ua=$yCOJu|24f1mxZ|PrEA?1p0Qa1FHXdEN4zBB3Xl*nQ9
zm@=>FeQG%hfMPPNzM*Nko!i+MNPnxiu>^8A{37co`kR!UPcUBQ*5p0^U~8v)=JV0d
zRlF5cACDidLw~;=_1N$y$iAA8?HSMcz<~UAll^z+`dw5zPo3*0syr{5VBvTCoc(!G
z^$j4ui{-^a_6ZlLk9k7Gwr7&P^-u1<GV;5ab~@Kx`6vBq{-j_1pX}dk;zz%qd3^Hx
z$bDNu_74y;FYM2LY>$X#FOdCKogdq|^81SHPy7Ktj)`bEA;;&%wlje2J9m}GJNt3W
z=Gw_+h-O`VTy`P9MDk;MMpyGnhR)tj=SB9z&mb{<O*@9q>-AsD`v`oVuK7mFd_Ju|
zAZ6a~)!#}P9?%!5HxYu*tYPBs?K|3;BIjr3ZM`(p;IVv>_~xxRQG#aQPclC1wI8I+
z^Zh;w%s0WgzBBZ_|H)KvmSjgiNN!K}P53<l{iq#(HCX8RXa2svtN7VB^=!7tworb5
zo4;pWyFkkPy>b2?wUO*Bi)6Vz7=JFu`{zQ>{qb?DkE3M8fMP%~pcqgLC<YV*e-{Q^
zt!G^6_5E^X$LlG!e|_3;ai5Ci#pKBAV8{KFI=P-*f6P;S&nx?Sxjs$_>LcGLMbE|R
zpRY36q#eg+JV^fg2+Igp5w0ifoGkqV2qOtMQF#p^$7SU9v3wys=^~Cd)psHPf3<uv
z)nj#0UzrR2AFVh0FN*U=>-~4xFQfftq4CM}IIhE3aRICzE5{jLFT%H^#eFcmcH2kS
zl_v=!={gd_@jLM_uTOlv=Fn67_`1gE;(n#W{@<8!H8(U@rG39V?%9W*zdV@I^tssP
zd@n|4FO}DkcD^va7^nXSHvGOUet((e8u@%2-_PLvp;dMf_j6gc(1XbQys?G2s(PuZ
zoiRnmv!qJYQ@V?u-(Wq*XMEu|@fa%?Ul94cQCHCX`C#Y;Bm6l7OQpPnv#I^1`upA@
zU?uUQRZD)VHb^|DHcRh|`hN2!-TkX`;*f%Q<oTfTC;7R9!(KO^Y4!5FtI>NEQp)6e
z81<iLa6FE4u(T=P73621BmHfJY;PplL|8>wO;}54Axt2wqPW$B9PeVWg5snQvR@_1
z^@Oz)zn-v;>!b2Y*6ZSMJVwJ@*?*0Z3kbPBR-0Vdarwof@h^&VvGx7C?F_`Tg~n%P
zzI;xd*O~gyC+l?M(b2wGIZpBZEU!IruIv1>9QQ`TWmFFyKlqv>SNEe&&~xlyA`|&7
zTOs%RPru0bhhFHuBHwSWq4zRe$iDtY-SKvw@1H;)y!rkxH&c(D>Aea(U-0)EYzrFS
zJE8aaBXZR|c}7XVFLFC2{XFvjO?FVh@3A3;V?Q_yC_|qwSJCIXDfAqHV}H?ob^Uz3
zO!qmkJ}k?XljOLnBCIClay?d4{k7Z<oq?l$q_?@S)4BcE#CmF9-M_h=wHM&^zed2H
zthkQ)W1;cMVLZfDj;2M@_~vnKcQGF8X@17J*?E4p(ep87g!7!9m*LN_xK3x}=WHB0
zYTs4)@;MDgo^NaBG@V!3kLTy?xw71q9ot_FG8Y$P`=7Saw~gl;Zil{{^|?9uC1rz3
z)8Wg^;(ZMkdOn4pOJSMM+lf@JmznPvD(BB<S?25RNmQ<vxjn_XMf5ry1}6;|Rgixj
zp^46`cHQ}v<FVc0e>Q!NBc)Et{Jav|UkJHAeZ1KekKbQ^p*UR5?bgS;lj5E4Sm7AW
zp+4R!iuXq^r<;8fl2e7oo0=Yf&DjSz+4KGjJR<qOiTtw(;q`PP;dphm;gy@KdK~v9
zWf<+xzwb%5n)^45+Dl=z&hurz`TEpVTTw6C`={SK<;))R`_IRLFa0)o6&80zp<+NW
zpcqgLC<YV*iUGxdVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r6a$I@#eiZ!F`yVw3@8Q^
z1BwB~fMP%~pcqgLC<YV*iUGxdVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r6a$I@#eiZ!
zF`yVw3@8Q^1BwB~fMP%~pcqgLC<YV*iUGxdVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r
z6a$I@#eiZ!F`yVw3@8Q^1BwB~fMP%~pcqgLC<YV*iUGxdVn8vV7*Gr-1{4E|0mXn~
zKrx^gPz)#r6a$I@#eiZ!F`yVw3@8Q^1BwB~fMP%~pcqgLC<YV*iUGxdVn8vV7*Gr-
z1{4E|0mXn~Krx^gPz)#r6a$I@#eiZ!F`yVw3@8Q^1BwB~fMP%~pcqgLC<YV*iUGxd
zVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r6a$I@#eiZ!F`yVw3@8Q^1BwB~fMP%~pcqgL
zC<YV*iUGxdVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r6a$I@#eiZ!F`yVw3@8Q^1BwB~
zfMP%~pcqgLC<YV*iUGxdVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r6a$I@#eiZ!F`yVw
z3@8Q^1BwB~fMP%~pcqgLC<YV*iUGxdVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r6a$I@
z#eiZ!F`yVw3@8Q^1BwB~fMP%~pcqgLC<YV*iUGxdVn8vV7*Gr-1{4E|0mXn~Krx^g
zPz)#r6a$I@#eiZ!F`yVw3@8Q^1BwB~fMP%~pcqgLC<YV*iUGxdVn8vV7*Gr-1{4E|
z0mXn~Krx^gPz)#r6a$I@#eiZ!F`yVw3@8Q^1BwB~fMP%~pcqgLC<YV*iUGxdVn8vV
z7*Gr-1{4E|0mXn~Krx^gPz)#r6a$I@#eiZ!F`yVw3@8Q^1BwB~fMP%~pcqgLC<YV*
ziUGxdVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r6a$I@#eiZ!F`yVw3@8Q^1BwB~fMP%~
zpcqgLC<YV*iUGxdVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r6a$I@#eiZ!F`yVw3@8Tv
zYYZ4i_Zj)cL`}2*($L}Td`&YPlbC&6qC2(B-CJf3jB7Zy9qb&bO6C?n-4wap6EYZI
zt29bgwM5(V5+ghHkormqEhISp&V$@+*z#eU3R}xzRDX&aNSYQP{%TrPgG?kk^jyyU
zIAdSJ?YE<CTXrC1K3IRpBTf6;kL)UjO5TD<UVgw=#O-Y9JrY#*1kz9VRr)i(Q*OSI
z>lYR)@@0AD(T4fhciF4P{$5_$a18a4{SF^R-W`0^{vc#5L%ro^xxYkE_+zB~=S7NV
z+SZDkWmb=$(Vp9nN{qqnuRBWno%T2Tal0N}Ai|vUr}h<686>8VxbL0q$(UXD)bMb`
zt{%ImAaXkUq}TPBe=7N<o~(0)*ly-CAwu&15b@4F%qPpd-x)z~$Fk1E59Xbdlbukv
zqeAAZEBP4+{awT{(fD9{_P6)dI^uC=LDC`3kByIWcVl*OUkn#xAfcBmw@A<`UZ4zs
zMT^v0!ryLMS7D3Cn5%T|acLB$SR;;&RnG_!P)4D{s9_(q&mxPpx-)WIv?e*7u$5nT
zF`l@**N%l^d>e>J8b+E2*yC11UDaa@#WPmjApDKLHyzJZ>;U`8=F5#EdyDz(>hFrv
zdF(r!Ak&lL+nVvm@nvTM0MQSdi(AU>S4F!lkjMRp-9vj$*thnu-AQpRoxc?J^~4+6
zIosFvZ#oZPIi}9AnK08J`)45@>b5i-!~V3yGDs=nFU*P8Yl43_XeIvRdC>lAQ+{Z7
zXtrVUxS`#vJ-F?1;m`ByP>QP`Q+3piw%et>jpAd!G4b)pevFRemFi`g>(@+jKB8fv
zCEqURLu0tq%jk`NCwKos@R~RKm~iQQS69*gyNLG+;=6+PFEubYOcUoftXJ&kQ%(7)
ziD_CNj~#Bu`XQ3H@t-|}rzCB1sI(9GO~$E=BY)}-xA@2%vOml}_n-S~FiSt?gZp1U
zT<Y!6PP9eDamVB4$o^TPidpx{{#V^7G53mjf*<GciT$mhb~^e`G6unNSN-R9)RO(y
zp>w*TkE8#mWxJMLp#MBh*pB;;u`oGg;*a}<deMKXpZme}3uos)%s(OXU-Q1?CFKsu
zQ*1(k*d<TW_^Dqm^^1!n4!%<&hQ|79sh`YWHOW|x>l;`6p)c}_h$*A(#}6QK`?H<q
z2h{WNAoB2s+t2>2cPUf9xnEnqY!C+ux0n0D^>aK6%{Q&Dk@E>ogIMNA2Fwq)FNN&c
z-`H@c7<VkYLVFK6uDSgw56Jo2uF3Oa-NRDm`2*YU%%3i91Gfw2OmSZ2_Ur7lYpFu6
zkI}eV*2nfdUpvk}0rI#{=_%3P9j9h@ttFlJ9_qJPw$J{|X^i1c_ALpui$sS#H{?Hy
zVqc%HN<4m@H{-|he8_k{-~VKp{(3=QcQ7gVJjvsL_Y?1Dp0D-(|1G(TF+jt$)Y^XX
z{LAflP<I@%{!inB{dT;eU4}~;k*ZqB^&RuXXs6>%BYQqx*`C}rV*(vFy5rlI3>Kux
z{%~CVyfC+aH*tRFc#Jp}(U$E|pNM~4qy#rPKWI;Nal~`>)w}(#?SFmy{w^}F|0{Yo
zT_5qjx~^N({o8qR)A60)F87<Aa2er2I1%G^H_49?zDQ^*oHKX&oEch~7Hl$^%)wf5
zo-HULF|#DQU|hne_@Ka|z=TmrSH}lNgoTBKYiJrYJtseCdTLR&rezOG&ndE{7N=!r
z6iPj6RZ<N6&oiK#zw5o^@nh&p$5Bg(SF!9PW$st<cdf+rb0U?GA@nykwPX8pP*;EG
z0U0j<4Dq^_+hd}3@%iJNIHc%c@sal@b?tOz|A+Ou_MUo8wwL+3^9?C;yIF519)G+`
zmiwQPHQ!Q{E;4*>`DuZir*N)vPS%IJNRjTB)zS&^{WdkpSWs;4a2euGl5eq%5VxZA
zGTJnkg0c(pGJ<Fp3@R#23$hgy6=xM@6bS*RI$Kt9dP$zGId7#?3@8Tv&J3u7`|rH>
zRNMcq45Z%qKeiswt#5e!QW7QCbJhM$*L6Af$oBuc)(?FDoA>X9?tk;=u$S(T>p=FO
zH|1AxiGCq}tOxI#SG1M=<oo9<uIwlLS3WH7^Ont#=<5D?E&1~@u9{?y@9O?9-#^Fu
z6`00+`$0J(-+#?+C)>sM#rgg!+F;`QzwFO;yuOrHPWM6R{xA0n%h8tG%l+W`*<YJ4
z?~il;`Tj4L8}F3m+#fq#7jXGAm&<<CoRHV=T&}-=yU8f;15}cIEya0<{4V4_{VV#s
zyqCPM#O*S6l=budU;bR4$1yL3`%r`5)zw$u`>rPUul4<Be)#?`_xq1xWVO6+`bYi#
zt@d=U|9o}e_;cR)zAxkXeE%I~3%mU;4#nOp<#Bs*_PNLJzf!Nd|M#!3r)jC=&Yv&y
ze8tCI#IJIm;Nze5-Zf3{|8Xn$b8o(XyBdyPE<>z6DnEa^6!H<y1-d*Rzx;W(u2`#}
zhVthFJddy)-owBY86f8yex9hJla%%Le4m@k>xS5!=NZT6E;!#ng3p#@D#|D<%}LA9
z2#uoBbN{e;LupP>Mp;^JNqUC3s84oeiZhCelZ(>$*yZylAGa5C{&cRE{r9wzEMKg@
zp^M~)&%eI!9u`awpyStgfzPiK$e+)@7Lu`Tcz(aw^Y7rT$L69BpZ^Y096qo9Rp(#s
z7uJOx<@RzvxPB~eF5!-K)>r$>=SBGZTi;Fc=}$(MtL6F7*$4Cb<L{AITZ{W?Y-gkU
zdUuk3{o9SlpLlLB!e=DgI>%$(aLVZCOG6LYK7L-rNHVquU8*L?_Hz5~8!Oj$d!at8
z*OVj6InGC<m-g^92YetP60L9V8z=pwvhlB->Rsn9`~4^($1T~}*kAPG_Ha4NDPQyx
z?Rut<tfzLeY+n_{KiOOAnZJNV#50wrdsF-9^QXGjG)p}o&s%kKCGz~{%qQ}ZK<gwN
z|M+>h#cBO@G0$&QkiBWs!Zz4X9)AZ(|5uOygVZlvzl!ls^>9D9eryXSnsyz}KRAzn
zEc0H|bp5IKUv<0WkDuRg_D6nr{br-*4}PU_&!4N}{b@}6Ii!{(_pg5a%InX1+W*|H
zN{Ywp&y$Z!e;F12@&}}(aFpxx?5{Zw0PsmAg82MZJyFJI-uktq^((Faxn910U_16_
zne7}h`5Wl@7u1U~TCW>;-R{%$b35Z;dELnI#*a0^EvU5A;?&%NtiY1|krRxmX}JT9
zg$1*d^Nn$d2>k)UE+8p3J~k@=ijCtFV`3AHBd#?TvS(^xVd|XZ)U4FJjDgY;B0*$6
ze=%~s7s5MAR*3VjWv5(+8@$Pb9$36@v*Y<-QSkKGrm~h2Am=BYpH`0}uA#w=_28WV
zn`7O=)lfOFFK}pLs-*UCyQCIAFG+4%znf8(;QD1IZ2I`#3G#Us6WMWl>|onmd%R=F
zUe_*i*EWoh^J5tJW2);e=XEK;U-HBDG#8)Cv8`Cn@m_;GCTOR4ZpTD%xZFeoGJ(o*
z{n7&+$PRZ`bM;-QKMNoP`eD50lJoN;$;}z)c;fkk4_7MIPSSHsDdQx6@f3^8d7hd3
zo;;2?&OPm<J;@XJ^D!>x=RfKw-h*_0DNE`r#wFKlp>hO1M?-cB(i#X4@#4A^*UFsu
zxht-Z+owmK7ui1H8aaNd$MqNWAFY)AVR`OnQsy}N_$+f=y^Q+`leJIxpPPIB)dbJm
z<N47ptQx-uK*L8o;~Y$7&4Kg&o(cvPyX3sEbD6J*%k}f|`Yi?F`1<}EsJyb}K;i%D
zA$j~7>3s1$4ZNy1t`OxPJ}c)<j=OV|x6r>!aqP72;M2r<l2fRC=!ehi==o(nE6Zg+
zSgz?UkGJYx61g8-&itbu>qLJX+h4<mx>m3)h0O-r0@$z&_X~bli5uGBhU*kWj)z%H
zPfwBSpk?$NJok%vOnisdNzz<<|5`7>2Z>tsB)Pv?*7`J^kFZX0z9$1d|FM0Y&Yty*
zhS~D|)Xk&rb-qp$&q3mKhJpN7&86pA*inagE^t=ar?(dK9LIMq`y+Ygi}_=ID-X&0
zVjRo%iyT*#`I0~W-T{|0-aFtXM?a0Abi{=`)>Sd4DgS4#UEAs5?OR^EP~0=u_IznY
z?!qHyuKoOpS&tq+0-?lp8lrOw{a&`a?37VFmsPd=cSn1i?KrQm_Z{kA8I7l|WXL@6
zeeO+kzk3T||KW1n#1L})u_R9-%p$yja24TWgu4j8Cd`SEaf=8S5#CPtEaB^fonoc^
zK*F_z&k~*@93Cg_#uCmQCFNTQA0vE@@HN7%gzjUdeHX%^gfj^5Alyc{m+&-UyK&Ng
zI^lA{M+vtPenNPZu>E-H7e{yvp^b16;d(;e4?Q@OLdOls4X2lP!Yn_ZuhE3tmdeu!
zasI(%srMA~2cPdFrCN+_%=(b*SVz^#@yF+BDi-q|=6U?FA3xW@_jy<r7g=D>vL6Lv
zS#*MQ2=p|FHmASjt2>o#AYS-7znsdQT$^T((FO6EhB)Ie@81WOMMfk?@anCpsge`}
ziUGxdVn8vV7*Gr-1{4E|0mXn~Krx^gPz)#r6a$KZzYzocu&em$Ytt`%!Pj4tbMlMe
z<M`a5lA^+(+=8^!T=)uD(X3qYy;uIe$e=;_1%ry_<fmza24&{tW<;GQCVs2T>5F7R
zSO<J#Hf^TJi3iW(=jtC!qvz4GGxAacp>n8(o?9DK1ipExQTUy-WVF)w8*AD%qsPUJ
zza}bDi;o^RDk>vCd16xF#0hbOBDI83G0_vEM?@#ZMmemLlB~%i<44Din-CRj3XBX4
z2@ExcM}$R~j3FV~#H857DE@M=upB>bT<pjRqsNcaTZ1$H3my`Rj!D+o`1q*C?+C{y
zC662*KRz)js<0@^WKPeCG6fF}4KwHFi%f($FFOjpcx=mw3NsB24K)`P2z|IYoio$Z
zWzkSmusO{sUr;FY!KlxJuqJa+NmS_Ip<%)1qO7P;)6lR`b6Tp%1e=Rzi%f{Q05V}i
z!$QnO#lkquoKYrf3^893Se#lY^1+zTn(N4?7N$AnXE^f3sf{+d*{KeF?%Xh^JaiDc
zj4^ZP!j5C1IJeQcxXdX(SN2>qVsN8=ZjR02+o+x5(4{vjotmCo*r;}@FxROPr@E)l
z4ROSq-so7|G~Z}b+-Q@YG2P)>Xv=ftXE<>>+bN&nlrM717dqurom!llmf?swx1f=t
zG+R!i;6#^Gztd71n>NFiCBlTm?wDcAXcV$?G93le^CZ=1WScJgBJx>IdFmtTi?ifz
z#eA8QeVMb3Q~ctb%*M)W`HhVzDs7UN>|)eBCnH;$9I--QQ0i!Po>SNIoV%76;$$?l
zF+V%QsjUT0eao}S@hfTwYjgm8qflBPSw*8+#g10wmo%Ep&TGu47BzOcu#v{Xywb)n
znYr>f!;q<^a;m_5X0CI-F=ke-quTVwY*|`kO=Yvw9VT;}T9)S2+l;(M`+}^-=mlAg
zJg25Y%V2_vz?PLbvYEL~`n)h01!R+#1NAoMae9)@c{81iY@v;MTVn+USuhI7zOd1J
zb}F=u)CD-=IBzR~sg;Bx8#J2oWkojVnGm$Gaq-cklA;F9*5b#H8W%klP6YJohY7KX
zW21`F3NteDwaGD~k`pJ6n=pE8Y;w%##HgUcl6*K1W)ueD>mjuGk;!9YlaiuG#U@3K
zgp*2gV(h3yQ2ddzUtnO6{Ew$aJbwihX5ja&i|~uql+|KJCndy3U(0PY<2go4h)s+S
zj~_kaU#rcQQJ8DK__MV(Hu|bq$sJG^lMMAIMkijIoDe<1dcKC7Zy}MJGhHL*l9Q|#
z(W53b5l4YV1=@t@gyc!FiAmzD9IE{*ZGr`gS+00;rU#{^rNScPB4x2cDr)o>vO{bv
z5GpA;O7VHcBC1FDIV+$cGfMM<W~7#;4vHL9PzX;xX5hl2v0djDr%%6-#f(A`L=5#L
z7`wn-^5~eT;9wZf(FqCS&@h=ol4+$EXv@`-<$4C*ZGwX#K6X-Ud{i*3bmAw)M}=s3
z{EUoFh)NncPaZ|{CQUHH;Sv-*E@`xJawxny#aNnCSX`2tYb?$v%*)A7%{5PJ-O3y@
zK5AO*)IRAW%<-`@7h_HwEpsEy6BA@E+C06;JiQo&DZ&lbK#(6}hVvodRpzu@M61;0
zVZKjJpY)-9(yit+;TD=D{9r+K{TOq;@E&Ql6`NsEGc7jK4C}IKv1#Talm_FnXj-h*
zTw((lEBs(_d%e{>JvZw5QRcM5sO!g?bBcRKnzIYQVye}gKN~RCT$~zp{b+Lz#_Dg9
zemT9uBF$+9y@H2?m{ar2W%=f^QgeKQxu{UYP8IErH%G@~pg!sGvUeDPdr)YGX&uC|
z<`^;>Yfi^@PE9b!jFfH(!YxO1JS^Lsk%Q{6Tv!sCN0W)gJX1uyGB|r$Y?QE{DagAr
zICom?a5J1~z~{=4@LY4L(1(VBAu6)XrNT250^%+!mX^V$up!yzVu}Gmt{J9atT)eG
zBD5ipgQYB}Li2=YkyL|}HyzdCkP*8g7&6(Y34shUGj*`^$P*r*ED`}m769qC)Ls!`
z=IA&PxKDbXIeG+U!!TQbyKHLckfC8iKtEYH!MSx>Y?gVH$()Mq3joBZ0V6~yoV%t4
zhna9oP0h^A%9=4_=FHsOyuAGUf&!au)~v$9qN3vBl9JNW*|W>a=FFKp_l6th&Aai&
zn{EmW95iUafPn+Evqz7Hl{a!@6WtBZF~dm*+mvIzO6-)e=9wl`1e<O$&kTm#_yqIJ
z5X7s6-^@_VCz@x5Atsq;h67Fw4hQQ9ksA!T!6FxVlk|ub(h$%O!CVMN9x8IdH<{zG
zj;SG`ki%N0hJ;D4IIL%C@X(M1a~xKM*{jWQvbLciiRL(2-O!LEQA?bxGkj<WM3OZQ
z7I{%;aHPmbi$Fs#j~v8_-4S9EWn%P&KrP@OBU%zF^06WxhWVM(qvi|8s{|pj=9wbE
z6!T26i@|q<d1fLtHrjlJXt#)hwOt`wI5Z^A+)H#7?5*ZrqMsocz8Cj1#@q+?b?np#
z)S_!2F}OtE`iL<Nt|QGe#XtlOErLVH%(S8#F=!F>fiQEN2nHE3N#U*<jT-Ec=qZpt
zvjjBbqeX2qaioujt`s6dJ$O`)k1@~8GsCQF9iKEI3a^EoUOc76!_+)7emq>Tin9_-
z`$4&|NU-JDGK_tS;J=n|P0aaDz`Cn$ZA?gFtXR^B%lc-|EpTdp**s~2FuV8{!&t||
zYO2YK0xC<2x$6A2!MS8KTqu{O76#>|=D<A$T8`lojL%CW$B(~ibZk_nX?l8S=8)-;
zkr5FQrs)w8nZZNS)5B9?NrowPP;mH=h{54uA(4Xv!-otG4UZTU5jJE<WN=t`m<Hbz
z7;$Z~Y@_S0r%xB>wq!h+jh_g+BR!)u$W~kw1m{KE72^|P$3@LEHn%S{Yjol`FmbFz
zLIXp?wWJBxMg<QE4-XE6D1%Ia+&x(BOd6dyAvq>CE_!18gye~E#+#LqS7Htd3M!gY
zRGg8gjgOD{gZq|S5DhmdCdI<J8PC(AFR_WRh;`}<-pZH&O>^i>+L-7`(HG(eMrp$M
z#0gRIEoe>l^`E<c^f$U&rkbi4Pz)#r6a$I@#eiZ!F`yVw3@8Q^1BwB~fMP%~pcqgL
zC<YV*iUGxdVn8vV7*Gr-1{4E|0mZ-t7;yEw))st$7$l4M3%}paFSZf7NO&)w1%8v=
zAl@Cu?{Fg1CU6zKP=}G<1BCA*!*~6p&<lX{Xb>+YgZFq5@_T;J2e&GE5s)5-K^Z1S
zexFbR{LVDye<VLWvOOdFVe{}ELfA*tIo>}6c}(0c{Jm{{#~SL<4=F@MJ$3`%g|r~d
z<QA8qEnr*pziHwZv$S_Et6F;1k^^mSyL{o@We?Bm^2o^iTdvsNw{O+-jJ(M=H+e@H
zxHL&EpcOW5FqL53p78cS1N{(q999UaBW&$JpF%G}8cY0k13lX7^;eL66}>2_1IU>0
zU43Y;*JC)0udjCk=-X;X&*=xgT^H?ZDL&SRDGD}>uh(nI$))K<1<7gf#i*IdSsBH}
zIe8hZB(WefJ#`N2bmeKb5-m3;zocvshJk081L66i(c>mgP7Vo_Z~AqI{nhjzwugVW
zd(E5wY=b|#*%h`JEP#Z4c89+wN}g?jWtgzPnD~8CY|HN*l%<XDw=~jzIR1M>0hXnZ
z%*R$0?LU5>)_N+Z-K$+s0r`Db`1|CT&NRq)0d$<5C7HtE>rh2~epD{K)L+GbV&H#~
z0X+VBs*lHc4CK!|JcJJC9z2Xkjhm=_J9hoPU0-Ft_3$sD55BqM^A){=vwFK-)?z5W
zR}W)MfcXRGsKtQu0DHrP@F-w01m6$14JMxUP_`d143OtFx4DU$TbaS_GT)Zo_!|tp
z!4~Jjld$!IO^fz3EOj5*#_QVF+rcE-Ck8><#b7!3Jcwu|JNG<cX!V&Y42MBcE^K9B
zC(c<SUR)cm*cB~q^;qQTp458lh@B(ei<Z&dtDA_LHV}NDhHV)TA<k{wp3&sLtEmIF
z{d3sXL%+qjP1uiZ;}s{{jpb*kT%4OkxwVZ~iY%XKE&tB{*@OPU*3I7-t*vUU%R2<`
zZD_C#*)rSxx~<mo+UvIZch@ol{kt!QD4KS{bKL=W`?PRdyZ{t2%RS3LBVsHMUxL|<
zZ59Gpmq!%<#4qoV0uX(o+O6cHnB{G*0wH!tP02{CFvkCl&DK_1;`@IHHCaEZi(OJ(
z)~)=Q_1AB#{!iB$p)rr{fk<vAud@!Rwfc|PvNfiBGnoGRiS?s{hxf#m*BoB@wph{x
zS7Sph+QjJc)6k{piP001;#cJDw=Vk8u&BYUc#i*VzOYLcHFy?}T-4y@zvK|KeYs~8
zgkSIGzvL{GL2Wl-ZRH&yH&5gyc@Oox6mliIt&6r9f~%)SPm8|px~)G~Eca|7JWGB7
z+en931bGed+`qe_q4=G{d%?8ZKFFWg>Q?gea?f3mEsx!AUGDia<gDdh%R9UQnfT>>
z4g)Ou&CP#tH)u+5^`hV0{TB~|?D94npk!#byWnrhi}7xC&~s~4!;brh`Y*Mh>4(-q
z+B^3TU9`nbs%Iaux^=Xc$9XT0-M^&Tf5`}VL+t3Z)0_4~hy6XVAJmH_4M<SQx68W?
zhS<<Rph1u_5JcI?ILL`cc7Y5uG7tb7Sv-*%_zc>&X%E^V-Hv^HZUa|C`Gp#|I2jsv
z=)vEyrjmUJXTllYy1Zncb%5u3h`7AN3-H(4%kvQg|L3+GzFnBbG@30CX3M47JTg19
z4FblO9}&i*$oTLs)`YWWRq%(0c7o~86^H%_I)I-*Y8>x4(8XZFfn;6M;J^5F=#v;(
z<)>qocRTYDj&0=bnBcA+WXWA?Y)N^G<sH5N6>xV9K<waGkrcUu7oty_kD(-HNVn$!
zN}hJ!!`AZ6{!6RT9FC6~+`}uDX%QLQtP4)z0GRy)>|(3iCD!t*ykSo-z6C77&id*H
zzb1YA*aO>))?a@>dGpCDt($*qY3;woTAt(`Ty0%)qGX)YKwwggu-GE%2SHnwcNh$R
zf%*Uhp#IR+u%`r}+fPf&!yf}-CxEc!9g@JPyyNl?;}Iu$9A3RCMKtFtf6v2r0m<kK
zGs#5Y5Jvw`A7Z5Mid$IATVA)dqA^X}bc1mF&c8)<1=tq1F5j}?n+CXgEAIsDF817f
z(6`(RkM=Rp23%#&(pt1=5d}gh2!$vVgV3K#vOvI>oQjgCpb<Drc69gfaPxn*9%n7_
z_h<jNPmUTn(rgTvIK3plxWpJ97#3(66j36i;G2RY0!?9o!6E%+fic7s9ASzu4S|=O
z7iAX}7Z#^Z*8;QhO9Hd;cfAAC=j0d7$&*-IC`(E+3X5_I@|y_B>B1&8FX#W-yLO$1
zfgpGd5*;lK5+#*|kW$fvE~4ZI6682aK@wyPqNL+1X!!^#BnpJY_fSwVWAB9MBv%v^
znQiv&ymPlVe%$U!w#!u1t0{hCkc#E1sZvqR$Fmx5_l@UMbf8))n(Dqu#pPAQuPoHJ
zWrIF%ejD%x=r!J{trpc1X60(p7BJ;M3e#uC@`}3;@#vvQ58GO|ezT~&63bF6jO)J;
zs|!B~y!0sS6vhi5&F$L8K7?2PAiW#?aXi`xuRP)aZR}fk<s;I{KVZ^Ibf+pl|DE{H
zq_zLbH>8=DQslqJkv_pOJYUv$<ulTZYbd$@T`R&86d3oY|49_2wa*~FHGk!6=fHAK
z!YkjF9_j#WD8fr$pp6Ty=F_)tK`cdnbT#}31_h2UeBb_s>~s7q!ykoX()-y7li%u(
z-;t*YIi#%jhhm@TB^yw@b=E1}p|I)ycjWuhXEF?(xsx6PuklYKv6jxq%kfWvKhDO{
zzSJOV7k*P`r?^Etgdfj6_^jDgGhhbHfEh3YX21-X0W)9*%)q}fFnWKTVDP=2w@-^1
rFaul00Pg<}-~U^7(mh=>5Deh{A7GEu>%XrEGhhbHfEh3YTg1Q*d=-0a
literal 0
HcmV?d00001
diff --git a/src/bin/pg_dump/kwlookup.c b/src/bin/pg_dump/kwlookup.c
new file mode 120000
index 0000000000..73c3abc1c7
--- /dev/null
+++ b/src/bin/pg_dump/kwlookup.c
@@ -0,0 +1 @@
+../../../src/backend/parser/kwlookup.c
\ No newline at end of file
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 9d0056a569..00d77ac16a 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -72,6 +72,21 @@ typedef struct _connParams
char *override_dbname;
} ConnParams;
+typedef enum
+{
+ COMPR_ALG_DEFAULT = -1,
+ COMPR_ALG_NONE,
+ COMPR_ALG_LIBZ,
+ COMPR_ALG_ZSTD,
+} CompressionAlgorithm;
+
+typedef struct Compress {
+ CompressionAlgorithm alg;
+ int level;
+ bool zstdlong;
+} Compress;
+
+
typedef struct _restoreOptions
{
int createDB; /* Issue commands to create the database */
@@ -125,7 +140,7 @@ typedef struct _restoreOptions
int noDataForFailedTables;
int exit_on_error;
- int compression;
+ Compress compression;
int suppressDumpWarnings; /* Suppress output of WARNING entries
* to stderr */
bool single_txn;
@@ -281,7 +296,7 @@ extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
/* Create a new archive */
extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupDumpWorker);
/* The --list option */
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 1f82c6499b..be2b0882d9 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -70,7 +70,7 @@ typedef struct _parallelReadyList
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupWorkerPtr);
static void _getObjectDescription(PQExpBuffer buf, TocEntry *te);
static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, bool isData);
@@ -98,7 +98,7 @@ static int _discoverArchiveFormat(ArchiveHandle *AH);
static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
-static void SetOutput(ArchiveHandle *AH, const char *filename, int compression);
+static void SetOutput(ArchiveHandle *AH, const char *filename, Compress *compress);
static OutputContext SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, OutputContext savedContext);
@@ -238,7 +238,7 @@ setupRestoreWorker(Archive *AHX)
/* Public */
Archive *
CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupDumpWorker)
{
@@ -253,7 +253,9 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
Archive *
OpenArchive(const char *FileSpec, const ArchiveFormat fmt)
{
- ArchiveHandle *AH = _allocAH(FileSpec, fmt, 0, true, archModeRead, setupRestoreWorker);
+ Compress compress = {0};
+ ArchiveHandle *AH = _allocAH(FileSpec, fmt, &compress, true, archModeRead,
+ setupRestoreWorker);
return (Archive *) AH;
}
@@ -382,7 +384,7 @@ RestoreArchive(Archive *AHX)
* Make sure we won't need (de)compression we haven't got
*/
#ifndef HAVE_LIBZ
- if (AH->compression != 0 && AH->PrintTocDataPtr != NULL)
+ if (AH->compression.alg != COMPR_ALG_NONE && AH->PrintTocDataPtr != NULL)
{
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
@@ -457,8 +459,8 @@ RestoreArchive(Archive *AHX)
* Setup the output file if necessary.
*/
sav = SaveOutput(AH);
- if (ropt->filename || ropt->compression)
- SetOutput(AH, ropt->filename, ropt->compression);
+ if (ropt->filename || ropt->compression.alg != COMPR_ALG_NONE)
+ SetOutput(AH, ropt->filename, &ropt->compression);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -738,7 +740,7 @@ RestoreArchive(Archive *AHX)
*/
AH->stage = STAGE_FINALIZING;
- if (ropt->filename || ropt->compression)
+ if (ropt->filename || ropt->compression.alg != COMPR_ALG_NONE)
RestoreOutput(AH, sav);
if (ropt->useDB)
@@ -1121,10 +1123,11 @@ PrintTOCSummary(Archive *AHX)
OutputContext sav;
const char *fmtName;
char stamp_str[64];
+ Compress nocompression = {0};
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, 0 /* no compression */ );
+ SetOutput(AH, ropt->filename, &nocompression);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1133,7 +1136,7 @@ PrintTOCSummary(Archive *AHX)
ahprintf(AH, ";\n; Archive created at %s\n", stamp_str);
ahprintf(AH, "; dbname: %s\n; TOC Entries: %d\n; Compression: %d\n",
sanitize_line(AH->archdbname, false),
- AH->tocCount, AH->compression);
+ AH->tocCount, AH->compression.alg);
switch (AH->format)
{
@@ -1487,7 +1490,7 @@ archprintf(Archive *AH, const char *fmt,...)
*******************************/
static void
-SetOutput(ArchiveHandle *AH, const char *filename, int compression)
+SetOutput(ArchiveHandle *AH, const char *filename, Compress *compression)
{
int fn;
@@ -1510,12 +1513,12 @@ SetOutput(ArchiveHandle *AH, const char *filename, int compression)
/* If compression explicitly requested, use gzopen */
#ifdef HAVE_LIBZ
- if (compression != 0)
+ if (compression->alg != COMPR_ALG_NONE)
{
char fmode[14];
/* Don't use PG_BINARY_x since this is zlib */
- sprintf(fmode, "wb%d", compression);
+ sprintf(fmode, "wb%d", compression->level);
if (fn >= 0)
AH->OF = gzdopen(dup(fn), fmode);
else
@@ -2259,7 +2262,7 @@ _discoverArchiveFormat(ArchiveHandle *AH)
*/
static ArchiveHandle *
_allocAH(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupWorkerPtr)
{
ArchiveHandle *AH;
@@ -2310,9 +2313,8 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
AH->toc->prev = AH->toc;
AH->mode = mode;
- AH->compression = compression;
+ AH->compression = *compression;
AH->dosync = dosync;
-
memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse));
/* Open stdout with no compression for AH output handle */
@@ -2325,7 +2327,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
* Force stdin/stdout into binary mode if that is what we are using.
*/
#ifdef WIN32
- if ((fmt != archNull || compression != 0) &&
+ if ((fmt != archNull || compression->alg != COMPR_ALG_NONE) &&
(AH->fSpec == NULL || strcmp(AH->fSpec, "") == 0))
{
if (mode == archModeWrite)
@@ -3741,7 +3743,7 @@ WriteHead(ArchiveHandle *AH)
AH->WriteBytePtr(AH, AH->intSize);
AH->WriteBytePtr(AH, AH->offSize);
AH->WriteBytePtr(AH, AH->format);
- WriteInt(AH, AH->compression);
+ WriteInt(AH, AH->compression.alg);
crtm = *localtime(&AH->createDate);
WriteInt(AH, crtm.tm_sec);
WriteInt(AH, crtm.tm_min);
@@ -3816,15 +3818,15 @@ ReadHead(ArchiveHandle *AH)
if (AH->version >= K_VERS_1_2)
{
if (AH->version < K_VERS_1_4)
- AH->compression = AH->ReadBytePtr(AH);
+ AH->compression.alg = AH->ReadBytePtr(AH);
else
- AH->compression = ReadInt(AH);
+ AH->compression.alg = ReadInt(AH);
}
else
- AH->compression = Z_DEFAULT_COMPRESSION;
+ AH->compression.alg = Z_DEFAULT_COMPRESSION;
#ifndef HAVE_LIBZ
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
pg_log_warning("archive is compressed, but this installation does not support compression -- no data will be available");
#endif
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 1a229ebedb..641ba2d043 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -47,8 +47,6 @@
#define GZWRITE(p, s, n, fh) (fwrite(p, s, n, fh) * (s))
#define GZREAD(p, s, n, fh) fread(p, s, n, fh)
#define GZEOF(fh) feof(fh)
-/* this is just the redefinition of a libz constant */
-#define Z_DEFAULT_COMPRESSION (-1)
typedef struct _z_stream
{
@@ -62,7 +60,6 @@ typedef z_stream *z_streamp;
#ifdef HAVE_LIBZSTD
#include <zstd.h>
-#define ZSTD_COMPRESSION -2
#endif /* HAVE_LIBZSTD */
/* Data block types */
@@ -334,13 +331,7 @@ struct _archiveHandle
DumpId *tableDataId; /* TABLE DATA ids, indexed by table dumpId */
struct _tocEntry *currToc; /* Used when dumping data */
- int compression; /*---------
- * Compression requested on open
- * Possible values for compression:
- * -2 ZSTD_COMPRESSION
- * -1 Z_DEFAULT_COMPRESSION
- * 0 COMPRESSION_NONE
- * 1-9 levels for gzip compression */
+ Compress compression; /* Compression requested on open */
bool dosync; /* data requested to be synced on sight */
ArchiveMode mode; /* File mode - r or w */
void *formatData; /* Header data specific to file format */
diff --git a/src/bin/pg_dump/pg_backup_custom.c b/src/bin/pg_dump/pg_backup_custom.c
index 77d402c323..55a887a236 100644
--- a/src/bin/pg_dump/pg_backup_custom.c
+++ b/src/bin/pg_dump/pg_backup_custom.c
@@ -298,7 +298,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te)
_WriteByte(AH, BLK_DATA); /* Block type */
WriteInt(AH, te->dumpId); /* For sanity check */
- ctx->cs = AllocateCompressor(AH->compression, _CustomWriteFunc);
+ ctx->cs = AllocateCompressor(&AH->compression, _CustomWriteFunc);
}
/*
@@ -377,7 +377,7 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid)
WriteInt(AH, oid);
- ctx->cs = AllocateCompressor(AH->compression, _CustomWriteFunc);
+ ctx->cs = AllocateCompressor(&AH->compression, _CustomWriteFunc);
}
/*
@@ -566,7 +566,7 @@ _PrintTocData(ArchiveHandle *AH, TocEntry *te)
static void
_PrintData(ArchiveHandle *AH)
{
- ReadDataFromArchive(AH, AH->compression, _CustomReadFunc);
+ ReadDataFromArchive(AH, _CustomReadFunc);
}
static void
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 6e55cb425e..dfa36d5dc1 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -202,7 +202,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH)
setFilePath(AH, fname, "toc.dat");
- tocFH = cfopen_read(fname, PG_BINARY_R, AH->compression);
+ tocFH = cfopen_read(fname, PG_BINARY_R, &AH->compression);
if (tocFH == NULL)
fatal("could not open input file \"%s\": %m", fname);
@@ -327,7 +327,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te)
setFilePath(AH, fname, tctx->filename);
- ctx->dataFH = cfopen_write(fname, PG_BINARY_W, AH->compression);
+ ctx->dataFH = cfopen_write(fname, PG_BINARY_W, &AH->compression);
if (ctx->dataFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
}
@@ -388,12 +388,12 @@ _PrintFileData(ArchiveHandle *AH, char *filename)
if (!filename)
return;
- cfp = cfopen_read(filename, PG_BINARY_R, AH->compression);
+ cfp = cfopen_read(filename, PG_BINARY_R, &AH->compression);
if (!cfp)
fatal("could not open input file \"%s\": %m", filename);
- buf = pg_malloc(ZLIB_OUT_SIZE);
+ buf = pg_malloc(ZLIB_OUT_SIZE); // XXX
buflen = ZLIB_OUT_SIZE;
while ((cnt = cfread(buf, buflen, cfp)))
@@ -435,12 +435,13 @@ _LoadBlobs(ArchiveHandle *AH)
lclContext *ctx = (lclContext *) AH->formatData;
char fname[MAXPGPATH];
char line[MAXPGPATH];
+ Compress nocompression = {0};
StartRestoreBlobs(AH);
setFilePath(AH, fname, "blobs.toc");
- ctx->blobsTocFH = cfopen_read(fname, PG_BINARY_R, AH->compression);
+ ctx->blobsTocFH = cfopen_read(fname, PG_BINARY_R, &nocompression);
if (ctx->blobsTocFH == NULL)
fatal("could not open large object TOC file \"%s\" for input: %m",
@@ -573,6 +574,7 @@ _CloseArchive(ArchiveHandle *AH)
{
cfp *tocFH;
char fname[MAXPGPATH];
+ Compress nocompression = {0};
setFilePath(AH, fname, "toc.dat");
@@ -580,7 +582,7 @@ _CloseArchive(ArchiveHandle *AH)
ctx->pstate = ParallelBackupStart(AH);
/* The TOC is always created uncompressed */
- tocFH = cfopen_write(fname, PG_BINARY_W, 0);
+ tocFH = cfopen_write(fname, PG_BINARY_W, &nocompression);
if (tocFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
ctx->dataFH = tocFH;
@@ -639,11 +641,12 @@ _StartBlobs(ArchiveHandle *AH, TocEntry *te)
{
lclContext *ctx = (lclContext *) AH->formatData;
char fname[MAXPGPATH];
+ Compress nocompression = {0};
setFilePath(AH, fname, "blobs.toc");
/* The blob TOC file is never compressed */
- ctx->blobsTocFH = cfopen_write(fname, "ab", 0);
+ ctx->blobsTocFH = cfopen_write(fname, "ab", &nocompression);
if (ctx->blobsTocFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
}
@@ -661,7 +664,7 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid)
snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid);
- ctx->dataFH = cfopen_write(fname, PG_BINARY_W, AH->compression);
+ ctx->dataFH = cfopen_write(fname, PG_BINARY_W, &AH->compression);
if (ctx->dataFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index 54e708875c..79e528ea89 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -196,10 +196,10 @@ InitArchiveFmt_Tar(ArchiveHandle *AH)
/*
* We don't support compression because reading the files back is not
- * possible since gzdopen uses buffered IO which totally screws file
+ * possible since gzdopen uses buffered IO which totally screws file XXX
* positioning.
*/
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
fatal("compression is not supported by tar archive format");
}
else
@@ -241,6 +241,22 @@ InitArchiveFmt_Tar(ArchiveHandle *AH)
}
}
+/*
+ * Return a string for the given AH's compression.
+ * The string is statically allocated.
+ */
+static const char *
+compress_suffix(ArchiveHandle *AH)
+{
+ CompressionAlgorithm alg;
+ alg = AH->compression.alg;
+ if (alg == COMPR_ALG_LIBZ)
+ return ".gz";
+ if (alg == COMPR_ALG_ZSTD)
+ return ".zstd";
+ return "";
+}
+
/*
* - Start a new TOC entry
* Setup the output file name.
@@ -254,14 +270,8 @@ _ArchiveEntry(ArchiveHandle *AH, TocEntry *te)
ctx = (lclTocEntry *) pg_malloc0(sizeof(lclTocEntry));
if (te->dataDumper != NULL)
{
-#ifdef HAVE_LIBZ
- if (AH->compression == 0)
- sprintf(fn, "%d.dat", te->dumpId);
- else
- sprintf(fn, "%d.dat.gz", te->dumpId);
-#else
- sprintf(fn, "%d.dat", te->dumpId);
-#endif
+ const char *suffix = compress_suffix(AH);
+ sprintf(fn, "%d.dat%s", te->dumpId, suffix);
ctx->filename = pg_strdup(fn);
}
else
@@ -352,7 +362,7 @@ tarOpen(ArchiveHandle *AH, const char *filename, char mode)
#ifdef HAVE_LIBZ
- if (AH->compression == 0)
+ if (AH->compression.alg == COMPR_ALG_NONE)
tm->nFH = ctx->tarFH;
else
fatal("compression is not supported by tar archive format");
@@ -413,9 +423,9 @@ tarOpen(ArchiveHandle *AH, const char *filename, char mode)
#ifdef HAVE_LIBZ
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
{
- sprintf(fmode, "wb%d", AH->compression);
+ sprintf(fmode, "wb%d", AH->compression.level);
tm->zFH = gzdopen(dup(fileno(tm->tmpFH)), fmode);
if (tm->zFH == NULL)
fatal("could not open temporary file");
@@ -443,7 +453,7 @@ tarClose(ArchiveHandle *AH, TAR_MEMBER *th)
/*
* Close the GZ file since we dup'd. This will flush the buffers.
*/
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
if (GZCLOSE(th->zFH) != 0)
fatal("could not close tar member");
@@ -868,7 +878,7 @@ _CloseArchive(ArchiveHandle *AH)
memcpy(ropt, AH->public.ropt, sizeof(RestoreOptions));
ropt->filename = NULL;
ropt->dropSchema = 1;
- ropt->compression = 0;
+ ropt->compression.alg = COMPR_ALG_NONE;
ropt->superuser = NULL;
ropt->suppressDumpWarnings = true;
@@ -952,16 +962,12 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid)
lclContext *ctx = (lclContext *) AH->formatData;
lclTocEntry *tctx = (lclTocEntry *) te->formatData;
char fname[255];
- char *sfx;
+ const char *sfx;
if (oid == 0)
fatal("invalid OID for large object (%u)", oid);
- if (AH->compression != 0)
- sfx = ".gz";
- else
- sfx = "";
-
+ sfx = compress_suffix(AH);
sprintf(fname, "blob_%u.dat%s", oid, sfx);
tarPrintf(ctx->blobToc, "%u %s\n", oid, fname);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index f21eb021c7..c8662f731b 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -59,6 +59,7 @@
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
+#include "compress_io.h"
#include "pg_backup_db.h"
#include "pg_backup_utils.h"
#include "pg_dump.h"
@@ -297,6 +298,99 @@ static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+/* Parse the string into compression options */
+static void
+parse_compression(const char *optarg, Compress *compress)
+{
+ if (optarg[0] == '0' && optarg[1] == '\0')
+ compress->alg = COMPR_ALG_NONE;
+ else if ((optarg[0] > '0' && optarg[0] <= '9') ||
+ optarg[0] == '-')
+ {
+ compress->alg = COMPR_ALG_LIBZ;
+ compress->level = atoi(optarg);
+ if (optarg[1] != '\0')
+ {
+ pg_log_error("compression level must be in range 0..9");
+ exit_nicely(1);
+ }
+ }
+ else
+ {
+ /* Parse a more flexible string like level=3 alg=zlib opts=long */
+ for (;;)
+ {
+ char *eq = strchr(optarg, '=');
+ int len;
+
+ if (eq == NULL)
+ {
+ pg_log_error("compression options must be key=value: %s", optarg);
+ exit_nicely(1);
+ }
+
+ len = eq - optarg;
+ if (strncmp(optarg, "alg", len) == 0)
+ {
+ if (strchr(eq, ' '))
+ len = strchr(eq, ' ') - eq - 1;
+ else
+ len = strlen(eq) - len;
+ if (strncmp(1+eq, "zlib", len) == 0 ||
+ strncmp(1+eq, "libz", len) == 0)
+ compress->alg = COMPR_ALG_LIBZ;
+ else if (strncmp(1+eq, "zstd", len) == 0)
+ compress->alg = COMPR_ALG_ZSTD;
+ else
+ {
+ pg_log_error("unknown compression algorithm: %s", 1+eq);
+ exit_nicely(1);
+ }
+ }
+ else if (strncmp(optarg, "level", len) == 0)
+ compress->level = atoi(1+eq);
+ else if (strncmp(optarg, "opt", len) == 0)
+ {
+ if (strchr(eq, ' '))
+ len = strchr(eq, ' ') - eq - 1;
+ else
+ len = strlen(eq) - len;
+ if (strncmp(1+eq, "zstdlong", len) == 0)
+ compress->zstdlong = true;
+ else
+ {
+ pg_log_error("unknown compression option: %s", 1+eq);
+ exit_nicely(1);
+ }
+ }
+ else
+ {
+ pg_log_error("unknown compression setting: %s", optarg);
+ exit_nicely(1);
+ }
+
+ optarg = strchr(eq, ' ');
+ if (!optarg++)
+ break;
+ }
+
+ /* zstd will check its own compression level later */
+ if (compress->alg != COMPR_ALG_ZSTD)
+ {
+ if (compress->level < 0 || compress->level > 9)
+ {
+ pg_log_error("compression level must be in range 0..9");
+ exit_nicely(1);
+ }
+ if (compress->zstdlong)
+ {
+ pg_log_error("compression option not supported with this algorithm");
+ exit_nicely(1);
+ }
+ }
+ }
+}
+
int
main(int argc, char **argv)
{
@@ -319,7 +413,7 @@ main(int argc, char **argv)
char *use_role = NULL;
long rowsPerInsert;
int numWorkers = 1;
- int compressLevel = -1;
+ Compress compress = { .alg = COMPR_ALG_DEFAULT };
int plainText = 0;
ArchiveFormat archiveFormat = archUnknown;
ArchiveMode archiveMode;
@@ -532,13 +626,7 @@ main(int argc, char **argv)
break;
case 'Z': /* Compression Level */
- compressLevel = atoi(optarg);
- if ((compressLevel < 0 || compressLevel > 9) &&
- compressLevel != -2)
- {
- pg_log_error("compression level must be in range 0..9");
- exit_nicely(1);
- }
+ parse_compression(optarg, &compress);
break;
case 0:
@@ -680,20 +768,39 @@ main(int argc, char **argv)
plainText = 1;
/* Custom and directory formats are compressed by default, others not */
- if (compressLevel == -1)
+ if (compress.alg == COMPR_ALG_DEFAULT)
{
-#ifdef HAVE_LIBZ
if (archiveFormat == archCustom || archiveFormat == archDirectory)
- compressLevel = Z_DEFAULT_COMPRESSION;
- else
+ {
+#ifdef HAVE_LIBZ
+ compress.alg = COMPR_ALG_LIBZ;
#endif
- compressLevel = 0;
+#ifdef HAVE_LIBZSTD
+ compress.alg = COMPR_ALG_ZSTD; // Set default for testing purposes
+#endif
+ compress.level = 3;
+ }
+ else
+ {
+ compress.alg = COMPR_ALG_NONE;
+ compress.level = 0;
+ }
}
#ifndef HAVE_LIBZ
- if (compressLevel != 0)
+ if (compress.alg == COMPR_ALG_LIBZ)
+ {
pg_log_warning("requested compression not available in this installation -- archive will be uncompressed");
- compressLevel = 0;
+ compress.alg = 0;
+ }
+#endif
+
+#ifndef HAVE_LIBZ
+ if (compress.alg == COMPR_ALG_ZSTD)
+ {
+ pg_log_warning("requested compression not available in this installation -- archive will be uncompressed");
+ compress.alg = 0;
+ }
#endif
/*
@@ -724,7 +831,7 @@ main(int argc, char **argv)
fatal("option --index-collation-versions-unknown only works in binary upgrade mode");
/* Open the output file */
- fout = CreateArchive(filename, archiveFormat, compressLevel, dosync,
+ fout = CreateArchive(filename, archiveFormat, &compress, dosync,
archiveMode, setupDumpWorker);
/* Make dump options accessible right away */
@@ -958,10 +1065,7 @@ main(int argc, char **argv)
ropt->sequence_data = dopt.sequence_data;
ropt->binary_upgrade = dopt.binary_upgrade;
- if (compressLevel == -1)
- ropt->compression = 0;
- else
- ropt->compression = compressLevel;
+ ropt->compression = compress;
ropt->suppressDumpWarnings = true; /* We've already shown them */
--
2.17.0
[text/x-diff] 0005-union-with-a-CompressionAlgorithm-alg.patch (11.6K, ../../20201221194924.GI30237@telsasoft.com/6-0005-union-with-a-CompressionAlgorithm-alg.patch)
download | inline diff:
From c9d537cd341c96f21d749006002291e30a383463 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 21 Dec 2020 00:11:43 -0600
Subject: [PATCH 5/7] union with a CompressionAlgorithm alg
---
src/bin/pg_dump/compress_io.c | 219 ++++++++++++++++------------------
1 file changed, 106 insertions(+), 113 deletions(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 10db22ff88..ae1ae7b0d0 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -614,22 +614,27 @@ WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
*/
struct cfp
{
- FILE *uncompressedfp;
+ CompressionAlgorithm alg;
+
+ union {
+ FILE *fp;
+
#ifdef HAVE_LIBZ
- gzFile compressedfp;
+ gzFile gzfp;
#endif
-#ifdef HAVE_LIBZSTD // XXX: this should be a union with a CompressionAlgorithm alg?
- /* This is a normal file to which we read/write compressed data */
- struct {
- FILE *fp;
- // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
- ZSTD_CStream *cstream;
- ZSTD_DStream *dstream;
- ZSTD_outBuffer output;
- ZSTD_inBuffer input;
- } zstd;
+#ifdef HAVE_LIBZSTD
+ struct {
+ /* This is a normal file to which we read/write compressed data */
+ FILE *fp;
+ // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ } zstd;
#endif
+ } u;
};
@@ -730,13 +735,7 @@ cfopen(const char *path, const char *mode, Compress *compression)
{
cfp *fp = pg_malloc(sizeof(cfp));
- fp->uncompressedfp = NULL;
-#ifdef HAVE_LIBZ
- fp->compressedfp = NULL;
-#endif
-#ifdef HAVE_LIBZSTD
- fp->zstd.fp = NULL;
-#endif
+ fp->alg = compression->alg;
switch (compression->alg)
{
@@ -749,15 +748,15 @@ cfopen(const char *path, const char *mode, Compress *compression)
snprintf(mode_compression, sizeof(mode_compression), "%s%d",
mode, compression->level);
- fp->compressedfp = gzopen(path, mode_compression);
+ fp->u.gzfp = gzopen(path, mode_compression);
}
else
{
/* don't specify a level, just use the zlib default */
- fp->compressedfp = gzopen(path, mode);
+ fp->u.gzfp = gzopen(path, mode);
}
- if (fp->compressedfp == NULL)
+ if (fp->u.gzfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
@@ -769,29 +768,29 @@ cfopen(const char *path, const char *mode, Compress *compression)
#ifdef HAVE_LIBZSTD
case COMPR_ALG_ZSTD:
- fp->zstd.fp = fopen(path, mode);
+ fp->u.zstd.fp = fopen(path, mode);
// XXX: save the compression params
- if (fp->zstd.fp == NULL)
+ if (fp->u.zstd.fp == NULL)
{
free_keep_errno(fp);
fp = NULL;
}
else if (strchr(mode, 'w'))
{
- fp->zstd.dstream = NULL;
- fp->zstd.output.size = ZSTD_CStreamOutSize(); // XXX
- fp->zstd.output.dst = pg_malloc0(fp->zstd.output.size);
- fp->zstd.cstream = ZSTD_createCStream();
- if (fp->zstd.cstream == NULL)
+ fp->u.zstd.dstream = NULL;
+ fp->u.zstd.output.size = ZSTD_CStreamOutSize(); // XXX
+ fp->u.zstd.output.dst = pg_malloc0(fp->u.zstd.output.size);
+ fp->u.zstd.cstream = ZSTD_createCStream();
+ if (fp->u.zstd.cstream == NULL)
fatal("could not initialize compression library");
}
else if (strchr(mode, 'r'))
{
- fp->zstd.cstream = NULL;
- fp->zstd.input.size = ZSTD_DStreamOutSize(); // XXX
- fp->zstd.input.src = pg_malloc0(fp->zstd.input.size);
- fp->zstd.dstream = ZSTD_createDStream();
- if (fp->zstd.dstream == NULL)
+ fp->u.zstd.cstream = NULL;
+ fp->u.zstd.input.size = ZSTD_DStreamOutSize(); // XXX
+ fp->u.zstd.input.src = pg_malloc0(fp->u.zstd.input.size);
+ fp->u.zstd.dstream = ZSTD_createDStream();
+ if (fp->u.zstd.dstream == NULL)
fatal("could not initialize compression library");
} // XXX else: bad mode
return fp;
@@ -799,8 +798,8 @@ cfopen(const char *path, const char *mode, Compress *compression)
#endif
case COMPR_ALG_NONE:
- fp->uncompressedfp = fopen(path, mode);
- if (fp->uncompressedfp == NULL)
+ fp->u.fp = fopen(path, mode);
+ if (fp->u.fp == NULL)
{
free_keep_errno(fp);
fp = NULL;
@@ -824,26 +823,26 @@ cfread(void *ptr, int size, cfp *fp)
return 0;
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
- ret = gzread(fp->compressedfp, ptr, size);
- if (ret != size && !gzeof(fp->compressedfp))
+ ret = gzread(fp->u.gzfp, ptr, size);
+ if (ret != size && !gzeof(fp->u.gzfp))
{
int errnum;
- const char *errmsg = gzerror(fp->compressedfp, &errnum);
+ const char *errmsg = gzerror(fp->u.gzfp, &errnum);
fatal("could not read from input file: %s",
errnum == Z_ERRNO ? strerror(errno) : errmsg);
}
+ return ret;
}
- else
#endif
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
- ZSTD_outBuffer *output = &fp->zstd.output;
- ZSTD_inBuffer *input = &fp->zstd.input;
+ ZSTD_outBuffer *output = &fp->u.zstd.output;
+ ZSTD_inBuffer *input = &fp->u.zstd.input;
size_t input_size = ZSTD_DStreamInSize();
size_t res, cnt;
@@ -852,7 +851,7 @@ cfread(void *ptr, int size, cfp *fp)
output->pos = 0;
/* read compressed data */
- while ((cnt = fread(unconstify(void *, input->src), 1, input_size, fp->zstd.fp)))
+ while ((cnt = fread(unconstify(void *, input->src), 1, input_size, fp->u.zstd.fp)))
{
input->size = cnt;
input->pos = 0;
@@ -860,7 +859,7 @@ cfread(void *ptr, int size, cfp *fp)
for ( ; input->pos < input->size; )
{
/* decompress */
- res = ZSTD_decompressStream(fp->zstd.dstream, output, input);
+ res = ZSTD_decompressStream(fp->u.zstd.dstream, output, input);
if (res == 0 || output->pos == output->size)
break;
if (ZSTD_isError(res))
@@ -871,16 +870,13 @@ cfread(void *ptr, int size, cfp *fp)
break; /* We read all the data that fits */
}
- ret = output->pos;
+ return output->pos;
}
- else
#endif
- {
- ret = fread(ptr, 1, size, fp->uncompressedfp);
- if (ret != size && !feof(fp->uncompressedfp))
- READ_ERROR_EXIT(fp->uncompressedfp);
- }
+ ret = fread(ptr, 1, size, fp->u.fp);
+ if (ret != size && !feof(fp->u.fp))
+ READ_ERROR_EXIT(fp->u.fp);
return ret;
}
@@ -888,16 +884,16 @@ int
cfwrite(const void *ptr, int size, cfp *fp)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
- return gzwrite(fp->compressedfp, ptr, size);
- else
+ if (fp->alg == COMPR_ALG_LIBZ)
+ return gzwrite(fp->u.gzfp, ptr, size);
#endif
+
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
size_t res, cnt;
- ZSTD_outBuffer *output = &fp->zstd.output;
- ZSTD_inBuffer *input = &fp->zstd.input;
+ ZSTD_outBuffer *output = &fp->u.zstd.output;
+ ZSTD_inBuffer *input = &fp->u.zstd.input;
input->src = ptr;
input->size = size;
@@ -907,21 +903,20 @@ cfwrite(const void *ptr, int size, cfp *fp)
while (input->pos != input->size)
{
output->pos = 0;
- res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_continue);
+ res = ZSTD_compressStream2(fp->u.zstd.cstream, output, input, ZSTD_e_continue);
if (ZSTD_isError(res))
fatal("could not compress data: %s", ZSTD_getErrorName(res));
- cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ cnt = fwrite(output->dst, 1, output->pos, fp->u.zstd.fp);
if (cnt != output->pos)
fatal("could not write data: %s", strerror(errno));
}
return size;
}
- else
#endif
- return fwrite(ptr, 1, size, fp->uncompressedfp);
+ return fwrite(ptr, 1, size, fp->u.fp);
}
int
@@ -930,39 +925,38 @@ cfgetc(cfp *fp)
int ret;
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
- ret = gzgetc(fp->compressedfp);
+ ret = gzgetc(fp->u.gzfp);
if (ret == EOF)
{
- if (!gzeof(fp->compressedfp))
+ if (!gzeof(fp->u.gzfp))
fatal("could not read from input file: %s", strerror(errno));
else
fatal("could not read from input file: end of file");
}
+ return ret;
}
- else
#endif
+
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
if (cfread(&ret, 1, fp) != 1)
{
- if (feof(fp->zstd.fp))
+ if (feof(fp->u.zstd.fp))
fatal("could not read from input file: end of file");
else
fatal("could not read from input file: %s", strerror(errno));
}
fprintf(stderr, "cfgetc %d\n", ret);
+ return ret;
}
#endif
- {
- ret = fgetc(fp->uncompressedfp);
- if (ret == EOF)
- READ_ERROR_EXIT(fp->uncompressedfp);
- }
-
+ ret = fgetc(fp->u.fp);
+ if (ret == EOF)
+ READ_ERROR_EXIT(fp->u.fp);
return ret;
}
@@ -970,12 +964,12 @@ char *
cfgets(cfp *fp, char *buf, int len)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
- return gzgets(fp->compressedfp, buf, len);
- else
+ if (fp->alg == COMPR_ALG_LIBZ)
+ return gzgets(fp->u.gzfp, buf, len);
#endif
+
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
int res;
res = cfread(buf, len, fp);
@@ -984,9 +978,9 @@ cfgets(cfp *fp, char *buf, int len)
*strchr(buf, '\n') = '\0';
return res > 0 ? buf : 0;
}
- else
#endif
- return fgets(buf, len, fp->uncompressedfp);
+
+ return fgets(buf, len, fp->u.fp);
}
int
@@ -1000,56 +994,55 @@ cfclose(cfp *fp)
return EOF;
}
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
- result = gzclose(fp->compressedfp);
- fp->compressedfp = NULL;
+ result = gzclose(fp->u.gzfp);
+ fp->u.gzfp = NULL;
+ return result;
}
- else
#endif
+
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
- ZSTD_outBuffer *output = &fp->zstd.output;
- ZSTD_inBuffer *input = &fp->zstd.input;
+ ZSTD_outBuffer *output = &fp->u.zstd.output;
+ ZSTD_inBuffer *input = &fp->u.zstd.input;
size_t res, cnt;
- if (fp->zstd.cstream)
+ if (fp->u.zstd.cstream)
{
for (;;)
{
output->pos = 0;
- res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_end);
+ res = ZSTD_compressStream2(fp->u.zstd.cstream, output, input, ZSTD_e_end);
if (ZSTD_isError(res))
fatal("could not compress data: %s", ZSTD_getErrorName(res));
- cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ cnt = fwrite(output->dst, 1, output->pos, fp->u.zstd.fp);
if (cnt != output->pos)
fatal("could not write data: %s", strerror(errno));
if (res == 0)
break;
}
- ZSTD_freeCStream(fp->zstd.cstream);
- pg_free(fp->zstd.output.dst);
+ ZSTD_freeCStream(fp->u.zstd.cstream);
+ pg_free(fp->u.zstd.output.dst);
}
- if (fp->zstd.dstream)
+ if (fp->u.zstd.dstream)
{
- ZSTD_freeDStream(fp->zstd.dstream);
- pg_free(unconstify(void *, fp->zstd.input.src));
+ ZSTD_freeDStream(fp->u.zstd.dstream);
+ pg_free(unconstify(void *, fp->u.zstd.input.src));
}
- result = fclose(fp->zstd.fp);
- fp->zstd.fp = NULL;
+ result = fclose(fp->u.zstd.fp);
+ fp->u.zstd.fp = NULL;
+ return result;
}
- else
#endif
- {
- result = fclose(fp->uncompressedfp);
- fp->uncompressedfp = NULL;
- }
- free_keep_errno(fp);
+ result = fclose(fp->u.fp);
+ fp->u.fp = NULL;
+ free_keep_errno(fp);
return result;
}
@@ -1057,26 +1050,26 @@ int
cfeof(cfp *fp)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
- return gzeof(fp->compressedfp);
- else
+ if (fp->alg == COMPR_ALG_LIBZ)
+ return gzeof(fp->u.gzfp);
#endif
+
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
- return feof(fp->zstd.fp);
- else
+ if (fp->alg == COMPR_ALG_ZSTD)
+ return feof(fp->u.zstd.fp);
#endif
- return feof(fp->uncompressedfp);
+
+ return feof(fp->u.fp);
}
const char *
get_cfp_error(cfp *fp)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
int errnum;
- const char *errmsg = gzerror(fp->compressedfp, &errnum);
+ const char *errmsg = gzerror(fp->u.gzfp, &errnum);
if (errnum != Z_ERRNO)
return errmsg;
--
2.17.0
[text/x-diff] 0006-Move-zlib-into-the-union.patch (4.4K, ../../20201221194924.GI30237@telsasoft.com/7-0006-Move-zlib-into-the-union.patch)
download | inline diff:
From d52903ad3661cf62c375371515a702a60beb3f63 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 11 Dec 2020 22:22:31 -0600
Subject: [PATCH 6/7] Move zlib into the union{}
---
src/bin/pg_dump/compress_io.c | 54 ++++++++++++++++++-----------------
1 file changed, 28 insertions(+), 26 deletions(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index ae1ae7b0d0..d93bc61fd8 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -67,23 +67,25 @@ struct CompressorState
CompressionAlgorithm comprAlg;
WriteFunc writeF;
+ union {
#ifdef HAVE_LIBZ
- z_streamp zp;
- char *zlibOut;
- size_t zlibOutSize;
+ struct {
+ z_streamp zp;
+ char *zlibOut;
+ size_t zlibOutSize;
+ } zlib;
#endif
#ifdef HAVE_LIBZSTD
- union {
+ /* This is used for compression but not decompression */
struct {
- ZSTD_outBuffer output;
- ZSTD_inBuffer input;
// XXX: use one separate ZSTD_CStream per thread: disable on windows ?
- ZSTD_CStream *cstream;
+ ZSTD_CStream *cstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
} zstd;
- } u;
#endif
-
+ } u;
};
/* Routines that support zlib compressed data I/O */
@@ -415,7 +417,7 @@ InitCompressorZlib(CompressorState *cs, Compress *compress)
{
z_streamp zp;
- zp = cs->zp = (z_streamp) pg_malloc(sizeof(z_stream));
+ zp = cs->u.zlib.zp = (z_streamp) pg_malloc(sizeof(z_stream));
zp->zalloc = Z_NULL;
zp->zfree = Z_NULL;
zp->opaque = Z_NULL;
@@ -425,22 +427,22 @@ InitCompressorZlib(CompressorState *cs, Compress *compress)
* actually allocate one extra byte because some routines want to append a
* trailing zero byte to the zlib output.
*/
- cs->zlibOut = (char *) pg_malloc(ZLIB_OUT_SIZE + 1);
- cs->zlibOutSize = ZLIB_OUT_SIZE;
+ cs->u.zlib.zlibOut = (char *) pg_malloc(ZLIB_OUT_SIZE + 1);
+ cs->u.zlib.zlibOutSize = ZLIB_OUT_SIZE;
if (deflateInit(zp, compress->level) != Z_OK)
fatal("could not initialize compression library: %s",
zp->msg);
/* Just be paranoid - maybe End is called after Start, with no Write */
- zp->next_out = (void *) cs->zlibOut;
- zp->avail_out = cs->zlibOutSize;
+ zp->next_out = (void *) cs->u.zlib.zlibOut;
+ zp->avail_out = cs->u.zlib.zlibOutSize;
}
static void
EndCompressorZlib(ArchiveHandle *AH, CompressorState *cs)
{
- z_streamp zp = cs->zp;
+ z_streamp zp = cs->u.zlib.zp;
zp->next_in = NULL;
zp->avail_in = 0;
@@ -451,23 +453,23 @@ EndCompressorZlib(ArchiveHandle *AH, CompressorState *cs)
if (deflateEnd(zp) != Z_OK)
fatal("could not close compression stream: %s", zp->msg);
- free(cs->zlibOut);
- free(cs->zp);
+ free(cs->u.zlib.zlibOut);
+ free(cs->u.zlib.zp);
}
static void
DeflateCompressorZlib(ArchiveHandle *AH, CompressorState *cs, bool flush)
{
- z_streamp zp = cs->zp;
- char *out = cs->zlibOut;
+ z_streamp zp = cs->u.zlib.zp;
+ char *out = cs->u.zlib.zlibOut;
int res = Z_OK;
- while (cs->zp->avail_in != 0 || flush)
+ while (cs->u.zlib.zp->avail_in != 0 || flush)
{
res = deflate(zp, flush ? Z_FINISH : Z_NO_FLUSH);
if (res == Z_STREAM_ERROR)
fatal("could not compress data: %s", zp->msg);
- if ((flush && (zp->avail_out < cs->zlibOutSize))
+ if ((flush && (zp->avail_out < cs->u.zlib.zlibOutSize))
|| (zp->avail_out == 0)
|| (zp->avail_in != 0)
)
@@ -477,18 +479,18 @@ DeflateCompressorZlib(ArchiveHandle *AH, CompressorState *cs, bool flush)
* chunk is the EOF marker in the custom format. This should never
* happen but...
*/
- if (zp->avail_out < cs->zlibOutSize)
+ if (zp->avail_out < cs->u.zlib.zlibOutSize)
{
/*
* Any write function should do its own error checking but to
* make sure we do a check here as well...
*/
- size_t len = cs->zlibOutSize - zp->avail_out;
+ size_t len = cs->u.zlib.zlibOutSize - zp->avail_out;
cs->writeF(AH, out, len);
}
zp->next_out = (void *) out;
- zp->avail_out = cs->zlibOutSize;
+ zp->avail_out = cs->u.zlib.zlibOutSize;
}
if (res == Z_STREAM_END)
@@ -500,8 +502,8 @@ static void
WriteDataToArchiveZlib(ArchiveHandle *AH, CompressorState *cs,
const char *data, size_t dLen)
{
- cs->zp->next_in = (void *) unconstify(char *, data);
- cs->zp->avail_in = dLen;
+ cs->u.zlib.zp->next_in = (void *) unconstify(char *, data);
+ cs->u.zlib.zp->avail_in = dLen;
DeflateCompressorZlib(AH, cs, false);
}
--
2.17.0
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2020-12-21 19:49 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2020-12-21 20:02 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-12-22 02:32 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 46+ messages in thread
From: Tom Lane @ 2020-12-21 20:02 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers
Justin Pryzby <pryzby@telsasoft.com> writes:
> I found that our largest tables are 40% smaller and 20% faster to pipe
> pg_dump -Fc -Z0 |zstd relative to native zlib
The patch might be a tad smaller if you hadn't included a core file in it.
regards, tom lane
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2020-12-21 19:49 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2020-12-21 20:02 ` Re: zstd compression for pg_dump Tom Lane <tgl@sss.pgh.pa.us>
@ 2020-12-22 02:32 ` Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 02:53 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2020-12-22 02:32 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers
On Mon, Dec 21, 2020 at 03:02:40PM -0500, Tom Lane wrote:
> Justin Pryzby <pryzby@telsasoft.com> writes:
> > I found that our largest tables are 40% smaller and 20% faster to pipe
> > pg_dump -Fc -Z0 |zstd relative to native zlib
>
> The patch might be a tad smaller if you hadn't included a core file in it.
About 89% smaller.
This also fixes the extension (.zst)
And fixes zlib default compression.
And a bunch of cleanup.
--
Justin
Attachments:
[text/x-diff] 0001-fix-pre-existing-docs-comments.patch (1.7K, ../../20201222023235.GJ30237@telsasoft.com/2-0001-fix-pre-existing-docs-comments.patch)
download | inline diff:
From c506c8a93ae70726a5b4b33a1d0098caf5665f3a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 21 Dec 2020 00:32:32 -0600
Subject: [PATCH 1/7] fix pre-existing docs/comments
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
src/bin/pg_dump/pg_backup_directory.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0aa35cf0c3..dcb25dc3cd 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -621,7 +621,7 @@ PostgreSQL documentation
<listitem>
<para>
Specify the compression level to use. Zero means no compression.
- For the custom archive format, this specifies compression of
+ For the custom and directory archive formats, this specifies compression of
individual table-data segments, and the default is to compress
at a moderate level.
For plain text output, setting a nonzero compression level causes
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 48fa7cb1a3..650b542fce 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -4,7 +4,7 @@
*
* A directory format dump is a directory, which contains a "toc.dat" file
* for the TOC, and a separate file for each data entry, named "<oid>.dat".
- * Large objects (BLOBs) are stored in separate files named "blob_<uid>.dat",
+ * Large objects (BLOBs) are stored in separate files named "blob_<oid>.dat",
* and there's a plain-text TOC file for them called "blobs.toc". If
* compression is used, each data file is individually compressed and the
* ".gz" suffix is added to the filenames. The TOC files are never
--
2.17.0
[text/x-diff] 0002-Fix-malformed-comment.patch (1.2K, ../../20201222023235.GJ30237@telsasoft.com/3-0002-Fix-malformed-comment.patch)
download | inline diff:
From 49f400b5b2e559cc2ed6be82e04a4dc4ba5c63b3 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 12 Dec 2020 00:11:37 -0600
Subject: [PATCH 2/7] Fix malformed comment..
broken since bf50caf10.
---
src/bin/pg_dump/pg_backup_archiver.h | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 177360ed6e..6a5a22637b 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -329,9 +329,12 @@ struct _archiveHandle
DumpId *tableDataId; /* TABLE DATA ids, indexed by table dumpId */
struct _tocEntry *currToc; /* Used when dumping data */
- int compression; /* Compression requested on open Possible
- * values for compression: -1
- * Z_DEFAULT_COMPRESSION 0 COMPRESSION_NONE
+ int compression; /*---------
+ * Compression requested on open
+ * Possible values for compression:
+ * -2 ZSTD_COMPRESSION
+ * -1 Z_DEFAULT_COMPRESSION
+ * 0 COMPRESSION_NONE
* 1-9 levels for gzip compression */
bool dosync; /* data requested to be synced on sight */
ArchiveMode mode; /* File mode - r or w */
--
2.17.0
[text/x-diff] 0003-pg_dump-zstd-compression.patch (30.0K, ../../20201222023235.GJ30237@telsasoft.com/4-0003-pg_dump-zstd-compression.patch)
download | inline diff:
From 8a8939a3060c984af289b5fe62754d94f2675248 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 11 Dec 2020 15:01:25 -0600
Subject: [PATCH 3/7] pg_dump: zstd compression
document any change in search for .gz?
docs
---
configure | 123 ++++++-
configure.ac | 22 ++
src/bin/pg_dump/compress_io.c | 475 ++++++++++++++++++++++++--
src/bin/pg_dump/compress_io.h | 5 +-
src/bin/pg_dump/pg_backup_archiver.h | 5 +
src/bin/pg_dump/pg_backup_directory.c | 6 +-
src/bin/pg_dump/pg_dump.c | 3 +-
src/include/pg_config.h.in | 3 +
8 files changed, 597 insertions(+), 45 deletions(-)
diff --git a/configure b/configure
index 11a4284e5b..240e536e04 100755
--- a/configure
+++ b/configure
@@ -698,6 +698,7 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+with_zstd
with_zlib
with_system_tzdata
with_libxslt
@@ -798,6 +799,7 @@ infodir
docdir
oldincludedir
includedir
+runstatedir
localstatedir
sharedstatedir
sysconfdir
@@ -866,6 +868,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_zstd
with_gnu_ld
enable_largefile
'
@@ -935,6 +938,7 @@ datadir='${datarootdir}'
sysconfdir='${prefix}/etc'
sharedstatedir='${prefix}/com'
localstatedir='${prefix}/var'
+runstatedir='${localstatedir}/run'
includedir='${prefix}/include'
oldincludedir='/usr/include'
docdir='${datarootdir}/doc/${PACKAGE_TARNAME}'
@@ -1187,6 +1191,15 @@ do
| -silent | --silent | --silen | --sile | --sil)
silent=yes ;;
+ -runstatedir | --runstatedir | --runstatedi | --runstated \
+ | --runstate | --runstat | --runsta | --runst | --runs \
+ | --run | --ru | --r)
+ ac_prev=runstatedir ;;
+ -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \
+ | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \
+ | --run=* | --ru=* | --r=*)
+ runstatedir=$ac_optarg ;;
+
-sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb)
ac_prev=sbindir ;;
-sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \
@@ -1324,7 +1337,7 @@ fi
for ac_var in exec_prefix prefix bindir sbindir libexecdir datarootdir \
datadir sysconfdir sharedstatedir localstatedir includedir \
oldincludedir docdir infodir htmldir dvidir pdfdir psdir \
- libdir localedir mandir
+ libdir localedir mandir runstatedir
do
eval ac_val=\$$ac_var
# Remove trailing slashes.
@@ -1477,6 +1490,7 @@ Fine tuning of the installation directories:
--sysconfdir=DIR read-only single-machine data [PREFIX/etc]
--sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com]
--localstatedir=DIR modifiable single-machine data [PREFIX/var]
+ --runstatedir=DIR modifiable per-process data [LOCALSTATEDIR/run]
--libdir=DIR object code libraries [EPREFIX/lib]
--includedir=DIR C header files [PREFIX/include]
--oldincludedir=DIR C header files for non-gcc [/usr/include]
@@ -1570,6 +1584,7 @@ Optional Packages:
--with-system-tzdata=DIR
use system time zone data in DIR
--without-zlib do not use Zlib
+ --without-zstd do not use Zstd
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
Some influential environment variables:
@@ -8601,6 +8616,35 @@ fi
+#
+# Zstd
+#
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+ :
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=yes
+
+fi
+
+
+
+
#
# Assignments
#
@@ -12092,6 +12136,59 @@ fi
fi
+if test "$with_zstd" = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compressStream2 in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compressStream2 in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compressStream2+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compressStream2 ();
+int
+main ()
+{
+return ZSTD_compressStream2 ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_compressStream2=yes
+else
+ ac_cv_lib_zstd_ZSTD_compressStream2=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compressStream2" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compressStream2" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compressStream2" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support." "$LINENO" 5
+fi
+
+fi
+
if test "$enable_spinlocks" = yes; then
$as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h
@@ -13295,6 +13392,20 @@ Use --without-zlib to disable zlib support." "$LINENO" 5
fi
+fi
+
+if test "$with_zstd" = yes; then
+ ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+
+else
+ as_fn_error $? "zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support." "$LINENO" 5
+fi
+
+
fi
if test "$with_gssapi" = yes ; then
@@ -14689,7 +14800,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14735,7 +14846,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14759,7 +14870,7 @@ rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14804,7 +14915,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14828,7 +14939,7 @@ rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
diff --git a/configure.ac b/configure.ac
index fc523c6aeb..7f7222159a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -999,6 +999,13 @@ PGAC_ARG_BOOL(with, zlib, yes,
[do not use Zlib])
AC_SUBST(with_zlib)
+#
+# Zstd
+#
+PGAC_ARG_BOOL(with, zstd, yes,
+ [do not use Zstd])
+AC_SUBST(with_zstd)
+
#
# Assignments
#
@@ -1186,6 +1193,14 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_LIB(zstd, ZSTD_compressStream2, [],
+ [AC_MSG_ERROR([zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support.])])
+fi
+
if test "$enable_spinlocks" = yes; then
AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.])
else
@@ -1400,6 +1415,13 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_HEADER(zstd.h, [], [AC_MSG_ERROR([zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support.])])
+fi
+
if test "$with_gssapi" = yes ; then
AC_CHECK_HEADERS(gssapi/gssapi.h, [],
[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 1417401086..b51ba680a2 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -13,7 +13,7 @@
* friends, providing an interface similar to those, but abstracts away
* the possible compression. Both APIs use libz for the compression, but
* the second API uses gzip headers, so the resulting files can be easily
- * manipulated with the gzip utility.
+ * manipulated with the gzip utility. XXX
*
* Compressor API
* --------------
@@ -41,7 +41,7 @@
* libz's gzopen() APIs. It allows you to use the same functions for
* compressed and uncompressed streams. cfopen_read() first tries to open
* the file with given name, and if it fails, it tries to open the same
- * file with the .gz suffix. cfopen_write() opens a file for writing, an
+ * file with the .gz suffix. cfopen_write() opens a file for writing, an XXX
* extra argument specifies if the file should be compressed, and adds the
* .gz suffix to the filename if so. This allows you to easily handle both
* compressed and uncompressed files.
@@ -72,6 +72,18 @@ struct CompressorState
char *zlibOut;
size_t zlibOutSize;
#endif
+
+#ifdef HAVE_LIBZSTD
+ union {
+ struct {
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
+ ZSTD_CStream *cstream;
+ } zstd;
+ } u;
+#endif
+
};
static void ParseCompressionOption(int compression, CompressionAlgorithm *alg,
@@ -88,6 +100,15 @@ static void WriteDataToArchiveZlib(ArchiveHandle *AH, CompressorState *cs,
static void EndCompressorZlib(ArchiveHandle *AH, CompressorState *cs);
#endif
+#ifdef HAVE_LIBZSTD
+static void InitCompressorZstd(CompressorState *cs, int level);
+static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void DeflateCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const char *data, size_t dLen);
+static void ReadDataFromArchiveZstd(ArchiveHandle *AH, ReadFunc readF);
+#endif
+
/* Routines that support uncompressed data I/O */
static void ReadDataFromArchiveNone(ArchiveHandle *AH, ReadFunc readF);
static void WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
@@ -101,15 +122,25 @@ static void WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
static void
ParseCompressionOption(int compression, CompressionAlgorithm *alg, int *level)
{
- if (compression == Z_DEFAULT_COMPRESSION ||
- (compression > 0 && compression <= 9))
- *alg = COMPR_ALG_LIBZ;
- else if (compression == 0)
- *alg = COMPR_ALG_NONE;
- else
+ switch (compression)
{
- fatal("invalid compression code: %d", compression);
- *alg = COMPR_ALG_NONE; /* keep compiler quiet */
+#ifdef HAVE_LIBZSTD
+ case ZSTD_COMPRESSION:
+ *alg = COMPR_ALG_ZSTD;
+ break;
+#endif
+#ifdef HAVE_ZLIB
+ case Z_DEFAULT_COMPRESSION:
+ case 1..9:
+ *alg = COMPR_ALG_LIBZ;
+ break;
+#endif
+ case 0:
+ *alg = COMPR_ALG_NONE;
+ break;
+ default:
+ fatal("invalid compression code: %d", compression);
+ *alg = COMPR_ALG_NONE; /* keep compiler quiet */
}
/* The level is just the passed-in value. */
@@ -141,10 +172,23 @@ AllocateCompressor(int compression, WriteFunc writeF)
/*
* Perform compression algorithm specific initialization.
*/
+ switch (alg)
+ {
#ifdef HAVE_LIBZ
- if (alg == COMPR_ALG_LIBZ)
+ case COMPR_ALG_LIBZ:
InitCompressorZlib(cs, level);
+ break;
+#endif
+#ifdef HAVE_LIBZSTD
+ case COMPR_ALG_ZSTD:
+ InitCompressorZstd(cs, level);
+ break;
#endif
+ case COMPR_ALG_NONE:
+ /* Do nothing */
+ break;
+ // default:
+ }
return cs;
}
@@ -162,12 +206,20 @@ ReadDataFromArchive(ArchiveHandle *AH, int compression, ReadFunc readF)
if (alg == COMPR_ALG_NONE)
ReadDataFromArchiveNone(AH, readF);
- if (alg == COMPR_ALG_LIBZ)
+ else if (alg == COMPR_ALG_LIBZ)
{
#ifdef HAVE_LIBZ
ReadDataFromArchiveZlib(AH, readF);
#else
fatal("not built with zlib support");
+#endif
+ }
+ else if (alg == COMPR_ALG_ZSTD)
+ {
+#ifdef HAVE_LIBZSTD
+ ReadDataFromArchiveZstd(AH, readF);
+#else
+ fatal("not built with zstd support");
#endif
}
}
@@ -188,6 +240,15 @@ WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs,
fatal("not built with zlib support");
#endif
break;
+
+ case COMPR_ALG_ZSTD:
+#ifdef HAVE_LIBZSTD
+ WriteDataToArchiveZstd(AH, cs, data, dLen);
+#else
+ fatal("not built with zstd support");
+#endif
+ break;
+
case COMPR_ALG_NONE:
WriteDataToArchiveNone(AH, cs, data, dLen);
break;
@@ -204,12 +265,172 @@ EndCompressor(ArchiveHandle *AH, CompressorState *cs)
if (cs->comprAlg == COMPR_ALG_LIBZ)
EndCompressorZlib(AH, cs);
#endif
+#ifdef HAVE_LIBZSTD
+ if (cs->comprAlg == COMPR_ALG_ZSTD)
+ EndCompressorZstd(AH, cs);
+#endif
+
free(cs);
}
/* Private routines, specific to each compression method. */
+// XXX: put in separate files ?
+
+#ifdef HAVE_LIBZSTD
+static void
+InitCompressorZstd(CompressorState *cs, int level)
+{
+ cs->u.zstd.cstream = ZSTD_createCStream();
+ if (cs->u.zstd.cstream == NULL)
+ fatal("could not initialize compression library");
+
+ /* XXX: initialize safely like the corresponding zlib "paranoia" */
+ cs->u.zstd.output.size = ZSTD_CStreamOutSize();
+ cs->u.zstd.output.dst = pg_malloc(cs->u.zstd.output.size);
+ cs->u.zstd.output.pos = 0;
+}
+
+static void
+EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZSTD_outBuffer *output = &cs->u.zstd.output;
+
+ for (;;)
+ {
+ size_t res;
+
+ res = ZSTD_compressStream2(cs->u.zstd.cstream, output,
+ &cs->u.zstd.input, ZSTD_e_end);
+
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ if (res == 0)
+ break;
+
+ if (ZSTD_isError(res))
+ fatal("could not close compression stream: %s",
+ ZSTD_getErrorName(res));
+ }
+
+ // XXX: retval
+ ZSTD_freeCStream(cs->u.zstd.cstream);
+}
+
+static void
+DeflateCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZSTD_inBuffer *input = &cs->u.zstd.input;
+ ZSTD_outBuffer *output = &cs->u.zstd.output;
+
+ while (input->pos != input->size)
+ {
+ size_t res;
+
+ res = ZSTD_compressStream2(cs->u.zstd.cstream, output,
+ input, ZSTD_e_continue);
+
+ if (output->pos == output->size ||
+ input->pos != input->size)
+ {
+ /*
+ * Extra paranoia: avoid zero-length chunks, since a zero length
+ * chunk is the EOF marker in the custom format. This should never
+ * happen but...
+ */
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ output->pos = 0;
+ }
+
+ if (ZSTD_isError(res))
+ fatal("could not compress data: %s", ZSTD_getErrorName(res));
+ }
+}
+
+static void
+WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const char *data, size_t dLen)
+{
+ cs->u.zstd.input.src = (void *) unconstify(char *, data);
+ cs->u.zstd.input.size = dLen;
+ cs->u.zstd.input.pos = 0;
+ DeflateCompressorZstd(AH, cs);
+}
+
+/* Read data from a compressed zstd archive */
+static void
+ReadDataFromArchiveZstd(ArchiveHandle *AH, ReadFunc readF)
+{
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ size_t res;
+ size_t input_size;
+
+ dstream = ZSTD_createDStream();
+ if (dstream == NULL)
+ fatal("could not initialize compression library");
+
+ input_size = ZSTD_DStreamInSize();
+ input.src = pg_malloc(input_size);
+
+ output.size = ZSTD_DStreamOutSize();
+ output.dst = pg_malloc(output.size);
+
+ /* read compressed data */
+ for (;;)
+ {
+ size_t cnt;
+
+ input.size = input_size; // XXX: the buffer can grow, we shouldn't keep resetting it to the original value..
+ cnt = readF(AH, (char **)unconstify(void **, &input.src), &input.size);
+ input.pos = 0;
+ input.size = cnt;
+
+ if (cnt == 0)
+ break;
+
+ while (input.pos < input.size)
+ {
+ /* decompress */
+ output.pos = 0;
+ res = ZSTD_decompressStream(dstream, &output, &input);
+
+ if (ZSTD_isError(res))
+ fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ /* write to output handle */
+ ((char *)output.dst)[output.pos] = '\0';
+ ahwrite(output.dst, 1, output.pos, AH);
+ }
+ }
+
+ /* write any remainder to output handle */
+ /* XXX: is it needed? */
+ /* If `input.pos < input.size`, some input has not been consumed. */
+ /* But if `output.pos == output.size`, there might be some data left within internal buffers., */
+#if 0
+ while (input.pos < input.size || output.pos == output.size)
+ {
+ output.pos = 0;
+ res = ZSTD_decompressStream(dstream, &output, &input);
+ if (ZSTD_isError(res))
+ fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+ ((char *)output.dst)[output.pos] = '\0';
+ ahwrite(output.dst, 1, output.pos, AH);
+ }
+#endif
+
+ pg_free(unconstify(void *, input.src));
+ pg_free(output.dst);
+}
+
+#endif /* HAVE_LIBZSTD */
#ifdef HAVE_LIBZ
+
/*
* Functions for zlib compressed output.
*/
@@ -422,6 +643,19 @@ struct cfp
#ifdef HAVE_LIBZ
gzFile compressedfp;
#endif
+
+#ifdef HAVE_LIBZSTD // XXX: this should be a union with a CompressionAlgorithm alg?
+ /* This is a normal file to which we read/write compressed data */
+ struct {
+ FILE *fp;
+ // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ } zstd;
+#endif
+
};
#ifdef HAVE_LIBZ
@@ -449,24 +683,25 @@ free_keep_errno(void *p)
* On failure, return NULL with an error code in errno.
*/
cfp *
-cfopen_read(const char *path, const char *mode)
+cfopen_read(const char *path, const char *mode, int compression)
{
cfp *fp;
#ifdef HAVE_LIBZ
- if (hasSuffix(path, ".gz"))
- fp = cfopen(path, mode, 1);
+ if (hasSuffix(path, ".gz") || hasSuffix(path, ".zst"))
+ fp = cfopen(path, mode, compression);
else
#endif
{
- fp = cfopen(path, mode, 0);
+ fp = cfopen(path, mode, compression);
#ifdef HAVE_LIBZ
if (fp == NULL)
{
char *fname;
+ char *suffix = compression == ZSTD_COMPRESSION ? "zst" : "gz";
- fname = psprintf("%s.gz", path);
- fp = cfopen(fname, mode, 1);
+ fname = psprintf("%s.%s", path, suffix);
+ fp = cfopen(fname, mode, compression);
free_keep_errno(fname);
}
#endif
@@ -491,13 +726,14 @@ cfopen_write(const char *path, const char *mode, int compression)
cfp *fp;
if (compression == 0)
- fp = cfopen(path, mode, 0);
+ fp = cfopen(path, mode, compression);
else
{
-#ifdef HAVE_LIBZ
+#ifdef HAVE_LIBZ // XXX
char *fname;
+ char *suffix = compression == ZSTD_COMPRESSION ? "zst" : "gz";
- fname = psprintf("%s.gz", path);
+ fname = psprintf("%s.%s", path, suffix);
fp = cfopen(fname, mode, compression);
free_keep_errno(fname);
#else
@@ -505,11 +741,12 @@ cfopen_write(const char *path, const char *mode, int compression)
fp = NULL; /* keep compiler quiet */
#endif
}
+
return fp;
}
/*
- * Opens file 'path' in 'mode'. If 'compression' is non-zero, the file
+ * Opens file 'path' in 'mode'. If 'alg' is COMPR_ALG_ZLIB, the file
* is opened with libz gzopen(), otherwise with plain fopen().
*
* On failure, return NULL with an error code in errno.
@@ -519,9 +756,19 @@ cfopen(const char *path, const char *mode, int compression)
{
cfp *fp = pg_malloc(sizeof(cfp));
- if (compression != 0)
+ fp->uncompressedfp = NULL;
+#ifdef HAVE_LIBZ
+ fp->compressedfp = NULL;
+#endif
+#ifdef HAVE_LIBZSTD
+ fp->zstd.fp = NULL;
+#endif
+
+ switch (compression)
{
#ifdef HAVE_LIBZ
+ case 1 ... 9: // XXX: nonportable
+ case Z_DEFAULT_COMPRESSION:
if (compression != Z_DEFAULT_COMPRESSION)
{
/* user has specified a compression level, so tell zlib to use it */
@@ -537,30 +784,57 @@ cfopen(const char *path, const char *mode, int compression)
fp->compressedfp = gzopen(path, mode);
}
- fp->uncompressedfp = NULL;
if (fp->compressedfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
}
-#else
- fatal("not built with zlib support");
+
+ return fp;
+ break;
#endif
- }
- else
- {
-#ifdef HAVE_LIBZ
- fp->compressedfp = NULL;
+
+#ifdef HAVE_LIBZSTD
+ case ZSTD_COMPRESSION:
+ fp->zstd.fp = fopen(path, mode);
+ // XXX: save the compression params
+ if (fp->zstd.fp == NULL)
+ {
+ free_keep_errno(fp);
+ fp = NULL;
+ }
+ else if (strchr(mode, 'w'))
+ {
+ fp->zstd.dstream = NULL;
+ fp->zstd.output.size = ZSTD_CStreamOutSize(); // XXX
+ fp->zstd.output.dst = pg_malloc0(fp->zstd.output.size);
+ fp->zstd.cstream = ZSTD_createCStream();
+ if (fp->zstd.cstream == NULL)
+ fatal("could not initialize compression library");
+ }
+ else if (strchr(mode, 'r'))
+ {
+ fp->zstd.cstream = NULL;
+ fp->zstd.input.size = ZSTD_DStreamOutSize(); // XXX
+ fp->zstd.input.src = pg_malloc0(fp->zstd.input.size);
+ fp->zstd.dstream = ZSTD_createDStream();
+ if (fp->zstd.dstream == NULL)
+ fatal("could not initialize compression library");
+ } // XXX else: bad mode
+ return fp;
+ break;
#endif
+
+ default:
fp->uncompressedfp = fopen(path, mode);
if (fp->uncompressedfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
}
- }
- return fp;
+ return fp;
+ }
}
@@ -587,6 +861,44 @@ cfread(void *ptr, int size, cfp *fp)
}
else
#endif
+
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ ZSTD_outBuffer *output = &fp->zstd.output;
+ ZSTD_inBuffer *input = &fp->zstd.input;
+ size_t input_size = ZSTD_DStreamInSize();
+ size_t res, cnt;
+
+ output->size = size;
+ output->dst = ptr;
+ output->pos = 0;
+
+ /* read compressed data */
+ while ((cnt = fread(unconstify(void *, input->src), 1, input_size, fp->zstd.fp)))
+ {
+ input->size = cnt;
+ input->pos = 0;
+
+ for ( ; input->pos < input->size; )
+ {
+ /* decompress */
+ res = ZSTD_decompressStream(fp->zstd.dstream, output, input);
+ if (res == 0 || output->pos == output->size)
+ break;
+ if (ZSTD_isError(res))
+ fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+ }
+
+ if (output->pos == output->size)
+ break; /* We read all the data that fits */
+ }
+
+ ret = output->pos;
+ }
+ else
+#endif
+
{
ret = fread(ptr, 1, size, fp->uncompressedfp);
if (ret != size && !feof(fp->uncompressedfp))
@@ -603,6 +915,35 @@ cfwrite(const void *ptr, int size, cfp *fp)
return gzwrite(fp->compressedfp, ptr, size);
else
#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ size_t res, cnt;
+ ZSTD_outBuffer *output = &fp->zstd.output;
+ ZSTD_inBuffer *input = &fp->zstd.input;
+
+ input->src = ptr;
+ input->size = size;
+ input->pos = 0;
+
+ /* Consume all input, and flush later */
+ while (input->pos != input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_continue);
+ if (ZSTD_isError(res))
+ fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ if (cnt != output->pos)
+ fatal("could not write data: %s", strerror(errno));
+ }
+
+ return size;
+ }
+ else
+#endif
+
return fwrite(ptr, 1, size, fp->uncompressedfp);
}
@@ -625,6 +966,20 @@ cfgetc(cfp *fp)
}
else
#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ if (cfread(&ret, 1, fp) != 1)
+ {
+ if (feof(fp->zstd.fp))
+ fatal("could not read from input file: end of file");
+ else
+ fatal("could not read from input file: %s", strerror(errno));
+ }
+fprintf(stderr, "cfgetc %d\n", ret);
+ }
+#endif
+
{
ret = fgetc(fp->uncompressedfp);
if (ret == EOF)
@@ -641,6 +996,18 @@ cfgets(cfp *fp, char *buf, int len)
if (fp->compressedfp)
return gzgets(fp->compressedfp, buf, len);
else
+#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ int res;
+ res = cfread(buf, len, fp);
+ buf[res] = 0;
+ if (strchr(buf, '\n'))
+ *strchr(buf, '\n') = '\0';
+ return buf;
+ }
+ else
#endif
return fgets(buf, len, fp->uncompressedfp);
}
@@ -662,6 +1029,43 @@ cfclose(cfp *fp)
fp->compressedfp = NULL;
}
else
+#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ ZSTD_outBuffer *output = &fp->zstd.output;
+ ZSTD_inBuffer *input = &fp->zstd.input;
+ size_t res, cnt;
+
+ if (fp->zstd.cstream)
+ {
+ for (;;)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_end);
+ if (ZSTD_isError(res))
+ fatal("could not compress data: %s", ZSTD_getErrorName(res));
+ cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ if (cnt != output->pos)
+ fatal("could not write data: %s", strerror(errno));
+ if (res == 0)
+ break;
+ }
+
+ ZSTD_freeCStream(fp->zstd.cstream);
+ pg_free(fp->zstd.output.dst);
+ }
+
+ if (fp->zstd.dstream)
+ {
+ ZSTD_freeDStream(fp->zstd.dstream);
+ pg_free(unconstify(void *, fp->zstd.input.src));
+ }
+
+ result = fclose(fp->zstd.fp);
+ fp->zstd.fp = NULL;
+ }
+ else
#endif
{
result = fclose(fp->uncompressedfp);
@@ -679,6 +1083,11 @@ cfeof(cfp *fp)
if (fp->compressedfp)
return gzeof(fp->compressedfp);
else
+#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ return feof(fp->zstd.fp);
+ else
#endif
return feof(fp->uncompressedfp);
}
diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h
index d2e6e1b854..f0ce06f176 100644
--- a/src/bin/pg_dump/compress_io.h
+++ b/src/bin/pg_dump/compress_io.h
@@ -24,7 +24,8 @@
typedef enum
{
COMPR_ALG_NONE,
- COMPR_ALG_LIBZ
+ COMPR_ALG_LIBZ,
+ COMPR_ALG_ZSTD,
} CompressionAlgorithm;
/* Prototype for callback function to WriteDataToArchive() */
@@ -57,7 +58,7 @@ extern void EndCompressor(ArchiveHandle *AH, CompressorState *cs);
typedef struct cfp cfp;
extern cfp *cfopen(const char *path, const char *mode, int compression);
-extern cfp *cfopen_read(const char *path, const char *mode);
+extern cfp *cfopen_read(const char *path, const char *mode, int compression);
extern cfp *cfopen_write(const char *path, const char *mode, int compression);
extern int cfread(void *ptr, int size, cfp *fp);
extern int cfwrite(const void *ptr, int size, cfp *fp);
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 6a5a22637b..1a229ebedb 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -60,6 +60,11 @@ typedef struct _z_stream
typedef z_stream *z_streamp;
#endif
+#ifdef HAVE_LIBZSTD
+#include <zstd.h>
+#define ZSTD_COMPRESSION -2
+#endif /* HAVE_LIBZSTD */
+
/* Data block types */
#define BLK_DATA 1
#define BLK_BLOBS 3
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 650b542fce..6e55cb425e 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -202,7 +202,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH)
setFilePath(AH, fname, "toc.dat");
- tocFH = cfopen_read(fname, PG_BINARY_R);
+ tocFH = cfopen_read(fname, PG_BINARY_R, AH->compression);
if (tocFH == NULL)
fatal("could not open input file \"%s\": %m", fname);
@@ -388,7 +388,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename)
if (!filename)
return;
- cfp = cfopen_read(filename, PG_BINARY_R);
+ cfp = cfopen_read(filename, PG_BINARY_R, AH->compression);
if (!cfp)
fatal("could not open input file \"%s\": %m", filename);
@@ -440,7 +440,7 @@ _LoadBlobs(ArchiveHandle *AH)
setFilePath(AH, fname, "blobs.toc");
- ctx->blobsTocFH = cfopen_read(fname, PG_BINARY_R);
+ ctx->blobsTocFH = cfopen_read(fname, PG_BINARY_R, AH->compression);
if (ctx->blobsTocFH == NULL)
fatal("could not open large object TOC file \"%s\" for input: %m",
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 8b1e5cc2b5..f21eb021c7 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -533,7 +533,8 @@ main(int argc, char **argv)
case 'Z': /* Compression Level */
compressLevel = atoi(optarg);
- if (compressLevel < 0 || compressLevel > 9)
+ if ((compressLevel < 0 || compressLevel > 9) &&
+ compressLevel != -2)
{
pg_log_error("compression level must be in range 0..9");
exit_nicely(1);
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index de8f838e53..da35415c72 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -346,6 +346,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if you have the `link' function. */
#undef HAVE_LINK
--
2.17.0
[text/x-diff] 0004-cmdline-parser-for-compression-alg-level-opts.patch (37.7K, ../../20201222023235.GJ30237@telsasoft.com/5-0004-cmdline-parser-for-compression-alg-level-opts.patch)
download | inline diff:
From 4b87bdba077da1e1609a1d7dbb40772d432c28dd Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 12 Dec 2020 00:40:39 -0600
Subject: [PATCH 4/7] cmdline parser for compression alg/level/opts
---
src/bin/pg_dump/compress_io.c | 235 ++++++++++++--------------
src/bin/pg_dump/compress_io.h | 20 +--
src/bin/pg_dump/pg_backup.h | 19 ++-
src/bin/pg_dump/pg_backup_archiver.c | 45 ++---
src/bin/pg_dump/pg_backup_archiver.h | 11 +-
src/bin/pg_dump/pg_backup_custom.c | 6 +-
src/bin/pg_dump/pg_backup_directory.c | 19 ++-
src/bin/pg_dump/pg_backup_tar.c | 33 ++--
src/bin/pg_dump/pg_dump.c | 157 ++++++++++++++---
9 files changed, 324 insertions(+), 221 deletions(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index b51ba680a2..421b8e04cb 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -86,12 +86,9 @@ struct CompressorState
};
-static void ParseCompressionOption(int compression, CompressionAlgorithm *alg,
- int *level);
-
/* Routines that support zlib compressed data I/O */
#ifdef HAVE_LIBZ
-static void InitCompressorZlib(CompressorState *cs, int level);
+static void InitCompressorZlib(CompressorState *cs, Compress *compress);
static void DeflateCompressorZlib(ArchiveHandle *AH, CompressorState *cs,
bool flush);
static void ReadDataFromArchiveZlib(ArchiveHandle *AH, ReadFunc readF);
@@ -101,9 +98,9 @@ static void EndCompressorZlib(ArchiveHandle *AH, CompressorState *cs);
#endif
#ifdef HAVE_LIBZSTD
-static void InitCompressorZstd(CompressorState *cs, int level);
+static ZSTD_CStream *ZstdCStreamParams(Compress *compress);
+static void InitCompressorZstd(CompressorState *cs, Compress *compress);
static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
-static void DeflateCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
const char *data, size_t dLen);
static void ReadDataFromArchiveZstd(ArchiveHandle *AH, ReadFunc readF);
@@ -114,80 +111,40 @@ static void ReadDataFromArchiveNone(ArchiveHandle *AH, ReadFunc readF);
static void WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
const char *data, size_t dLen);
-/*
- * Interprets a numeric 'compression' value. The algorithm implied by the
- * value (zlib or none at the moment), is returned in *alg, and the
- * zlib compression level in *level.
- */
-static void
-ParseCompressionOption(int compression, CompressionAlgorithm *alg, int *level)
-{
- switch (compression)
- {
-#ifdef HAVE_LIBZSTD
- case ZSTD_COMPRESSION:
- *alg = COMPR_ALG_ZSTD;
- break;
-#endif
-#ifdef HAVE_ZLIB
- case Z_DEFAULT_COMPRESSION:
- case 1..9:
- *alg = COMPR_ALG_LIBZ;
- break;
-#endif
- case 0:
- *alg = COMPR_ALG_NONE;
- break;
- default:
- fatal("invalid compression code: %d", compression);
- *alg = COMPR_ALG_NONE; /* keep compiler quiet */
- }
-
- /* The level is just the passed-in value. */
- if (level)
- *level = compression;
-}
-
/* Public interface routines */
/* Allocate a new compressor */
CompressorState *
-AllocateCompressor(int compression, WriteFunc writeF)
+AllocateCompressor(Compress *compression, WriteFunc writeF)
{
CompressorState *cs;
- CompressionAlgorithm alg;
- int level;
-
- ParseCompressionOption(compression, &alg, &level);
-
-#ifndef HAVE_LIBZ
- if (alg == COMPR_ALG_LIBZ)
- fatal("not built with zlib support");
-#endif
cs = (CompressorState *) pg_malloc0(sizeof(CompressorState));
cs->writeF = writeF;
- cs->comprAlg = alg;
+ cs->comprAlg = compression->alg;
/*
* Perform compression algorithm specific initialization.
*/
- switch (alg)
+ Assert (compression->alg != COMPR_ALG_DEFAULT);
+ switch (compression->alg)
{
#ifdef HAVE_LIBZ
case COMPR_ALG_LIBZ:
- InitCompressorZlib(cs, level);
+ InitCompressorZlib(cs, compression);
break;
#endif
#ifdef HAVE_LIBZSTD
case COMPR_ALG_ZSTD:
- InitCompressorZstd(cs, level);
+ InitCompressorZstd(cs, compression);
break;
#endif
case COMPR_ALG_NONE:
/* Do nothing */
break;
- // default:
+ default:
+ /* Should not happen */
+ fatal("requested compression not available in this installation");
}
return cs;
@@ -198,12 +155,9 @@ AllocateCompressor(int compression, WriteFunc writeF)
* out with ahwrite().
*/
void
-ReadDataFromArchive(ArchiveHandle *AH, int compression, ReadFunc readF)
+ReadDataFromArchive(ArchiveHandle *AH, ReadFunc readF)
{
- CompressionAlgorithm alg;
-
- ParseCompressionOption(compression, &alg, NULL);
-
+ CompressionAlgorithm alg = AH->compression.alg;
if (alg == COMPR_ALG_NONE)
ReadDataFromArchiveNone(AH, readF);
else if (alg == COMPR_ALG_LIBZ)
@@ -233,25 +187,25 @@ WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs,
{
switch (cs->comprAlg)
{
- case COMPR_ALG_LIBZ:
#ifdef HAVE_LIBZ
+ case COMPR_ALG_LIBZ:
WriteDataToArchiveZlib(AH, cs, data, dLen);
-#else
- fatal("not built with zlib support");
-#endif
break;
+#endif
- case COMPR_ALG_ZSTD:
#ifdef HAVE_LIBZSTD
+ case COMPR_ALG_ZSTD:
WriteDataToArchiveZstd(AH, cs, data, dLen);
-#else
- fatal("not built with zstd support");
-#endif
break;
+#endif
case COMPR_ALG_NONE:
WriteDataToArchiveNone(AH, cs, data, dLen);
break;
+
+ default:
+ /* Should not happen */
+ fatal("requested compression not available in this installation");
}
}
@@ -277,13 +231,42 @@ EndCompressor(ArchiveHandle *AH, CompressorState *cs)
// XXX: put in separate files ?
#ifdef HAVE_LIBZSTD
-static void
-InitCompressorZstd(CompressorState *cs, int level)
+static ZSTD_CStream*
+ZstdCStreamParams(Compress *compress)
{
- cs->u.zstd.cstream = ZSTD_createCStream();
- if (cs->u.zstd.cstream == NULL)
+ size_t res;
+ ZSTD_CStream *cstream;
+
+ cstream = ZSTD_createCStream();
+ if (cstream == NULL)
fatal("could not initialize compression library");
+ if (compress->level != 0) // XXX: ZSTD_CLEVEL_DEFAULT
+ {
+ res = ZSTD_CCtx_setParameter(cstream,
+ ZSTD_c_compressionLevel, compress->level);
+ if (ZSTD_isError(res))
+ fatal("could not set compression level: %s",
+ ZSTD_getErrorName(res));
+ }
+
+ if (compress->zstdlong)
+ {
+ res = ZSTD_CCtx_setParameter(cstream,
+ ZSTD_c_enableLongDistanceMatching, 1); // XXX: allow to disable it
+ if (ZSTD_isError(res))
+ fatal("could not set compression parameter: %s",
+ ZSTD_getErrorName(res));
+ }
+
+ return cstream;
+}
+
+static void
+InitCompressorZstd(CompressorState *cs, Compress *compress)
+{
+ cs->u.zstd.cstream = ZstdCStreamParams(compress);
+
/* XXX: initialize safely like the corresponding zlib "paranoia" */
cs->u.zstd.output.size = ZSTD_CStreamOutSize();
cs->u.zstd.output.dst = pg_malloc(cs->u.zstd.output.size);
@@ -295,6 +278,8 @@ EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
{
ZSTD_outBuffer *output = &cs->u.zstd.output;
+
+
for (;;)
{
size_t res;
@@ -318,11 +303,16 @@ EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
}
static void
-DeflateCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const char *data, size_t dLen)
{
ZSTD_inBuffer *input = &cs->u.zstd.input;
ZSTD_outBuffer *output = &cs->u.zstd.output;
+ input->src = (void *) unconstify(char *, data);
+ input->size = dLen;
+ input->pos = 0;
+
while (input->pos != input->size)
{
size_t res;
@@ -349,16 +339,6 @@ DeflateCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
}
}
-static void
-WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
- const char *data, size_t dLen)
-{
- cs->u.zstd.input.src = (void *) unconstify(char *, data);
- cs->u.zstd.input.size = dLen;
- cs->u.zstd.input.pos = 0;
- DeflateCompressorZstd(AH, cs);
-}
-
/* Read data from a compressed zstd archive */
static void
ReadDataFromArchiveZstd(ArchiveHandle *AH, ReadFunc readF)
@@ -436,7 +416,7 @@ ReadDataFromArchiveZstd(ArchiveHandle *AH, ReadFunc readF)
*/
static void
-InitCompressorZlib(CompressorState *cs, int level)
+InitCompressorZlib(CompressorState *cs, Compress *compress)
{
z_streamp zp;
@@ -453,7 +433,7 @@ InitCompressorZlib(CompressorState *cs, int level)
cs->zlibOut = (char *) pg_malloc(ZLIB_OUT_SIZE + 1);
cs->zlibOutSize = ZLIB_OUT_SIZE;
- if (deflateInit(zp, level) != Z_OK)
+ if (deflateInit(zp, compress->level) != Z_OK)
fatal("could not initialize compression library: %s",
zp->msg);
@@ -676,35 +656,31 @@ free_keep_errno(void *p)
* Open a file for reading. 'path' is the file to open, and 'mode' should
* be either "r" or "rb".
*
- * If the file at 'path' does not exist, we append the ".gz" suffix (if 'path'
- * doesn't already have it) and try again. So if you pass "foo" as 'path',
+ * If the file at 'path' does not exist, we search with compressed suffix (if 'path'
+ * doesn't already have one) and try again. So if you pass "foo" as 'path',
* this will open either "foo" or "foo.gz".
*
* On failure, return NULL with an error code in errno.
*/
cfp *
-cfopen_read(const char *path, const char *mode, int compression)
+cfopen_read(const char *path, const char *mode, Compress *compression)
{
cfp *fp;
-#ifdef HAVE_LIBZ
if (hasSuffix(path, ".gz") || hasSuffix(path, ".zst"))
fp = cfopen(path, mode, compression);
else
-#endif
{
fp = cfopen(path, mode, compression);
-#ifdef HAVE_LIBZ
if (fp == NULL)
{
char *fname;
- char *suffix = compression == ZSTD_COMPRESSION ? "zst" : "gz";
+ const char *suffix = compress_suffix(compression);
- fname = psprintf("%s.%s", path, suffix);
+ fname = psprintf("%s%s", path, suffix);
fp = cfopen(fname, mode, compression);
free_keep_errno(fname);
}
-#endif
}
return fp;
}
@@ -714,32 +690,26 @@ cfopen_read(const char *path, const char *mode, int compression)
* be a filemode as accepted by fopen() and gzopen() that indicates writing
* ("w", "wb", "a", or "ab").
*
- * If 'compression' is non-zero, a gzip compressed stream is opened, and
- * 'compression' indicates the compression level used. The ".gz" suffix
- * is automatically added to 'path' in that case.
+ * Use compression if specified.
+ * The appropriate suffix is automatically added to 'path' in that case.
*
* On failure, return NULL with an error code in errno.
*/
cfp *
-cfopen_write(const char *path, const char *mode, int compression)
+cfopen_write(const char *path, const char *mode, Compress *compression)
{
cfp *fp;
- if (compression == 0)
+ if (compression->alg == COMPR_ALG_NONE)
fp = cfopen(path, mode, compression);
else
{
-#ifdef HAVE_LIBZ // XXX
char *fname;
- char *suffix = compression == ZSTD_COMPRESSION ? "zst" : "gz";
+ const char *suffix = compress_suffix(compression);
- fname = psprintf("%s.%s", path, suffix);
+ fname = psprintf("%s%s", path, suffix);
fp = cfopen(fname, mode, compression);
free_keep_errno(fname);
-#else
- fatal("not built with zlib support");
- fp = NULL; /* keep compiler quiet */
-#endif
}
return fp;
@@ -752,7 +722,7 @@ cfopen_write(const char *path, const char *mode, int compression)
* On failure, return NULL with an error code in errno.
*/
cfp *
-cfopen(const char *path, const char *mode, int compression)
+cfopen(const char *path, const char *mode, Compress *compression)
{
cfp *fp = pg_malloc(sizeof(cfp));
@@ -764,18 +734,17 @@ cfopen(const char *path, const char *mode, int compression)
fp->zstd.fp = NULL;
#endif
- switch (compression)
+ switch (compression->alg)
{
#ifdef HAVE_LIBZ
- case 1 ... 9: // XXX: nonportable
- case Z_DEFAULT_COMPRESSION:
- if (compression != Z_DEFAULT_COMPRESSION)
+ case COMPR_ALG_LIBZ:
+ if (compression->level != Z_DEFAULT_COMPRESSION)
{
/* user has specified a compression level, so tell zlib to use it */
char mode_compression[32];
snprintf(mode_compression, sizeof(mode_compression), "%s%d",
- mode, compression);
+ mode, compression->level);
fp->compressedfp = gzopen(path, mode_compression);
}
else
@@ -795,9 +764,8 @@ cfopen(const char *path, const char *mode, int compression)
#endif
#ifdef HAVE_LIBZSTD
- case ZSTD_COMPRESSION:
+ case COMPR_ALG_ZSTD:
fp->zstd.fp = fopen(path, mode);
- // XXX: save the compression params
if (fp->zstd.fp == NULL)
{
free_keep_errno(fp);
@@ -806,16 +774,14 @@ cfopen(const char *path, const char *mode, int compression)
else if (strchr(mode, 'w'))
{
fp->zstd.dstream = NULL;
- fp->zstd.output.size = ZSTD_CStreamOutSize(); // XXX
+ fp->zstd.output.size = ZSTD_CStreamOutSize();
fp->zstd.output.dst = pg_malloc0(fp->zstd.output.size);
- fp->zstd.cstream = ZSTD_createCStream();
- if (fp->zstd.cstream == NULL)
- fatal("could not initialize compression library");
+ fp->zstd.cstream = ZstdCStreamParams(compression);
}
else if (strchr(mode, 'r'))
{
fp->zstd.cstream = NULL;
- fp->zstd.input.size = ZSTD_DStreamOutSize(); // XXX
+ fp->zstd.input.size = ZSTD_DStreamInSize();
fp->zstd.input.src = pg_malloc0(fp->zstd.input.size);
fp->zstd.dstream = ZSTD_createDStream();
if (fp->zstd.dstream == NULL)
@@ -825,15 +791,19 @@ cfopen(const char *path, const char *mode, int compression)
break;
#endif
- default:
+ case COMPR_ALG_NONE:
fp->uncompressedfp = fopen(path, mode);
if (fp->uncompressedfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
}
-
return fp;
+ break;
+
+ default:
+ /* should not happen */
+ fatal("requested compression not available in this installation");
}
}
@@ -944,7 +914,7 @@ cfwrite(const void *ptr, int size, cfp *fp)
else
#endif
- return fwrite(ptr, 1, size, fp->uncompressedfp);
+ return fwrite(ptr, 1, size, fp->uncompressedfp);
}
int
@@ -1005,7 +975,7 @@ cfgets(cfp *fp, char *buf, int len)
buf[res] = 0;
if (strchr(buf, '\n'))
*strchr(buf, '\n') = '\0';
- return buf;
+ return res > 0 ? buf : 0;
}
else
#endif
@@ -1122,5 +1092,22 @@ hasSuffix(const char *filename, const char *suffix)
suffix,
suffixlen) == 0;
}
-
#endif
+
+/*
+ * Return a string for the given AH's compression.
+ * The string is statically allocated.
+ */
+const char *
+compress_suffix(Compress *compression)
+{
+ switch (compression->alg)
+ {
+ case COMPR_ALG_LIBZ:
+ return ".gz";
+ case COMPR_ALG_ZSTD:
+ return ".zst";
+ default:
+ return "";
+ }
+}
diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h
index f0ce06f176..2c073676eb 100644
--- a/src/bin/pg_dump/compress_io.h
+++ b/src/bin/pg_dump/compress_io.h
@@ -21,13 +21,6 @@
#define ZLIB_OUT_SIZE 4096
#define ZLIB_IN_SIZE 4096
-typedef enum
-{
- COMPR_ALG_NONE,
- COMPR_ALG_LIBZ,
- COMPR_ALG_ZSTD,
-} CompressionAlgorithm;
-
/* Prototype for callback function to WriteDataToArchive() */
typedef void (*WriteFunc) (ArchiveHandle *AH, const char *buf, size_t len);
@@ -47,8 +40,8 @@ typedef size_t (*ReadFunc) (ArchiveHandle *AH, char **buf, size_t *buflen);
/* struct definition appears in compress_io.c */
typedef struct CompressorState CompressorState;
-extern CompressorState *AllocateCompressor(int compression, WriteFunc writeF);
-extern void ReadDataFromArchive(ArchiveHandle *AH, int compression,
+extern CompressorState *AllocateCompressor(Compress *compression, WriteFunc writeF);
+extern void ReadDataFromArchive(ArchiveHandle *AH,
ReadFunc readF);
extern void WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs,
const void *data, size_t dLen);
@@ -57,9 +50,9 @@ extern void EndCompressor(ArchiveHandle *AH, CompressorState *cs);
typedef struct cfp cfp;
-extern cfp *cfopen(const char *path, const char *mode, int compression);
-extern cfp *cfopen_read(const char *path, const char *mode, int compression);
-extern cfp *cfopen_write(const char *path, const char *mode, int compression);
+extern cfp *cfopen(const char *path, const char *mode, Compress *compression);
+extern cfp *cfopen_read(const char *path, const char *mode, Compress *compression);
+extern cfp *cfopen_write(const char *path, const char *mode, Compress *compression);
extern int cfread(void *ptr, int size, cfp *fp);
extern int cfwrite(const void *ptr, int size, cfp *fp);
extern int cfgetc(cfp *fp);
@@ -68,4 +61,7 @@ extern int cfclose(cfp *fp);
extern int cfeof(cfp *fp);
extern const char *get_cfp_error(cfp *fp);
+/* also used by tar */
+extern const char * compress_suffix(Compress *compression);
+
#endif
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 9d0056a569..1c1954dd1a 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -72,6 +72,21 @@ typedef struct _connParams
char *override_dbname;
} ConnParams;
+typedef enum
+{
+ COMPR_ALG_DEFAULT = -1,
+ COMPR_ALG_NONE,
+ COMPR_ALG_LIBZ,
+ COMPR_ALG_ZSTD,
+} CompressionAlgorithm;
+
+typedef struct Compress {
+ CompressionAlgorithm alg;
+ int level; // XXX: default
+ bool zstdlong;
+} Compress;
+
+
typedef struct _restoreOptions
{
int createDB; /* Issue commands to create the database */
@@ -125,7 +140,7 @@ typedef struct _restoreOptions
int noDataForFailedTables;
int exit_on_error;
- int compression;
+ Compress compression;
int suppressDumpWarnings; /* Suppress output of WARNING entries
* to stderr */
bool single_txn;
@@ -281,7 +296,7 @@ extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
/* Create a new archive */
extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupDumpWorker);
/* The --list option */
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 1f82c6499b..2f1d59ce7a 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -70,7 +70,7 @@ typedef struct _parallelReadyList
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupWorkerPtr);
static void _getObjectDescription(PQExpBuffer buf, TocEntry *te);
static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, bool isData);
@@ -98,7 +98,7 @@ static int _discoverArchiveFormat(ArchiveHandle *AH);
static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
-static void SetOutput(ArchiveHandle *AH, const char *filename, int compression);
+static void SetOutput(ArchiveHandle *AH, const char *filename, Compress *compress);
static OutputContext SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, OutputContext savedContext);
@@ -238,7 +238,7 @@ setupRestoreWorker(Archive *AHX)
/* Public */
Archive *
CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupDumpWorker)
{
@@ -253,7 +253,9 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
Archive *
OpenArchive(const char *FileSpec, const ArchiveFormat fmt)
{
- ArchiveHandle *AH = _allocAH(FileSpec, fmt, 0, true, archModeRead, setupRestoreWorker);
+ Compress compress = {0};
+ ArchiveHandle *AH = _allocAH(FileSpec, fmt, &compress, true, archModeRead,
+ setupRestoreWorker);
return (Archive *) AH;
}
@@ -382,7 +384,7 @@ RestoreArchive(Archive *AHX)
* Make sure we won't need (de)compression we haven't got
*/
#ifndef HAVE_LIBZ
- if (AH->compression != 0 && AH->PrintTocDataPtr != NULL)
+ if (AH->compression.alg != COMPR_ALG_NONE && AH->PrintTocDataPtr != NULL)
{
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
@@ -457,8 +459,8 @@ RestoreArchive(Archive *AHX)
* Setup the output file if necessary.
*/
sav = SaveOutput(AH);
- if (ropt->filename || ropt->compression)
- SetOutput(AH, ropt->filename, ropt->compression);
+ if (ropt->filename || ropt->compression.alg != COMPR_ALG_NONE)
+ SetOutput(AH, ropt->filename, &ropt->compression);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -738,7 +740,7 @@ RestoreArchive(Archive *AHX)
*/
AH->stage = STAGE_FINALIZING;
- if (ropt->filename || ropt->compression)
+ if (ropt->filename || ropt->compression.alg != COMPR_ALG_NONE)
RestoreOutput(AH, sav);
if (ropt->useDB)
@@ -1121,10 +1123,11 @@ PrintTOCSummary(Archive *AHX)
OutputContext sav;
const char *fmtName;
char stamp_str[64];
+ Compress nocompression = {0};
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, 0 /* no compression */ );
+ SetOutput(AH, ropt->filename, &nocompression);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1133,7 +1136,7 @@ PrintTOCSummary(Archive *AHX)
ahprintf(AH, ";\n; Archive created at %s\n", stamp_str);
ahprintf(AH, "; dbname: %s\n; TOC Entries: %d\n; Compression: %d\n",
sanitize_line(AH->archdbname, false),
- AH->tocCount, AH->compression);
+ AH->tocCount, AH->compression.alg);
switch (AH->format)
{
@@ -1487,7 +1490,7 @@ archprintf(Archive *AH, const char *fmt,...)
*******************************/
static void
-SetOutput(ArchiveHandle *AH, const char *filename, int compression)
+SetOutput(ArchiveHandle *AH, const char *filename, Compress *compression)
{
int fn;
@@ -1510,12 +1513,12 @@ SetOutput(ArchiveHandle *AH, const char *filename, int compression)
/* If compression explicitly requested, use gzopen */
#ifdef HAVE_LIBZ
- if (compression != 0)
+ if (compression->alg != COMPR_ALG_NONE)
{
char fmode[14];
/* Don't use PG_BINARY_x since this is zlib */
- sprintf(fmode, "wb%d", compression);
+ sprintf(fmode, "wb%d", compression->level);
if (fn >= 0)
AH->OF = gzdopen(dup(fn), fmode);
else
@@ -2259,7 +2262,7 @@ _discoverArchiveFormat(ArchiveHandle *AH)
*/
static ArchiveHandle *
_allocAH(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupWorkerPtr)
{
ArchiveHandle *AH;
@@ -2310,7 +2313,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
AH->toc->prev = AH->toc;
AH->mode = mode;
- AH->compression = compression;
+ AH->compression = *compression;
AH->dosync = dosync;
memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse));
@@ -2325,7 +2328,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
* Force stdin/stdout into binary mode if that is what we are using.
*/
#ifdef WIN32
- if ((fmt != archNull || compression != 0) &&
+ if ((fmt != archNull || compression->alg != COMPR_ALG_NONE) &&
(AH->fSpec == NULL || strcmp(AH->fSpec, "") == 0))
{
if (mode == archModeWrite)
@@ -3741,7 +3744,7 @@ WriteHead(ArchiveHandle *AH)
AH->WriteBytePtr(AH, AH->intSize);
AH->WriteBytePtr(AH, AH->offSize);
AH->WriteBytePtr(AH, AH->format);
- WriteInt(AH, AH->compression);
+ WriteInt(AH, AH->compression.alg);
crtm = *localtime(&AH->createDate);
WriteInt(AH, crtm.tm_sec);
WriteInt(AH, crtm.tm_min);
@@ -3816,15 +3819,15 @@ ReadHead(ArchiveHandle *AH)
if (AH->version >= K_VERS_1_2)
{
if (AH->version < K_VERS_1_4)
- AH->compression = AH->ReadBytePtr(AH);
+ AH->compression.alg = AH->ReadBytePtr(AH);
else
- AH->compression = ReadInt(AH);
+ AH->compression.alg = ReadInt(AH);
}
else
- AH->compression = Z_DEFAULT_COMPRESSION;
+ AH->compression.alg = Z_DEFAULT_COMPRESSION;
#ifndef HAVE_LIBZ
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
pg_log_warning("archive is compressed, but this installation does not support compression -- no data will be available");
#endif
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 1a229ebedb..641ba2d043 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -47,8 +47,6 @@
#define GZWRITE(p, s, n, fh) (fwrite(p, s, n, fh) * (s))
#define GZREAD(p, s, n, fh) fread(p, s, n, fh)
#define GZEOF(fh) feof(fh)
-/* this is just the redefinition of a libz constant */
-#define Z_DEFAULT_COMPRESSION (-1)
typedef struct _z_stream
{
@@ -62,7 +60,6 @@ typedef z_stream *z_streamp;
#ifdef HAVE_LIBZSTD
#include <zstd.h>
-#define ZSTD_COMPRESSION -2
#endif /* HAVE_LIBZSTD */
/* Data block types */
@@ -334,13 +331,7 @@ struct _archiveHandle
DumpId *tableDataId; /* TABLE DATA ids, indexed by table dumpId */
struct _tocEntry *currToc; /* Used when dumping data */
- int compression; /*---------
- * Compression requested on open
- * Possible values for compression:
- * -2 ZSTD_COMPRESSION
- * -1 Z_DEFAULT_COMPRESSION
- * 0 COMPRESSION_NONE
- * 1-9 levels for gzip compression */
+ Compress compression; /* Compression requested on open */
bool dosync; /* data requested to be synced on sight */
ArchiveMode mode; /* File mode - r or w */
void *formatData; /* Header data specific to file format */
diff --git a/src/bin/pg_dump/pg_backup_custom.c b/src/bin/pg_dump/pg_backup_custom.c
index 77d402c323..55a887a236 100644
--- a/src/bin/pg_dump/pg_backup_custom.c
+++ b/src/bin/pg_dump/pg_backup_custom.c
@@ -298,7 +298,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te)
_WriteByte(AH, BLK_DATA); /* Block type */
WriteInt(AH, te->dumpId); /* For sanity check */
- ctx->cs = AllocateCompressor(AH->compression, _CustomWriteFunc);
+ ctx->cs = AllocateCompressor(&AH->compression, _CustomWriteFunc);
}
/*
@@ -377,7 +377,7 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid)
WriteInt(AH, oid);
- ctx->cs = AllocateCompressor(AH->compression, _CustomWriteFunc);
+ ctx->cs = AllocateCompressor(&AH->compression, _CustomWriteFunc);
}
/*
@@ -566,7 +566,7 @@ _PrintTocData(ArchiveHandle *AH, TocEntry *te)
static void
_PrintData(ArchiveHandle *AH)
{
- ReadDataFromArchive(AH, AH->compression, _CustomReadFunc);
+ ReadDataFromArchive(AH, _CustomReadFunc);
}
static void
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 6e55cb425e..dfa36d5dc1 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -202,7 +202,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH)
setFilePath(AH, fname, "toc.dat");
- tocFH = cfopen_read(fname, PG_BINARY_R, AH->compression);
+ tocFH = cfopen_read(fname, PG_BINARY_R, &AH->compression);
if (tocFH == NULL)
fatal("could not open input file \"%s\": %m", fname);
@@ -327,7 +327,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te)
setFilePath(AH, fname, tctx->filename);
- ctx->dataFH = cfopen_write(fname, PG_BINARY_W, AH->compression);
+ ctx->dataFH = cfopen_write(fname, PG_BINARY_W, &AH->compression);
if (ctx->dataFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
}
@@ -388,12 +388,12 @@ _PrintFileData(ArchiveHandle *AH, char *filename)
if (!filename)
return;
- cfp = cfopen_read(filename, PG_BINARY_R, AH->compression);
+ cfp = cfopen_read(filename, PG_BINARY_R, &AH->compression);
if (!cfp)
fatal("could not open input file \"%s\": %m", filename);
- buf = pg_malloc(ZLIB_OUT_SIZE);
+ buf = pg_malloc(ZLIB_OUT_SIZE); // XXX
buflen = ZLIB_OUT_SIZE;
while ((cnt = cfread(buf, buflen, cfp)))
@@ -435,12 +435,13 @@ _LoadBlobs(ArchiveHandle *AH)
lclContext *ctx = (lclContext *) AH->formatData;
char fname[MAXPGPATH];
char line[MAXPGPATH];
+ Compress nocompression = {0};
StartRestoreBlobs(AH);
setFilePath(AH, fname, "blobs.toc");
- ctx->blobsTocFH = cfopen_read(fname, PG_BINARY_R, AH->compression);
+ ctx->blobsTocFH = cfopen_read(fname, PG_BINARY_R, &nocompression);
if (ctx->blobsTocFH == NULL)
fatal("could not open large object TOC file \"%s\" for input: %m",
@@ -573,6 +574,7 @@ _CloseArchive(ArchiveHandle *AH)
{
cfp *tocFH;
char fname[MAXPGPATH];
+ Compress nocompression = {0};
setFilePath(AH, fname, "toc.dat");
@@ -580,7 +582,7 @@ _CloseArchive(ArchiveHandle *AH)
ctx->pstate = ParallelBackupStart(AH);
/* The TOC is always created uncompressed */
- tocFH = cfopen_write(fname, PG_BINARY_W, 0);
+ tocFH = cfopen_write(fname, PG_BINARY_W, &nocompression);
if (tocFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
ctx->dataFH = tocFH;
@@ -639,11 +641,12 @@ _StartBlobs(ArchiveHandle *AH, TocEntry *te)
{
lclContext *ctx = (lclContext *) AH->formatData;
char fname[MAXPGPATH];
+ Compress nocompression = {0};
setFilePath(AH, fname, "blobs.toc");
/* The blob TOC file is never compressed */
- ctx->blobsTocFH = cfopen_write(fname, "ab", 0);
+ ctx->blobsTocFH = cfopen_write(fname, "ab", &nocompression);
if (ctx->blobsTocFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
}
@@ -661,7 +664,7 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid)
snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid);
- ctx->dataFH = cfopen_write(fname, PG_BINARY_W, AH->compression);
+ ctx->dataFH = cfopen_write(fname, PG_BINARY_W, &AH->compression);
if (ctx->dataFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index 54e708875c..5c9d17bae7 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -39,6 +39,7 @@
#include "pg_backup_archiver.h"
#include "pg_backup_tar.h"
#include "pg_backup_utils.h"
+#include "compress_io.h"
#include "pgtar.h"
static void _ArchiveEntry(ArchiveHandle *AH, TocEntry *te);
@@ -196,10 +197,10 @@ InitArchiveFmt_Tar(ArchiveHandle *AH)
/*
* We don't support compression because reading the files back is not
- * possible since gzdopen uses buffered IO which totally screws file
+ * possible since gzdopen uses buffered IO which totally screws file XXX
* positioning.
*/
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
fatal("compression is not supported by tar archive format");
}
else
@@ -254,14 +255,8 @@ _ArchiveEntry(ArchiveHandle *AH, TocEntry *te)
ctx = (lclTocEntry *) pg_malloc0(sizeof(lclTocEntry));
if (te->dataDumper != NULL)
{
-#ifdef HAVE_LIBZ
- if (AH->compression == 0)
- sprintf(fn, "%d.dat", te->dumpId);
- else
- sprintf(fn, "%d.dat.gz", te->dumpId);
-#else
- sprintf(fn, "%d.dat", te->dumpId);
-#endif
+ const char *suffix = compress_suffix(&AH->compression);
+ sprintf(fn, "%d.dat%s", te->dumpId, suffix);
ctx->filename = pg_strdup(fn);
}
else
@@ -352,7 +347,7 @@ tarOpen(ArchiveHandle *AH, const char *filename, char mode)
#ifdef HAVE_LIBZ
- if (AH->compression == 0)
+ if (AH->compression.alg == COMPR_ALG_NONE)
tm->nFH = ctx->tarFH;
else
fatal("compression is not supported by tar archive format");
@@ -413,9 +408,9 @@ tarOpen(ArchiveHandle *AH, const char *filename, char mode)
#ifdef HAVE_LIBZ
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
{
- sprintf(fmode, "wb%d", AH->compression);
+ sprintf(fmode, "wb%d", AH->compression.level);
tm->zFH = gzdopen(dup(fileno(tm->tmpFH)), fmode);
if (tm->zFH == NULL)
fatal("could not open temporary file");
@@ -443,7 +438,7 @@ tarClose(ArchiveHandle *AH, TAR_MEMBER *th)
/*
* Close the GZ file since we dup'd. This will flush the buffers.
*/
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
if (GZCLOSE(th->zFH) != 0)
fatal("could not close tar member");
@@ -868,7 +863,7 @@ _CloseArchive(ArchiveHandle *AH)
memcpy(ropt, AH->public.ropt, sizeof(RestoreOptions));
ropt->filename = NULL;
ropt->dropSchema = 1;
- ropt->compression = 0;
+ ropt->compression.alg = COMPR_ALG_NONE;
ropt->superuser = NULL;
ropt->suppressDumpWarnings = true;
@@ -952,16 +947,12 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid)
lclContext *ctx = (lclContext *) AH->formatData;
lclTocEntry *tctx = (lclTocEntry *) te->formatData;
char fname[255];
- char *sfx;
+ const char *sfx;
if (oid == 0)
fatal("invalid OID for large object (%u)", oid);
- if (AH->compression != 0)
- sfx = ".gz";
- else
- sfx = "";
-
+ sfx = compress_suffix(&AH->compression);
sprintf(fname, "blob_%u.dat%s", oid, sfx);
tarPrintf(ctx->blobToc, "%u %s\n", oid, fname);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index f21eb021c7..5f80d05716 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -59,6 +59,7 @@
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
+#include "compress_io.h"
#include "pg_backup_db.h"
#include "pg_backup_utils.h"
#include "pg_dump.h"
@@ -297,6 +298,111 @@ static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+/* Parse the string into compression options */
+static void
+parse_compression(const char *optarg, Compress *compress)
+{
+ if (optarg[0] == '0' && optarg[1] == '\0')
+ compress->alg = COMPR_ALG_NONE;
+ else if ((optarg[0] > '0' && optarg[0] <= '9') ||
+ optarg[0] == '-')
+ {
+ compress->alg = COMPR_ALG_LIBZ;
+ compress->level = atoi(optarg);
+ if (optarg[1] != '\0')
+ {
+ pg_log_error("compression level must be in range 0..9");
+ exit_nicely(1);
+ }
+ }
+ else
+ {
+ /* Parse a more flexible string like level=3 alg=zlib opts=long */
+ for (;;)
+ {
+ char *eq = strchr(optarg, '=');
+ int len;
+
+ if (eq == NULL)
+ {
+ pg_log_error("compression options must be key=value: %s", optarg);
+ exit_nicely(1);
+ }
+
+ len = eq - optarg;
+ if (strncmp(optarg, "alg", len) == 0)
+ {
+ if (strchr(eq, ' '))
+ len = strchr(eq, ' ') - eq - 1;
+ else
+ len = strlen(eq) - len;
+ if (strncmp(1+eq, "zlib", len) == 0 ||
+ strncmp(1+eq, "libz", len) == 0)
+ compress->alg = COMPR_ALG_LIBZ;
+ else if (strncmp(1+eq, "zstd", len) == 0)
+ compress->alg = COMPR_ALG_ZSTD;
+ else
+ {
+ pg_log_error("unknown compression algorithm: %s", 1+eq);
+ exit_nicely(1);
+ }
+ }
+ else if (strncmp(optarg, "level", len) == 0)
+ compress->level = atoi(1+eq);
+ else if (strncmp(optarg, "opt", len) == 0)
+ {
+ if (strchr(eq, ' '))
+ len = strchr(eq, ' ') - eq - 1;
+ else
+ len = strlen(eq) - len;
+ if (strncmp(1+eq, "zstdlong", len) == 0)
+ compress->zstdlong = true;
+ else
+ {
+ pg_log_error("unknown compression option: %s", 1+eq);
+ exit_nicely(1);
+ }
+ }
+ else
+ {
+ pg_log_error("unknown compression setting: %s", optarg);
+ exit_nicely(1);
+ }
+
+ optarg = strchr(eq, ' ');
+ if (!optarg++)
+ break;
+ }
+
+ /* zstd will check its own compression level later */
+ if (compress->alg != COMPR_ALG_ZSTD)
+ {
+ if (compress->level < 0 || compress->level > 9)
+ {
+ pg_log_error("compression level must be in range 0..9");
+ exit_nicely(1);
+ }
+ if (compress->zstdlong)
+ {
+ pg_log_error("compression option not supported with this algorithm");
+ exit_nicely(1);
+ }
+ }
+
+ /* XXX: 0 means "unset" */
+ if (compress->level == 0)
+ {
+ const int default_compress_level[] = {
+ 0, /* COMPR_ALG_NONE */
+ Z_DEFAULT_COMPRESSION, /* COMPR_ALG_ZLIB */
+ ZSTD_CLEVEL_DEFAULT, /* COMPR_ALG_ZSTD */
+ };
+
+ compress->level = default_compress_level[compress->alg];
+ }
+ }
+}
+
int
main(int argc, char **argv)
{
@@ -319,7 +425,7 @@ main(int argc, char **argv)
char *use_role = NULL;
long rowsPerInsert;
int numWorkers = 1;
- int compressLevel = -1;
+ Compress compress = { .alg = COMPR_ALG_DEFAULT };
int plainText = 0;
ArchiveFormat archiveFormat = archUnknown;
ArchiveMode archiveMode;
@@ -532,13 +638,7 @@ main(int argc, char **argv)
break;
case 'Z': /* Compression Level */
- compressLevel = atoi(optarg);
- if ((compressLevel < 0 || compressLevel > 9) &&
- compressLevel != -2)
- {
- pg_log_error("compression level must be in range 0..9");
- exit_nicely(1);
- }
+ parse_compression(optarg, &compress);
break;
case 0:
@@ -680,20 +780,40 @@ main(int argc, char **argv)
plainText = 1;
/* Custom and directory formats are compressed by default, others not */
- if (compressLevel == -1)
+ if (compress.alg == COMPR_ALG_DEFAULT)
{
-#ifdef HAVE_LIBZ
if (archiveFormat == archCustom || archiveFormat == archDirectory)
- compressLevel = Z_DEFAULT_COMPRESSION;
- else
+ {
+#ifdef HAVE_LIBZ
+ compress.alg = COMPR_ALG_LIBZ;
+ compress.level = Z_DEFAULT_COMPRESSION;
+#endif
+#ifdef HAVE_LIBZSTD
+ compress.alg = COMPR_ALG_ZSTD; // Set default for testing purposes
+ compress.level = ZSTD_CLEVEL_DEFAULT;
#endif
- compressLevel = 0;
+ }
+ else
+ {
+ compress.alg = COMPR_ALG_NONE;
+ compress.level = 0;
+ }
}
#ifndef HAVE_LIBZ
- if (compressLevel != 0)
+ if (compress.alg == COMPR_ALG_LIBZ)
+ {
pg_log_warning("requested compression not available in this installation -- archive will be uncompressed");
- compressLevel = 0;
+ compress.alg = 0;
+ }
+#endif
+
+#ifndef HAVE_LIBZ
+ if (compress.alg == COMPR_ALG_ZSTD)
+ {
+ pg_log_warning("requested compression not available in this installation -- archive will be uncompressed");
+ compress.alg = 0;
+ }
#endif
/*
@@ -724,7 +844,7 @@ main(int argc, char **argv)
fatal("option --index-collation-versions-unknown only works in binary upgrade mode");
/* Open the output file */
- fout = CreateArchive(filename, archiveFormat, compressLevel, dosync,
+ fout = CreateArchive(filename, archiveFormat, &compress, dosync,
archiveMode, setupDumpWorker);
/* Make dump options accessible right away */
@@ -958,10 +1078,7 @@ main(int argc, char **argv)
ropt->sequence_data = dopt.sequence_data;
ropt->binary_upgrade = dopt.binary_upgrade;
- if (compressLevel == -1)
- ropt->compression = 0;
- else
- ropt->compression = compressLevel;
+ ropt->compression = compress;
ropt->suppressDumpWarnings = true; /* We've already shown them */
--
2.17.0
[text/x-diff] 0005-union-with-a-CompressionAlgorithm-alg.patch (11.5K, ../../20201222023235.GJ30237@telsasoft.com/6-0005-union-with-a-CompressionAlgorithm-alg.patch)
download | inline diff:
From bc6bc514d784da4cab7262c8386b0b8ed29ed02f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 21 Dec 2020 00:11:43 -0600
Subject: [PATCH 5/7] union{} with a CompressionAlgorithm alg
---
src/bin/pg_dump/compress_io.c | 218 ++++++++++++++++------------------
1 file changed, 105 insertions(+), 113 deletions(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 421b8e04cb..06005c3fba 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -619,23 +619,27 @@ WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
*/
struct cfp
{
- FILE *uncompressedfp;
+ CompressionAlgorithm alg;
+
+ union {
+ FILE *fp;
+
#ifdef HAVE_LIBZ
- gzFile compressedfp;
+ gzFile gzfp;
#endif
-#ifdef HAVE_LIBZSTD // XXX: this should be a union with a CompressionAlgorithm alg?
- /* This is a normal file to which we read/write compressed data */
- struct {
- FILE *fp;
- // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
- ZSTD_CStream *cstream;
- ZSTD_DStream *dstream;
- ZSTD_outBuffer output;
- ZSTD_inBuffer input;
- } zstd;
+#ifdef HAVE_LIBZSTD
+ struct {
+ /* This is a normal file to which we read/write compressed data */
+ FILE *fp;
+ // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ } zstd;
#endif
-
+ } u;
};
#ifdef HAVE_LIBZ
@@ -726,13 +730,7 @@ cfopen(const char *path, const char *mode, Compress *compression)
{
cfp *fp = pg_malloc(sizeof(cfp));
- fp->uncompressedfp = NULL;
-#ifdef HAVE_LIBZ
- fp->compressedfp = NULL;
-#endif
-#ifdef HAVE_LIBZSTD
- fp->zstd.fp = NULL;
-#endif
+ fp->alg = compression->alg;
switch (compression->alg)
{
@@ -745,15 +743,15 @@ cfopen(const char *path, const char *mode, Compress *compression)
snprintf(mode_compression, sizeof(mode_compression), "%s%d",
mode, compression->level);
- fp->compressedfp = gzopen(path, mode_compression);
+ fp->u.gzfp = gzopen(path, mode_compression);
}
else
{
/* don't specify a level, just use the zlib default */
- fp->compressedfp = gzopen(path, mode);
+ fp->u.gzfp = gzopen(path, mode);
}
- if (fp->compressedfp == NULL)
+ if (fp->u.gzfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
@@ -765,26 +763,26 @@ cfopen(const char *path, const char *mode, Compress *compression)
#ifdef HAVE_LIBZSTD
case COMPR_ALG_ZSTD:
- fp->zstd.fp = fopen(path, mode);
- if (fp->zstd.fp == NULL)
+ fp->u.zstd.fp = fopen(path, mode);
+ if (fp->u.zstd.fp == NULL)
{
free_keep_errno(fp);
fp = NULL;
}
else if (strchr(mode, 'w'))
{
- fp->zstd.dstream = NULL;
- fp->zstd.output.size = ZSTD_CStreamOutSize();
- fp->zstd.output.dst = pg_malloc0(fp->zstd.output.size);
- fp->zstd.cstream = ZstdCStreamParams(compression);
+ fp->u.zstd.dstream = NULL;
+ fp->u.zstd.output.size = ZSTD_CStreamOutSize();
+ fp->u.zstd.output.dst = pg_malloc0(fp->u.zstd.output.size);
+ fp->u.zstd.cstream = ZstdCStreamParams(compression);
}
else if (strchr(mode, 'r'))
{
- fp->zstd.cstream = NULL;
- fp->zstd.input.size = ZSTD_DStreamInSize();
- fp->zstd.input.src = pg_malloc0(fp->zstd.input.size);
- fp->zstd.dstream = ZSTD_createDStream();
- if (fp->zstd.dstream == NULL)
+ fp->u.zstd.cstream = NULL;
+ fp->u.zstd.input.size = ZSTD_DStreamInSize();
+ fp->u.zstd.input.src = pg_malloc0(fp->u.zstd.input.size);
+ fp->u.zstd.dstream = ZSTD_createDStream();
+ if (fp->u.zstd.dstream == NULL)
fatal("could not initialize compression library");
} // XXX else: bad mode
return fp;
@@ -792,8 +790,8 @@ cfopen(const char *path, const char *mode, Compress *compression)
#endif
case COMPR_ALG_NONE:
- fp->uncompressedfp = fopen(path, mode);
- if (fp->uncompressedfp == NULL)
+ fp->u.fp = fopen(path, mode);
+ if (fp->u.fp == NULL)
{
free_keep_errno(fp);
fp = NULL;
@@ -817,26 +815,26 @@ cfread(void *ptr, int size, cfp *fp)
return 0;
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
- ret = gzread(fp->compressedfp, ptr, size);
- if (ret != size && !gzeof(fp->compressedfp))
+ ret = gzread(fp->u.gzfp, ptr, size);
+ if (ret != size && !gzeof(fp->u.gzfp))
{
int errnum;
- const char *errmsg = gzerror(fp->compressedfp, &errnum);
+ const char *errmsg = gzerror(fp->u.gzfp, &errnum);
fatal("could not read from input file: %s",
errnum == Z_ERRNO ? strerror(errno) : errmsg);
}
+ return ret;
}
- else
#endif
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
- ZSTD_outBuffer *output = &fp->zstd.output;
- ZSTD_inBuffer *input = &fp->zstd.input;
+ ZSTD_outBuffer *output = &fp->u.zstd.output;
+ ZSTD_inBuffer *input = &fp->u.zstd.input;
size_t input_size = ZSTD_DStreamInSize();
size_t res, cnt;
@@ -845,7 +843,7 @@ cfread(void *ptr, int size, cfp *fp)
output->pos = 0;
/* read compressed data */
- while ((cnt = fread(unconstify(void *, input->src), 1, input_size, fp->zstd.fp)))
+ while ((cnt = fread(unconstify(void *, input->src), 1, input_size, fp->u.zstd.fp)))
{
input->size = cnt;
input->pos = 0;
@@ -853,7 +851,7 @@ cfread(void *ptr, int size, cfp *fp)
for ( ; input->pos < input->size; )
{
/* decompress */
- res = ZSTD_decompressStream(fp->zstd.dstream, output, input);
+ res = ZSTD_decompressStream(fp->u.zstd.dstream, output, input);
if (res == 0 || output->pos == output->size)
break;
if (ZSTD_isError(res))
@@ -864,16 +862,13 @@ cfread(void *ptr, int size, cfp *fp)
break; /* We read all the data that fits */
}
- ret = output->pos;
+ return output->pos;
}
- else
#endif
- {
- ret = fread(ptr, 1, size, fp->uncompressedfp);
- if (ret != size && !feof(fp->uncompressedfp))
- READ_ERROR_EXIT(fp->uncompressedfp);
- }
+ ret = fread(ptr, 1, size, fp->u.fp);
+ if (ret != size && !feof(fp->u.fp))
+ READ_ERROR_EXIT(fp->u.fp);
return ret;
}
@@ -881,16 +876,16 @@ int
cfwrite(const void *ptr, int size, cfp *fp)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
- return gzwrite(fp->compressedfp, ptr, size);
- else
+ if (fp->alg == COMPR_ALG_LIBZ)
+ return gzwrite(fp->u.gzfp, ptr, size);
#endif
+
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
size_t res, cnt;
- ZSTD_outBuffer *output = &fp->zstd.output;
- ZSTD_inBuffer *input = &fp->zstd.input;
+ ZSTD_outBuffer *output = &fp->u.zstd.output;
+ ZSTD_inBuffer *input = &fp->u.zstd.input;
input->src = ptr;
input->size = size;
@@ -900,21 +895,20 @@ cfwrite(const void *ptr, int size, cfp *fp)
while (input->pos != input->size)
{
output->pos = 0;
- res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_continue);
+ res = ZSTD_compressStream2(fp->u.zstd.cstream, output, input, ZSTD_e_continue);
if (ZSTD_isError(res))
fatal("could not compress data: %s", ZSTD_getErrorName(res));
- cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ cnt = fwrite(output->dst, 1, output->pos, fp->u.zstd.fp);
if (cnt != output->pos)
fatal("could not write data: %s", strerror(errno));
}
return size;
}
- else
#endif
- return fwrite(ptr, 1, size, fp->uncompressedfp);
+ return fwrite(ptr, 1, size, fp->u.fp);
}
int
@@ -923,39 +917,38 @@ cfgetc(cfp *fp)
int ret;
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
- ret = gzgetc(fp->compressedfp);
+ ret = gzgetc(fp->u.gzfp);
if (ret == EOF)
{
- if (!gzeof(fp->compressedfp))
+ if (!gzeof(fp->u.gzfp))
fatal("could not read from input file: %s", strerror(errno));
else
fatal("could not read from input file: end of file");
}
+ return ret;
}
- else
#endif
+
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
if (cfread(&ret, 1, fp) != 1)
{
- if (feof(fp->zstd.fp))
+ if (feof(fp->u.zstd.fp))
fatal("could not read from input file: end of file");
else
fatal("could not read from input file: %s", strerror(errno));
}
fprintf(stderr, "cfgetc %d\n", ret);
+ return ret;
}
#endif
- {
- ret = fgetc(fp->uncompressedfp);
- if (ret == EOF)
- READ_ERROR_EXIT(fp->uncompressedfp);
- }
-
+ ret = fgetc(fp->u.fp);
+ if (ret == EOF)
+ READ_ERROR_EXIT(fp->u.fp);
return ret;
}
@@ -963,12 +956,12 @@ char *
cfgets(cfp *fp, char *buf, int len)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
- return gzgets(fp->compressedfp, buf, len);
- else
+ if (fp->alg == COMPR_ALG_LIBZ)
+ return gzgets(fp->u.gzfp, buf, len);
#endif
+
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
int res;
res = cfread(buf, len, fp);
@@ -977,9 +970,9 @@ cfgets(cfp *fp, char *buf, int len)
*strchr(buf, '\n') = '\0';
return res > 0 ? buf : 0;
}
- else
#endif
- return fgets(buf, len, fp->uncompressedfp);
+
+ return fgets(buf, len, fp->u.fp);
}
int
@@ -993,56 +986,55 @@ cfclose(cfp *fp)
return EOF;
}
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
- result = gzclose(fp->compressedfp);
- fp->compressedfp = NULL;
+ result = gzclose(fp->u.gzfp);
+ fp->u.gzfp = NULL;
+ return result;
}
- else
#endif
+
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
- ZSTD_outBuffer *output = &fp->zstd.output;
- ZSTD_inBuffer *input = &fp->zstd.input;
+ ZSTD_outBuffer *output = &fp->u.zstd.output;
+ ZSTD_inBuffer *input = &fp->u.zstd.input;
size_t res, cnt;
- if (fp->zstd.cstream)
+ if (fp->u.zstd.cstream)
{
for (;;)
{
output->pos = 0;
- res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_end);
+ res = ZSTD_compressStream2(fp->u.zstd.cstream, output, input, ZSTD_e_end);
if (ZSTD_isError(res))
fatal("could not compress data: %s", ZSTD_getErrorName(res));
- cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ cnt = fwrite(output->dst, 1, output->pos, fp->u.zstd.fp);
if (cnt != output->pos)
fatal("could not write data: %s", strerror(errno));
if (res == 0)
break;
}
- ZSTD_freeCStream(fp->zstd.cstream);
- pg_free(fp->zstd.output.dst);
+ ZSTD_freeCStream(fp->u.zstd.cstream);
+ pg_free(fp->u.zstd.output.dst);
}
- if (fp->zstd.dstream)
+ if (fp->u.zstd.dstream)
{
- ZSTD_freeDStream(fp->zstd.dstream);
- pg_free(unconstify(void *, fp->zstd.input.src));
+ ZSTD_freeDStream(fp->u.zstd.dstream);
+ pg_free(unconstify(void *, fp->u.zstd.input.src));
}
- result = fclose(fp->zstd.fp);
- fp->zstd.fp = NULL;
+ result = fclose(fp->u.zstd.fp);
+ fp->u.zstd.fp = NULL;
+ return result;
}
- else
#endif
- {
- result = fclose(fp->uncompressedfp);
- fp->uncompressedfp = NULL;
- }
- free_keep_errno(fp);
+ result = fclose(fp->u.fp);
+ fp->u.fp = NULL;
+ free_keep_errno(fp);
return result;
}
@@ -1050,26 +1042,26 @@ int
cfeof(cfp *fp)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
- return gzeof(fp->compressedfp);
- else
+ if (fp->alg == COMPR_ALG_LIBZ)
+ return gzeof(fp->u.gzfp);
#endif
+
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
- return feof(fp->zstd.fp);
- else
+ if (fp->alg == COMPR_ALG_ZSTD)
+ return feof(fp->u.zstd.fp);
#endif
- return feof(fp->uncompressedfp);
+
+ return feof(fp->u.fp);
}
const char *
get_cfp_error(cfp *fp)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
int errnum;
- const char *errmsg = gzerror(fp->compressedfp, &errnum);
+ const char *errmsg = gzerror(fp->u.gzfp, &errnum);
if (errnum != Z_ERRNO)
return errmsg;
--
2.17.0
[text/x-diff] 0006-Move-zlib-into-the-union.patch (4.4K, ../../20201222023235.GJ30237@telsasoft.com/7-0006-Move-zlib-into-the-union.patch)
download | inline diff:
From 15cd65b4d650009cf18f48357e9dfe9173d32446 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 11 Dec 2020 22:22:31 -0600
Subject: [PATCH 6/7] Move zlib into the union{}
---
src/bin/pg_dump/compress_io.c | 54 ++++++++++++++++++-----------------
1 file changed, 28 insertions(+), 26 deletions(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 06005c3fba..a94efbea4e 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -67,23 +67,25 @@ struct CompressorState
CompressionAlgorithm comprAlg;
WriteFunc writeF;
+ union {
#ifdef HAVE_LIBZ
- z_streamp zp;
- char *zlibOut;
- size_t zlibOutSize;
+ struct {
+ z_streamp zp;
+ char *zlibOut;
+ size_t zlibOutSize;
+ } zlib;
#endif
#ifdef HAVE_LIBZSTD
- union {
+ /* This is used for compression but not decompression */
struct {
- ZSTD_outBuffer output;
- ZSTD_inBuffer input;
// XXX: use one separate ZSTD_CStream per thread: disable on windows ?
- ZSTD_CStream *cstream;
+ ZSTD_CStream *cstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
} zstd;
- } u;
#endif
-
+ } u;
};
/* Routines that support zlib compressed data I/O */
@@ -420,7 +422,7 @@ InitCompressorZlib(CompressorState *cs, Compress *compress)
{
z_streamp zp;
- zp = cs->zp = (z_streamp) pg_malloc(sizeof(z_stream));
+ zp = cs->u.zlib.zp = (z_streamp) pg_malloc(sizeof(z_stream));
zp->zalloc = Z_NULL;
zp->zfree = Z_NULL;
zp->opaque = Z_NULL;
@@ -430,22 +432,22 @@ InitCompressorZlib(CompressorState *cs, Compress *compress)
* actually allocate one extra byte because some routines want to append a
* trailing zero byte to the zlib output.
*/
- cs->zlibOut = (char *) pg_malloc(ZLIB_OUT_SIZE + 1);
- cs->zlibOutSize = ZLIB_OUT_SIZE;
+ cs->u.zlib.zlibOut = (char *) pg_malloc(ZLIB_OUT_SIZE + 1);
+ cs->u.zlib.zlibOutSize = ZLIB_OUT_SIZE;
if (deflateInit(zp, compress->level) != Z_OK)
fatal("could not initialize compression library: %s",
zp->msg);
/* Just be paranoid - maybe End is called after Start, with no Write */
- zp->next_out = (void *) cs->zlibOut;
- zp->avail_out = cs->zlibOutSize;
+ zp->next_out = (void *) cs->u.zlib.zlibOut;
+ zp->avail_out = cs->u.zlib.zlibOutSize;
}
static void
EndCompressorZlib(ArchiveHandle *AH, CompressorState *cs)
{
- z_streamp zp = cs->zp;
+ z_streamp zp = cs->u.zlib.zp;
zp->next_in = NULL;
zp->avail_in = 0;
@@ -456,23 +458,23 @@ EndCompressorZlib(ArchiveHandle *AH, CompressorState *cs)
if (deflateEnd(zp) != Z_OK)
fatal("could not close compression stream: %s", zp->msg);
- free(cs->zlibOut);
- free(cs->zp);
+ free(cs->u.zlib.zlibOut);
+ free(cs->u.zlib.zp);
}
static void
DeflateCompressorZlib(ArchiveHandle *AH, CompressorState *cs, bool flush)
{
- z_streamp zp = cs->zp;
- char *out = cs->zlibOut;
+ z_streamp zp = cs->u.zlib.zp;
+ char *out = cs->u.zlib.zlibOut;
int res = Z_OK;
- while (cs->zp->avail_in != 0 || flush)
+ while (cs->u.zlib.zp->avail_in != 0 || flush)
{
res = deflate(zp, flush ? Z_FINISH : Z_NO_FLUSH);
if (res == Z_STREAM_ERROR)
fatal("could not compress data: %s", zp->msg);
- if ((flush && (zp->avail_out < cs->zlibOutSize))
+ if ((flush && (zp->avail_out < cs->u.zlib.zlibOutSize))
|| (zp->avail_out == 0)
|| (zp->avail_in != 0)
)
@@ -482,18 +484,18 @@ DeflateCompressorZlib(ArchiveHandle *AH, CompressorState *cs, bool flush)
* chunk is the EOF marker in the custom format. This should never
* happen but...
*/
- if (zp->avail_out < cs->zlibOutSize)
+ if (zp->avail_out < cs->u.zlib.zlibOutSize)
{
/*
* Any write function should do its own error checking but to
* make sure we do a check here as well...
*/
- size_t len = cs->zlibOutSize - zp->avail_out;
+ size_t len = cs->u.zlib.zlibOutSize - zp->avail_out;
cs->writeF(AH, out, len);
}
zp->next_out = (void *) out;
- zp->avail_out = cs->zlibOutSize;
+ zp->avail_out = cs->u.zlib.zlibOutSize;
}
if (res == Z_STREAM_END)
@@ -505,8 +507,8 @@ static void
WriteDataToArchiveZlib(ArchiveHandle *AH, CompressorState *cs,
const char *data, size_t dLen)
{
- cs->zp->next_in = (void *) unconstify(char *, data);
- cs->zp->avail_in = dLen;
+ cs->u.zlib.zp->next_in = (void *) unconstify(char *, data);
+ cs->u.zlib.zp->avail_in = dLen;
DeflateCompressorZlib(AH, cs, false);
}
--
2.17.0
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2020-12-21 19:49 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2020-12-21 20:02 ` Re: zstd compression for pg_dump Tom Lane <tgl@sss.pgh.pa.us>
2020-12-22 02:32 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2021-01-04 02:53 ` Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 06:04 ` Re: zstd compression for pg_dump Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-10 21:06 ` Re: zstd compression for pg_dump Andreas Karlsson <andreas@proxel.se>
2021-04-08 15:19 ` Re: zstd compression for pg_dump David Steele <david@pgmasters.net>
0 siblings, 3 replies; 46+ messages in thread
From: Justin Pryzby @ 2021-01-04 02:53 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers
On Mon, Dec 21, 2020 at 01:49:24PM -0600, Justin Pryzby wrote:
> a big disadvantage of piping through zstd is that it's not identified as a
> PGDMP file, and, /usr/bin/file on centos7 fails to even identify zstd by its
> magic number..
Other reasons are that pg_dump |zstd >output.zst loses the exit status of
pg_dump, and that it's not "transparent" (one needs to type
"zstd -dq |pg_restore").
On Mon, Dec 21, 2020 at 08:32:35PM -0600, Justin Pryzby wrote:
> On Mon, Dec 21, 2020 at 03:02:40PM -0500, Tom Lane wrote:
> > Justin Pryzby <pryzby@telsasoft.com> writes:
> > > I found that our largest tables are 40% smaller and 20% faster to pipe
> > > pg_dump -Fc -Z0 |zstd relative to native zlib
> >
> > The patch might be a tad smaller if you hadn't included a core file in it.
>
> About 89% smaller.
>
> This also fixes the extension (.zst)
> And fixes zlib default compression.
> And a bunch of cleanup.
I rebased so the "typedef struct compression" patch is first and zstd on top of
that (say, in case someone wants to bikeshed about which compression algorithm
to support). And made a central struct with all the compression-specific info
to further isolate the compress-specific changes.
And handle compression of "plain" archive format.
And fix compilation for MSVC and make --without-zstd the default.
And fix cfgets() (which I think is actually unused code for the code paths for
compressed FP).
And add fix for pre-existing problem: ftello() on unseekable input.
I also started a patch to allow compression of "tar" format, but I didn't
include that here yet.
Note, there's currently several "compression" patches in CF app. This patch
seems to be independent of the others, but probably shouldn't be totally
uncoordinated (like adding lz4 in one and ztsd in another might be poor
execution).
https://commitfest.postgresql.org/31/2897/
- Faster pglz compression
https://commitfest.postgresql.org/31/2813/
- custom compression methods for toast
https://commitfest.postgresql.org/31/2773/
- libpq compression
--
Justin
Attachments:
[text/x-diff] 0001-fix-preeexisting.patch (977B, ../../20210104025321.GA9712@telsasoft.com/2-0001-fix-preeexisting.patch)
download | inline diff:
From d2fc2673e19a95629edfe9a0f4ead75e1f1f2754 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 23 Dec 2020 23:56:54 -0600
Subject: [PATCH 01/20] fix!preeexisting
---
src/bin/pg_dump/compress_io.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 1417401086..6a428978d4 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -28,7 +28,7 @@
* The interface for reading an archive consists of just one function:
* ReadDataFromArchive. ReadDataFromArchive reads the whole compressed input
* stream, by repeatedly calling the given ReadFunc. ReadFunc returns the
- * compressed data chunk at a time, and ReadDataFromArchive decompresses it
+ * compressed data one chunk at a time, and ReadDataFromArchive decompresses it
* and passes the decompressed data to ahwrite(), until ReadFunc returns 0
* to signal EOF.
*
--
2.17.0
[text/x-diff] 0002-Fix-broken-error-message-on-unseekable-input.patch (1.4K, ../../20210104025321.GA9712@telsasoft.com/3-0002-Fix-broken-error-message-on-unseekable-input.patch)
download | inline diff:
From e8bb61dd633aefb2cc7a14887a15cc60e05cd9c5 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 22 Dec 2020 15:40:08 -0600
Subject: [PATCH 02/20] Fix broken error message on unseekable input..
pg_dump -Fd -f dir.dir
cat dir.dir/toc.dat | pg_restore -l # I realize this isn't how it's intended to be used
pg_restore: error: corrupt tar header found in PGDMP (expected 0, computed 18577) file position 18446744073709551615
See also 929c69aa19
---
src/bin/pg_dump/pg_backup_tar.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index 54e708875c..61c9c87a9f 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -1280,12 +1280,14 @@ _tarGetHeader(ArchiveHandle *AH, TAR_MEMBER *th)
if (chk != sum)
{
- char posbuf[32];
+ off_t off = ftello(ctx->tarFH);
- snprintf(posbuf, sizeof(posbuf), UINT64_FORMAT,
- (uint64) ftello(ctx->tarFH));
- fatal("corrupt tar header found in %s (expected %d, computed %d) file position %s",
- tag, sum, chk, posbuf);
+ if (off == -1)
+ fatal("corrupt tar header found in %s (expected %d, computed %d)",
+ tag, sum, chk);
+ else
+ fatal("corrupt tar header found in %s (expected %d, computed %d) file position " UINT64_FORMAT,
+ tag, sum, chk, off);
}
th->targetFile = pg_strdup(tag);
--
2.17.0
[text/x-diff] 0003-Support-multiple-compression-algs-levels-opts.patch (34.1K, ../../20210104025321.GA9712@telsasoft.com/4-0003-Support-multiple-compression-algs-levels-opts.patch)
download | inline diff:
From bbf91d13a25bcb2f0718f40984362f81741ca66a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 22 Dec 2020 00:23:43 -0600
Subject: [PATCH 03/20] Support multiple compression algs/levels/opts..
The existing implementation abtracts compressed and noncompressed I/O.
This preliminary commit is intended to also allow for multiple compression
algorithms.
---
src/bin/pg_dump/compress_io.c | 220 +++++++++++---------------
src/bin/pg_dump/compress_io.h | 19 +--
src/bin/pg_dump/pg_backup.h | 23 ++-
src/bin/pg_dump/pg_backup_archiver.c | 45 +++---
src/bin/pg_dump/pg_backup_archiver.h | 12 +-
src/bin/pg_dump/pg_backup_custom.c | 6 +-
src/bin/pg_dump/pg_backup_directory.c | 17 +-
src/bin/pg_dump/pg_backup_tar.c | 33 ++--
src/bin/pg_dump/pg_dump.c | 115 +++++++++++---
9 files changed, 272 insertions(+), 218 deletions(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 6a428978d4..db16fd33f2 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -74,12 +74,9 @@ struct CompressorState
#endif
};
-static void ParseCompressionOption(int compression, CompressionAlgorithm *alg,
- int *level);
-
/* Routines that support zlib compressed data I/O */
#ifdef HAVE_LIBZ
-static void InitCompressorZlib(CompressorState *cs, int level);
+static void InitCompressorZlib(CompressorState *cs, Compress *compress);
static void DeflateCompressorZlib(ArchiveHandle *AH, CompressorState *cs,
bool flush);
static void ReadDataFromArchiveZlib(ArchiveHandle *AH, ReadFunc readF);
@@ -93,58 +90,36 @@ static void ReadDataFromArchiveNone(ArchiveHandle *AH, ReadFunc readF);
static void WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
const char *data, size_t dLen);
-/*
- * Interprets a numeric 'compression' value. The algorithm implied by the
- * value (zlib or none at the moment), is returned in *alg, and the
- * zlib compression level in *level.
- */
-static void
-ParseCompressionOption(int compression, CompressionAlgorithm *alg, int *level)
-{
- if (compression == Z_DEFAULT_COMPRESSION ||
- (compression > 0 && compression <= 9))
- *alg = COMPR_ALG_LIBZ;
- else if (compression == 0)
- *alg = COMPR_ALG_NONE;
- else
- {
- fatal("invalid compression code: %d", compression);
- *alg = COMPR_ALG_NONE; /* keep compiler quiet */
- }
-
- /* The level is just the passed-in value. */
- if (level)
- *level = compression;
-}
-
/* Public interface routines */
/* Allocate a new compressor */
CompressorState *
-AllocateCompressor(int compression, WriteFunc writeF)
+AllocateCompressor(Compress *compression, WriteFunc writeF)
{
CompressorState *cs;
- CompressionAlgorithm alg;
- int level;
-
- ParseCompressionOption(compression, &alg, &level);
-
-#ifndef HAVE_LIBZ
- if (alg == COMPR_ALG_LIBZ)
- fatal("not built with zlib support");
-#endif
cs = (CompressorState *) pg_malloc0(sizeof(CompressorState));
cs->writeF = writeF;
- cs->comprAlg = alg;
+ cs->comprAlg = compression->alg;
/*
* Perform compression algorithm specific initialization.
*/
+ Assert (compression->alg != COMPR_ALG_DEFAULT);
+ switch (compression->alg)
+ {
#ifdef HAVE_LIBZ
- if (alg == COMPR_ALG_LIBZ)
- InitCompressorZlib(cs, level);
+ case COMPR_ALG_LIBZ:
+ InitCompressorZlib(cs, compression);
+ break;
#endif
+ case COMPR_ALG_NONE:
+ /* Do nothing */
+ break;
+ default:
+ /* Should not happen */
+ fatal("requested compression not available in this installation");
+ }
return cs;
}
@@ -154,21 +129,21 @@ AllocateCompressor(int compression, WriteFunc writeF)
* out with ahwrite().
*/
void
-ReadDataFromArchive(ArchiveHandle *AH, int compression, ReadFunc readF)
+ReadDataFromArchive(ArchiveHandle *AH, ReadFunc readF)
{
- CompressionAlgorithm alg;
-
- ParseCompressionOption(compression, &alg, NULL);
-
- if (alg == COMPR_ALG_NONE)
- ReadDataFromArchiveNone(AH, readF);
- if (alg == COMPR_ALG_LIBZ)
+ switch (AH->compression.alg)
{
+ case COMPR_ALG_NONE:
+ ReadDataFromArchiveNone(AH, readF);
+ break;
#ifdef HAVE_LIBZ
+ case COMPR_ALG_LIBZ:
ReadDataFromArchiveZlib(AH, readF);
-#else
- fatal("not built with zlib support");
+ break;
#endif
+ default:
+ /* Should not happen */
+ fatal("requested compression not available in this installation");
}
}
@@ -181,16 +156,18 @@ WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs,
{
switch (cs->comprAlg)
{
- case COMPR_ALG_LIBZ:
#ifdef HAVE_LIBZ
+ case COMPR_ALG_LIBZ:
WriteDataToArchiveZlib(AH, cs, data, dLen);
-#else
- fatal("not built with zlib support");
-#endif
break;
+#endif
case COMPR_ALG_NONE:
WriteDataToArchiveNone(AH, cs, data, dLen);
break;
+
+ default:
+ /* Should not happen */
+ fatal("requested compression not available in this installation");
}
}
@@ -215,7 +192,7 @@ EndCompressor(ArchiveHandle *AH, CompressorState *cs)
*/
static void
-InitCompressorZlib(CompressorState *cs, int level)
+InitCompressorZlib(CompressorState *cs, Compress *compress)
{
z_streamp zp;
@@ -232,7 +209,7 @@ InitCompressorZlib(CompressorState *cs, int level)
cs->zlibOut = (char *) pg_malloc(ZLIB_OUT_SIZE + 1);
cs->zlibOutSize = ZLIB_OUT_SIZE;
- if (deflateInit(zp, level) != Z_OK)
+ if (deflateInit(zp, compress->level) != Z_OK)
fatal("could not initialize compression library: %s",
zp->msg);
@@ -424,9 +401,7 @@ struct cfp
#endif
};
-#ifdef HAVE_LIBZ
static int hasSuffix(const char *filename, const char *suffix);
-#endif
/* free() without changing errno; useful in several places below */
static void
@@ -442,34 +417,31 @@ free_keep_errno(void *p)
* Open a file for reading. 'path' is the file to open, and 'mode' should
* be either "r" or "rb".
*
- * If the file at 'path' does not exist, we append the ".gz" suffix (if 'path'
- * doesn't already have it) and try again. So if you pass "foo" as 'path',
+ * If the file at 'path' does not exist, we search with compressed suffix (if 'path'
+ * doesn't already have one) and try again. So if you pass "foo" as 'path',
* this will open either "foo" or "foo.gz".
*
* On failure, return NULL with an error code in errno.
*/
cfp *
-cfopen_read(const char *path, const char *mode)
+cfopen_read(const char *path, const char *mode, Compress *compression)
{
cfp *fp;
-#ifdef HAVE_LIBZ
if (hasSuffix(path, ".gz"))
- fp = cfopen(path, mode, 1);
+ fp = cfopen(path, mode, compression);
else
-#endif
{
- fp = cfopen(path, mode, 0);
-#ifdef HAVE_LIBZ
+ fp = cfopen(path, mode, compression);
if (fp == NULL)
{
char *fname;
+ const char *suffix = compress_suffix(compression);
- fname = psprintf("%s.gz", path);
- fp = cfopen(fname, mode, 1);
+ fname = psprintf("%s%s", path, suffix);
+ fp = cfopen(fname, mode, compression);
free_keep_errno(fname);
}
-#endif
}
return fp;
}
@@ -479,31 +451,26 @@ cfopen_read(const char *path, const char *mode)
* be a filemode as accepted by fopen() and gzopen() that indicates writing
* ("w", "wb", "a", or "ab").
*
- * If 'compression' is non-zero, a gzip compressed stream is opened, and
- * 'compression' indicates the compression level used. The ".gz" suffix
- * is automatically added to 'path' in that case.
+ * Use compression if specified.
+ * The appropriate suffix is automatically added to 'path' in that case.
*
* On failure, return NULL with an error code in errno.
*/
cfp *
-cfopen_write(const char *path, const char *mode, int compression)
+cfopen_write(const char *path, const char *mode, Compress *compression)
{
cfp *fp;
- if (compression == 0)
- fp = cfopen(path, mode, 0);
+ if (compression->alg == COMPR_ALG_NONE)
+ fp = cfopen(path, mode, compression);
else
{
-#ifdef HAVE_LIBZ
char *fname;
+ const char *suffix = compress_suffix(compression);
- fname = psprintf("%s.gz", path);
+ fname = psprintf("%s%s", path, suffix);
fp = cfopen(fname, mode, compression);
free_keep_errno(fname);
-#else
- fatal("not built with zlib support");
- fp = NULL; /* keep compiler quiet */
-#endif
}
return fp;
}
@@ -515,20 +482,21 @@ cfopen_write(const char *path, const char *mode, int compression)
* On failure, return NULL with an error code in errno.
*/
cfp *
-cfopen(const char *path, const char *mode, int compression)
+cfopen(const char *path, const char *mode, Compress *compression)
{
- cfp *fp = pg_malloc(sizeof(cfp));
+ cfp *fp = pg_malloc0(sizeof(cfp));
- if (compression != 0)
+ switch (compression->alg)
{
#ifdef HAVE_LIBZ
- if (compression != Z_DEFAULT_COMPRESSION)
+ case COMPR_ALG_LIBZ:
+ if (compression->level != Z_DEFAULT_COMPRESSION)
{
/* user has specified a compression level, so tell zlib to use it */
char mode_compression[32];
snprintf(mode_compression, sizeof(mode_compression), "%s%d",
- mode, compression);
+ mode, compression->level);
fp->compressedfp = gzopen(path, mode_compression);
}
else
@@ -537,30 +505,27 @@ cfopen(const char *path, const char *mode, int compression)
fp->compressedfp = gzopen(path, mode);
}
- fp->uncompressedfp = NULL;
if (fp->compressedfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
}
-#else
- fatal("not built with zlib support");
-#endif
- }
- else
- {
-#ifdef HAVE_LIBZ
- fp->compressedfp = NULL;
+ return fp;
#endif
+
+ case COMPR_ALG_NONE:
fp->uncompressedfp = fopen(path, mode);
if (fp->uncompressedfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
}
- }
+ return fp;
- return fp;
+ default:
+ /* Should not happen */
+ fatal("requested compression not available in this installation");
+ }
}
@@ -584,14 +549,13 @@ cfread(void *ptr, int size, cfp *fp)
fatal("could not read from input file: %s",
errnum == Z_ERRNO ? strerror(errno) : errmsg);
}
+ return ret;
}
- else
#endif
- {
- ret = fread(ptr, 1, size, fp->uncompressedfp);
- if (ret != size && !feof(fp->uncompressedfp))
- READ_ERROR_EXIT(fp->uncompressedfp);
- }
+
+ ret = fread(ptr, 1, size, fp->uncompressedfp);
+ if (ret != size && !feof(fp->uncompressedfp))
+ READ_ERROR_EXIT(fp->uncompressedfp);
return ret;
}
@@ -601,9 +565,8 @@ cfwrite(const void *ptr, int size, cfp *fp)
#ifdef HAVE_LIBZ
if (fp->compressedfp)
return gzwrite(fp->compressedfp, ptr, size);
- else
#endif
- return fwrite(ptr, 1, size, fp->uncompressedfp);
+ return fwrite(ptr, 1, size, fp->uncompressedfp);
}
int
@@ -622,15 +585,12 @@ cfgetc(cfp *fp)
else
fatal("could not read from input file: end of file");
}
+ return ret;
}
- else
#endif
- {
- ret = fgetc(fp->uncompressedfp);
- if (ret == EOF)
- READ_ERROR_EXIT(fp->uncompressedfp);
- }
-
+ ret = fgetc(fp->uncompressedfp);
+ if (ret == EOF)
+ READ_ERROR_EXIT(fp->uncompressedfp);
return ret;
}
@@ -640,9 +600,8 @@ cfgets(cfp *fp, char *buf, int len)
#ifdef HAVE_LIBZ
if (fp->compressedfp)
return gzgets(fp->compressedfp, buf, len);
- else
#endif
- return fgets(buf, len, fp->uncompressedfp);
+ return fgets(buf, len, fp->uncompressedfp);
}
int
@@ -660,15 +619,13 @@ cfclose(cfp *fp)
{
result = gzclose(fp->compressedfp);
fp->compressedfp = NULL;
+ return result;
}
- else
#endif
- {
- result = fclose(fp->uncompressedfp);
- fp->uncompressedfp = NULL;
- }
- free_keep_errno(fp);
+ result = fclose(fp->uncompressedfp);
+ fp->uncompressedfp = NULL;
+ free_keep_errno(fp);
return result;
}
@@ -678,9 +635,9 @@ cfeof(cfp *fp)
#ifdef HAVE_LIBZ
if (fp->compressedfp)
return gzeof(fp->compressedfp);
- else
#endif
- return feof(fp->uncompressedfp);
+
+ return feof(fp->uncompressedfp);
}
const char *
@@ -699,7 +656,6 @@ get_cfp_error(cfp *fp)
return strerror(errno);
}
-#ifdef HAVE_LIBZ
static int
hasSuffix(const char *filename, const char *suffix)
{
@@ -714,4 +670,18 @@ hasSuffix(const char *filename, const char *suffix)
suffixlen) == 0;
}
-#endif
+/*
+ * Return a string for the given AH's compression.
+ * The string is statically allocated.
+ */
+const char *
+compress_suffix(Compress *compression)
+{
+ switch (compression->alg)
+ {
+ case COMPR_ALG_LIBZ:
+ return ".gz";
+ default:
+ return "";
+ }
+}
diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h
index d2e6e1b854..2c073676eb 100644
--- a/src/bin/pg_dump/compress_io.h
+++ b/src/bin/pg_dump/compress_io.h
@@ -21,12 +21,6 @@
#define ZLIB_OUT_SIZE 4096
#define ZLIB_IN_SIZE 4096
-typedef enum
-{
- COMPR_ALG_NONE,
- COMPR_ALG_LIBZ
-} CompressionAlgorithm;
-
/* Prototype for callback function to WriteDataToArchive() */
typedef void (*WriteFunc) (ArchiveHandle *AH, const char *buf, size_t len);
@@ -46,8 +40,8 @@ typedef size_t (*ReadFunc) (ArchiveHandle *AH, char **buf, size_t *buflen);
/* struct definition appears in compress_io.c */
typedef struct CompressorState CompressorState;
-extern CompressorState *AllocateCompressor(int compression, WriteFunc writeF);
-extern void ReadDataFromArchive(ArchiveHandle *AH, int compression,
+extern CompressorState *AllocateCompressor(Compress *compression, WriteFunc writeF);
+extern void ReadDataFromArchive(ArchiveHandle *AH,
ReadFunc readF);
extern void WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs,
const void *data, size_t dLen);
@@ -56,9 +50,9 @@ extern void EndCompressor(ArchiveHandle *AH, CompressorState *cs);
typedef struct cfp cfp;
-extern cfp *cfopen(const char *path, const char *mode, int compression);
-extern cfp *cfopen_read(const char *path, const char *mode);
-extern cfp *cfopen_write(const char *path, const char *mode, int compression);
+extern cfp *cfopen(const char *path, const char *mode, Compress *compression);
+extern cfp *cfopen_read(const char *path, const char *mode, Compress *compression);
+extern cfp *cfopen_write(const char *path, const char *mode, Compress *compression);
extern int cfread(void *ptr, int size, cfp *fp);
extern int cfwrite(const void *ptr, int size, cfp *fp);
extern int cfgetc(cfp *fp);
@@ -67,4 +61,7 @@ extern int cfclose(cfp *fp);
extern int cfeof(cfp *fp);
extern const char *get_cfp_error(cfp *fp);
+/* also used by tar */
+extern const char * compress_suffix(Compress *compression);
+
#endif
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 9d0056a569..f2390b7937 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -72,6 +72,25 @@ typedef struct _connParams
char *override_dbname;
} ConnParams;
+typedef enum
+{
+ COMPR_ALG_DEFAULT = -1,
+ COMPR_ALG_NONE,
+ COMPR_ALG_LIBZ,
+} CompressionAlgorithm;
+/* Should be called "method" or "library" ? */
+
+typedef struct Compress {
+ CompressionAlgorithm alg;
+ int level;
+ /* Is a nondefault level set ? This is useful since different compression
+ * methods have different "default" levels. For now we assume the levels
+ * are all integer, though.
+ */
+ bool level_set;
+} Compress;
+
+
typedef struct _restoreOptions
{
int createDB; /* Issue commands to create the database */
@@ -125,7 +144,7 @@ typedef struct _restoreOptions
int noDataForFailedTables;
int exit_on_error;
- int compression;
+ Compress compression;
int suppressDumpWarnings; /* Suppress output of WARNING entries
* to stderr */
bool single_txn;
@@ -281,7 +300,7 @@ extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
/* Create a new archive */
extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupDumpWorker);
/* The --list option */
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 1f82c6499b..3eb6c55600 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -70,7 +70,7 @@ typedef struct _parallelReadyList
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupWorkerPtr);
static void _getObjectDescription(PQExpBuffer buf, TocEntry *te);
static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, bool isData);
@@ -98,7 +98,7 @@ static int _discoverArchiveFormat(ArchiveHandle *AH);
static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
-static void SetOutput(ArchiveHandle *AH, const char *filename, int compression);
+static void SetOutput(ArchiveHandle *AH, const char *filename, Compress *compress);
static OutputContext SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, OutputContext savedContext);
@@ -238,7 +238,7 @@ setupRestoreWorker(Archive *AHX)
/* Public */
Archive *
CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupDumpWorker)
{
@@ -253,7 +253,9 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
Archive *
OpenArchive(const char *FileSpec, const ArchiveFormat fmt)
{
- ArchiveHandle *AH = _allocAH(FileSpec, fmt, 0, true, archModeRead, setupRestoreWorker);
+ Compress compress = {0};
+ ArchiveHandle *AH = _allocAH(FileSpec, fmt, &compress, true, archModeRead,
+ setupRestoreWorker);
return (Archive *) AH;
}
@@ -382,7 +384,7 @@ RestoreArchive(Archive *AHX)
* Make sure we won't need (de)compression we haven't got
*/
#ifndef HAVE_LIBZ
- if (AH->compression != 0 && AH->PrintTocDataPtr != NULL)
+ if (AH->compression.alg != COMPR_ALG_NONE && AH->PrintTocDataPtr != NULL)
{
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
@@ -457,8 +459,8 @@ RestoreArchive(Archive *AHX)
* Setup the output file if necessary.
*/
sav = SaveOutput(AH);
- if (ropt->filename || ropt->compression)
- SetOutput(AH, ropt->filename, ropt->compression);
+ if (ropt->filename || ropt->compression.alg != COMPR_ALG_NONE)
+ SetOutput(AH, ropt->filename, &ropt->compression);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -738,7 +740,7 @@ RestoreArchive(Archive *AHX)
*/
AH->stage = STAGE_FINALIZING;
- if (ropt->filename || ropt->compression)
+ if (ropt->filename || ropt->compression.alg != COMPR_ALG_NONE)
RestoreOutput(AH, sav);
if (ropt->useDB)
@@ -1123,8 +1125,9 @@ PrintTOCSummary(Archive *AHX)
char stamp_str[64];
sav = SaveOutput(AH);
+ Assert(ropt->compression.alg == COMPR_ALG_NONE);
if (ropt->filename)
- SetOutput(AH, ropt->filename, 0 /* no compression */ );
+ SetOutput(AH, ropt->filename, &ropt->compression);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1133,7 +1136,7 @@ PrintTOCSummary(Archive *AHX)
ahprintf(AH, ";\n; Archive created at %s\n", stamp_str);
ahprintf(AH, "; dbname: %s\n; TOC Entries: %d\n; Compression: %d\n",
sanitize_line(AH->archdbname, false),
- AH->tocCount, AH->compression);
+ AH->tocCount, AH->compression.alg);
switch (AH->format)
{
@@ -1487,7 +1490,7 @@ archprintf(Archive *AH, const char *fmt,...)
*******************************/
static void
-SetOutput(ArchiveHandle *AH, const char *filename, int compression)
+SetOutput(ArchiveHandle *AH, const char *filename, Compress *compression)
{
int fn;
@@ -1510,12 +1513,12 @@ SetOutput(ArchiveHandle *AH, const char *filename, int compression)
/* If compression explicitly requested, use gzopen */
#ifdef HAVE_LIBZ
- if (compression != 0)
+ if (compression->alg != COMPR_ALG_NONE)
{
char fmode[14];
/* Don't use PG_BINARY_x since this is zlib */
- sprintf(fmode, "wb%d", compression);
+ sprintf(fmode, "wb%d", compression->level);
if (fn >= 0)
AH->OF = gzdopen(dup(fn), fmode);
else
@@ -2259,7 +2262,7 @@ _discoverArchiveFormat(ArchiveHandle *AH)
*/
static ArchiveHandle *
_allocAH(const char *FileSpec, const ArchiveFormat fmt,
- const int compression, bool dosync, ArchiveMode mode,
+ Compress *compression, bool dosync, ArchiveMode mode,
SetupWorkerPtrType setupWorkerPtr)
{
ArchiveHandle *AH;
@@ -2310,7 +2313,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
AH->toc->prev = AH->toc;
AH->mode = mode;
- AH->compression = compression;
+ AH->compression = *compression;
AH->dosync = dosync;
memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse));
@@ -2325,7 +2328,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
* Force stdin/stdout into binary mode if that is what we are using.
*/
#ifdef WIN32
- if ((fmt != archNull || compression != 0) &&
+ if ((fmt != archNull || compression->alg != COMPR_ALG_NONE) &&
(AH->fSpec == NULL || strcmp(AH->fSpec, "") == 0))
{
if (mode == archModeWrite)
@@ -3741,7 +3744,7 @@ WriteHead(ArchiveHandle *AH)
AH->WriteBytePtr(AH, AH->intSize);
AH->WriteBytePtr(AH, AH->offSize);
AH->WriteBytePtr(AH, AH->format);
- WriteInt(AH, AH->compression);
+ WriteInt(AH, AH->compression.alg);
crtm = *localtime(&AH->createDate);
WriteInt(AH, crtm.tm_sec);
WriteInt(AH, crtm.tm_min);
@@ -3816,15 +3819,15 @@ ReadHead(ArchiveHandle *AH)
if (AH->version >= K_VERS_1_2)
{
if (AH->version < K_VERS_1_4)
- AH->compression = AH->ReadBytePtr(AH);
+ AH->compression.alg = AH->ReadBytePtr(AH);
else
- AH->compression = ReadInt(AH);
+ AH->compression.alg = ReadInt(AH);
}
else
- AH->compression = Z_DEFAULT_COMPRESSION;
+ AH->compression.alg = Z_DEFAULT_COMPRESSION;
#ifndef HAVE_LIBZ
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
pg_log_warning("archive is compressed, but this installation does not support compression -- no data will be available");
#endif
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index a8ea5c7eae..6e033d040e 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -47,7 +47,8 @@
#define GZWRITE(p, s, n, fh) (fwrite(p, s, n, fh) * (s))
#define GZREAD(p, s, n, fh) fread(p, s, n, fh)
#define GZEOF(fh) feof(fh)
-/* this is just the redefinition of a libz constant */
+/* this is just the redefinition of a libz constant, in case zlib isn't
+ * available */
#define Z_DEFAULT_COMPRESSION (-1)
typedef struct _z_stream
@@ -329,14 +330,7 @@ struct _archiveHandle
DumpId *tableDataId; /* TABLE DATA ids, indexed by table dumpId */
struct _tocEntry *currToc; /* Used when dumping data */
- int compression; /*---------
- * Compression requested on open().
- * Possible values for compression:
- * -1 Z_DEFAULT_COMPRESSION
- * 0 COMPRESSION_NONE
- * 1-9 levels for gzip compression
- *---------
- */
+ Compress compression; /* Compression requested on open */
bool dosync; /* data requested to be synced on sight */
ArchiveMode mode; /* File mode - r or w */
void *formatData; /* Header data specific to file format */
diff --git a/src/bin/pg_dump/pg_backup_custom.c b/src/bin/pg_dump/pg_backup_custom.c
index 77d402c323..55a887a236 100644
--- a/src/bin/pg_dump/pg_backup_custom.c
+++ b/src/bin/pg_dump/pg_backup_custom.c
@@ -298,7 +298,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te)
_WriteByte(AH, BLK_DATA); /* Block type */
WriteInt(AH, te->dumpId); /* For sanity check */
- ctx->cs = AllocateCompressor(AH->compression, _CustomWriteFunc);
+ ctx->cs = AllocateCompressor(&AH->compression, _CustomWriteFunc);
}
/*
@@ -377,7 +377,7 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid)
WriteInt(AH, oid);
- ctx->cs = AllocateCompressor(AH->compression, _CustomWriteFunc);
+ ctx->cs = AllocateCompressor(&AH->compression, _CustomWriteFunc);
}
/*
@@ -566,7 +566,7 @@ _PrintTocData(ArchiveHandle *AH, TocEntry *te)
static void
_PrintData(ArchiveHandle *AH)
{
- ReadDataFromArchive(AH, AH->compression, _CustomReadFunc);
+ ReadDataFromArchive(AH, _CustomReadFunc);
}
static void
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 650b542fce..8bf869c6ca 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -202,7 +202,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH)
setFilePath(AH, fname, "toc.dat");
- tocFH = cfopen_read(fname, PG_BINARY_R);
+ tocFH = cfopen_read(fname, PG_BINARY_R, &AH->compression);
if (tocFH == NULL)
fatal("could not open input file \"%s\": %m", fname);
@@ -327,7 +327,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te)
setFilePath(AH, fname, tctx->filename);
- ctx->dataFH = cfopen_write(fname, PG_BINARY_W, AH->compression);
+ ctx->dataFH = cfopen_write(fname, PG_BINARY_W, &AH->compression);
if (ctx->dataFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
}
@@ -388,7 +388,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename)
if (!filename)
return;
- cfp = cfopen_read(filename, PG_BINARY_R);
+ cfp = cfopen_read(filename, PG_BINARY_R, &AH->compression);
if (!cfp)
fatal("could not open input file \"%s\": %m", filename);
@@ -435,12 +435,13 @@ _LoadBlobs(ArchiveHandle *AH)
lclContext *ctx = (lclContext *) AH->formatData;
char fname[MAXPGPATH];
char line[MAXPGPATH];
+ Compress nocompression = {0};
StartRestoreBlobs(AH);
setFilePath(AH, fname, "blobs.toc");
- ctx->blobsTocFH = cfopen_read(fname, PG_BINARY_R);
+ ctx->blobsTocFH = cfopen_read(fname, PG_BINARY_R, &nocompression);
if (ctx->blobsTocFH == NULL)
fatal("could not open large object TOC file \"%s\" for input: %m",
@@ -573,6 +574,7 @@ _CloseArchive(ArchiveHandle *AH)
{
cfp *tocFH;
char fname[MAXPGPATH];
+ Compress nocompression = {0};
setFilePath(AH, fname, "toc.dat");
@@ -580,7 +582,7 @@ _CloseArchive(ArchiveHandle *AH)
ctx->pstate = ParallelBackupStart(AH);
/* The TOC is always created uncompressed */
- tocFH = cfopen_write(fname, PG_BINARY_W, 0);
+ tocFH = cfopen_write(fname, PG_BINARY_W, &nocompression);
if (tocFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
ctx->dataFH = tocFH;
@@ -639,11 +641,12 @@ _StartBlobs(ArchiveHandle *AH, TocEntry *te)
{
lclContext *ctx = (lclContext *) AH->formatData;
char fname[MAXPGPATH];
+ Compress nocompression = {0};
setFilePath(AH, fname, "blobs.toc");
/* The blob TOC file is never compressed */
- ctx->blobsTocFH = cfopen_write(fname, "ab", 0);
+ ctx->blobsTocFH = cfopen_write(fname, "ab", &nocompression);
if (ctx->blobsTocFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
}
@@ -661,7 +664,7 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid)
snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid);
- ctx->dataFH = cfopen_write(fname, PG_BINARY_W, AH->compression);
+ ctx->dataFH = cfopen_write(fname, PG_BINARY_W, &AH->compression);
if (ctx->dataFH == NULL)
fatal("could not open output file \"%s\": %m", fname);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index 61c9c87a9f..4ba79ab924 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -39,6 +39,7 @@
#include "pg_backup_archiver.h"
#include "pg_backup_tar.h"
#include "pg_backup_utils.h"
+#include "compress_io.h"
#include "pgtar.h"
static void _ArchiveEntry(ArchiveHandle *AH, TocEntry *te);
@@ -196,10 +197,10 @@ InitArchiveFmt_Tar(ArchiveHandle *AH)
/*
* We don't support compression because reading the files back is not
- * possible since gzdopen uses buffered IO which totally screws file
+ * possible since gzdopen uses buffered IO which totally screws file XXX
* positioning.
*/
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
fatal("compression is not supported by tar archive format");
}
else
@@ -254,14 +255,8 @@ _ArchiveEntry(ArchiveHandle *AH, TocEntry *te)
ctx = (lclTocEntry *) pg_malloc0(sizeof(lclTocEntry));
if (te->dataDumper != NULL)
{
-#ifdef HAVE_LIBZ
- if (AH->compression == 0)
- sprintf(fn, "%d.dat", te->dumpId);
- else
- sprintf(fn, "%d.dat.gz", te->dumpId);
-#else
- sprintf(fn, "%d.dat", te->dumpId);
-#endif
+ const char *suffix = compress_suffix(&AH->compression);
+ sprintf(fn, "%d.dat%s", te->dumpId, suffix);
ctx->filename = pg_strdup(fn);
}
else
@@ -352,7 +347,7 @@ tarOpen(ArchiveHandle *AH, const char *filename, char mode)
#ifdef HAVE_LIBZ
- if (AH->compression == 0)
+ if (AH->compression.alg == COMPR_ALG_NONE)
tm->nFH = ctx->tarFH;
else
fatal("compression is not supported by tar archive format");
@@ -413,9 +408,9 @@ tarOpen(ArchiveHandle *AH, const char *filename, char mode)
#ifdef HAVE_LIBZ
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
{
- sprintf(fmode, "wb%d", AH->compression);
+ sprintf(fmode, "wb%d", AH->compression.level);
tm->zFH = gzdopen(dup(fileno(tm->tmpFH)), fmode);
if (tm->zFH == NULL)
fatal("could not open temporary file");
@@ -443,7 +438,7 @@ tarClose(ArchiveHandle *AH, TAR_MEMBER *th)
/*
* Close the GZ file since we dup'd. This will flush the buffers.
*/
- if (AH->compression != 0)
+ if (AH->compression.alg != COMPR_ALG_NONE)
if (GZCLOSE(th->zFH) != 0)
fatal("could not close tar member");
@@ -868,7 +863,7 @@ _CloseArchive(ArchiveHandle *AH)
memcpy(ropt, AH->public.ropt, sizeof(RestoreOptions));
ropt->filename = NULL;
ropt->dropSchema = 1;
- ropt->compression = 0;
+ ropt->compression.alg = COMPR_ALG_NONE;
ropt->superuser = NULL;
ropt->suppressDumpWarnings = true;
@@ -952,16 +947,12 @@ _StartBlob(ArchiveHandle *AH, TocEntry *te, Oid oid)
lclContext *ctx = (lclContext *) AH->formatData;
lclTocEntry *tctx = (lclTocEntry *) te->formatData;
char fname[255];
- char *sfx;
+ const char *sfx;
if (oid == 0)
fatal("invalid OID for large object (%u)", oid);
- if (AH->compression != 0)
- sfx = ".gz";
- else
- sfx = "";
-
+ sfx = compress_suffix(&AH->compression);
sprintf(fname, "blob_%u.dat%s", oid, sfx);
tarPrintf(ctx->blobToc, "%u %s\n", oid, fname);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 1ab98a2286..4cbc79aedc 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -59,6 +59,7 @@
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
+#include "compress_io.h"
#include "pg_backup_db.h"
#include "pg_backup_utils.h"
#include "pg_dump.h"
@@ -297,6 +298,82 @@ static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+/* Parse the string into compression options */
+static void
+parse_compression(const char *optarg, Compress *compress)
+{
+ if (optarg[0] == '0' && optarg[1] == '\0')
+ compress->alg = COMPR_ALG_NONE;
+ else if ((optarg[0] > '0' && optarg[0] <= '9') ||
+ optarg[0] == '-')
+ {
+ compress->alg = COMPR_ALG_LIBZ;
+ compress->level_set = true;
+ compress->level = atoi(optarg);
+ if (optarg[1] != '\0')
+ {
+ pg_log_error("compression level must be in range 0..9");
+ exit_nicely(1);
+ }
+ }
+ else
+ {
+ /* Parse a more flexible string like level=3 alg=zlib opts=long */
+ for (;;)
+ {
+ char *eq = strchr(optarg, '=');
+ int len;
+
+ if (eq == NULL)
+ {
+ pg_log_error("compression options must be key=value: %s", optarg);
+ exit_nicely(1);
+ }
+
+ len = eq - optarg;
+ if (strncmp(optarg, "alg", len) == 0)
+ {
+ if (strchr(eq, ' '))
+ len = strchr(eq, ' ') - eq - 1;
+ else
+ len = strlen(eq) - len;
+ if (strncmp(1+eq, "zlib", len) == 0 ||
+ strncmp(1+eq, "libz", len) == 0)
+ compress->alg = COMPR_ALG_LIBZ;
+ else
+ {
+ pg_log_error("unknown compression algorithm: %s", 1+eq);
+ exit_nicely(1);
+ }
+ }
+ else if (strncmp(optarg, "level", len) == 0)
+ {
+ compress->level = atoi(1+eq);
+ compress->level_set = true;
+ }
+ else
+ {
+ pg_log_error("unknown compression setting: %s", optarg);
+ exit_nicely(1);
+ }
+
+ optarg = strchr(eq, ' ');
+ if (!optarg++)
+ break;
+ }
+
+ if (!compress->level_set)
+ {
+ const int default_compress_level[] = {
+ 0, /* COMPR_ALG_NONE */
+ Z_DEFAULT_COMPRESSION, /* COMPR_ALG_ZLIB */
+ };
+
+ compress->level = default_compress_level[compress->alg];
+ }
+ }
+}
+
int
main(int argc, char **argv)
{
@@ -319,7 +396,7 @@ main(int argc, char **argv)
char *use_role = NULL;
long rowsPerInsert;
int numWorkers = 1;
- int compressLevel = -1;
+ Compress compress = { .alg = COMPR_ALG_DEFAULT };
int plainText = 0;
ArchiveFormat archiveFormat = archUnknown;
ArchiveMode archiveMode;
@@ -532,12 +609,7 @@ main(int argc, char **argv)
break;
case 'Z': /* Compression Level */
- compressLevel = atoi(optarg);
- if (compressLevel < 0 || compressLevel > 9)
- {
- pg_log_error("compression level must be in range 0..9");
- exit_nicely(1);
- }
+ parse_compression(optarg, &compress);
break;
case 0:
@@ -679,20 +751,28 @@ main(int argc, char **argv)
plainText = 1;
/* Custom and directory formats are compressed by default, others not */
- if (compressLevel == -1)
+ if (compress.alg == COMPR_ALG_DEFAULT)
{
-#ifdef HAVE_LIBZ
if (archiveFormat == archCustom || archiveFormat == archDirectory)
- compressLevel = Z_DEFAULT_COMPRESSION;
- else
+ {
+#ifdef HAVE_LIBZ
+ compress.alg = COMPR_ALG_LIBZ;
+ compress.level = Z_DEFAULT_COMPRESSION;
#endif
- compressLevel = 0;
+ }
+ else
+ {
+ compress.alg = COMPR_ALG_NONE;
+ compress.level = 0;
+ }
}
#ifndef HAVE_LIBZ
- if (compressLevel != 0)
+ if (compress.alg == COMPR_ALG_LIBZ)
+ {
pg_log_warning("requested compression not available in this installation -- archive will be uncompressed");
- compressLevel = 0;
+ compress.alg = 0;
+ }
#endif
/*
@@ -723,7 +803,7 @@ main(int argc, char **argv)
fatal("option --index-collation-versions-unknown only works in binary upgrade mode");
/* Open the output file */
- fout = CreateArchive(filename, archiveFormat, compressLevel, dosync,
+ fout = CreateArchive(filename, archiveFormat, &compress, dosync,
archiveMode, setupDumpWorker);
/* Make dump options accessible right away */
@@ -957,10 +1037,7 @@ main(int argc, char **argv)
ropt->sequence_data = dopt.sequence_data;
ropt->binary_upgrade = dopt.binary_upgrade;
- if (compressLevel == -1)
- ropt->compression = 0;
- else
- ropt->compression = compressLevel;
+ ropt->compression = compress;
ropt->suppressDumpWarnings = true; /* We've already shown them */
--
2.17.0
[text/x-diff] 0004-struct-compressLibs.patch (5.9K, ../../20210104025321.GA9712@telsasoft.com/5-0004-struct-compressLibs.patch)
download | inline diff:
From b26d3fa15723ab38057276471942e79cf4c7789b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Thu, 24 Dec 2020 22:08:43 -0600
Subject: [PATCH 04/20] struct compressLibs
---
src/bin/pg_dump/compress_io.c | 55 +++++++++++++++++++--------
src/bin/pg_dump/compress_io.h | 7 ++++
src/bin/pg_dump/pg_backup_directory.c | 20 ++++++----
src/bin/pg_dump/pg_dump.c | 25 +++++++-----
4 files changed, 75 insertions(+), 32 deletions(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index db16fd33f2..21957d68f3 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -56,6 +56,18 @@
#include "compress_io.h"
#include "pg_backup_utils.h"
+const struct compressLibs
+compresslibs[] = {
+ { COMPR_ALG_NONE, "no", "", 0, },
+ { COMPR_ALG_NONE, "none", "", 0, }, /* Alternate name */
+
+// #ifdef HAVE_LIBZ?
+ { COMPR_ALG_LIBZ, "libz", ".gz", Z_DEFAULT_COMPRESSION },
+ { COMPR_ALG_LIBZ, "zlib", ".gz", Z_DEFAULT_COMPRESSION }, /* Alternate name */
+
+ { 0, NULL, } /* sentinel */
+};
+
/*----------------------
* Compressor API
*----------------------
@@ -401,7 +413,7 @@ struct cfp
#endif
};
-static int hasSuffix(const char *filename, const char *suffix);
+static int hasSuffix(const char *filename);
/* free() without changing errno; useful in several places below */
static void
@@ -428,7 +440,7 @@ cfopen_read(const char *path, const char *mode, Compress *compression)
{
cfp *fp;
- if (hasSuffix(path, ".gz"))
+ if (hasSuffix(path))
fp = cfopen(path, mode, compression);
else
{
@@ -656,18 +668,29 @@ get_cfp_error(cfp *fp)
return strerror(errno);
}
+/* Return true iff the filename has a known compression suffix */
static int
-hasSuffix(const char *filename, const char *suffix)
+hasSuffix(const char *filename)
{
- int filenamelen = strlen(filename);
- int suffixlen = strlen(suffix);
+ for (int i = 0; compresslibs[i].name != NULL; ++i)
+ {
+ const char *suffix = compresslibs[i].suffix;
+ int filenamelen = strlen(filename);
+ int suffixlen = strlen(suffix);
- if (filenamelen < suffixlen)
- return 0;
+ /* COMPR_ALG_NONE has an empty "suffix", which doesn't count */
+ if (suffixlen == 0)
+ continue;
+
+ if (filenamelen < suffixlen)
+ continue;
- return memcmp(&filename[filenamelen - suffixlen],
- suffix,
- suffixlen) == 0;
+ if (memcmp(&filename[filenamelen - suffixlen],
+ suffix, suffixlen) == 0)
+ return true;
+ }
+
+ return false;
}
/*
@@ -677,11 +700,13 @@ hasSuffix(const char *filename, const char *suffix)
const char *
compress_suffix(Compress *compression)
{
- switch (compression->alg)
+ for (int i = 0; compresslibs[i].name != NULL; ++i)
{
- case COMPR_ALG_LIBZ:
- return ".gz";
- default:
- return "";
+ if (compression->alg != compresslibs[i].alg)
+ continue;
+
+ return compresslibs[i].suffix;
}
+
+ return "";
}
diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h
index 2c073676eb..fb9d659acc 100644
--- a/src/bin/pg_dump/compress_io.h
+++ b/src/bin/pg_dump/compress_io.h
@@ -47,6 +47,13 @@ extern void WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs,
const void *data, size_t dLen);
extern void EndCompressor(ArchiveHandle *AH, CompressorState *cs);
+struct compressLibs {
+ const CompressionAlgorithm alg;
+ const char *name; /* Name in -Z alg= */
+ const char *suffix; /* file extension */
+ const int defaultlevel; /* Default compression level */
+};
+extern const struct compressLibs compresslibs[];
typedef struct cfp cfp;
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 8bf869c6ca..75c1bf22e4 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -768,14 +768,20 @@ _PrepParallelRestore(ArchiveHandle *AH)
*/
setFilePath(AH, fname, tctx->filename);
- if (stat(fname, &st) == 0)
- te->dataLength = st.st_size;
- else
+ for (int i = 0; compresslibs[i].name != NULL; ++i)
{
- /* It might be compressed */
- strlcat(fname, ".gz", sizeof(fname));
- if (stat(fname, &st) == 0)
- te->dataLength = st.st_size;
+ char filename[MAXPGPATH];
+ int ret;
+
+ snprintf(filename, sizeof(filename), "%s%s", fname,
+ compresslibs[i].suffix);
+
+ ret = stat(fname, &st);
+ if (ret < 0) // && errno == ENOENT)
+ continue;
+
+ te->dataLength = st.st_size;
+ break;
}
/*
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 4cbc79aedc..75985fd4d3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -318,7 +318,7 @@ parse_compression(const char *optarg, Compress *compress)
}
else
{
- /* Parse a more flexible string like level=3 alg=zlib opts=long */
+ /* Parse a more flexible string like -Z level=3 -Z alg=zlib -Z checksum=1 */
for (;;)
{
char *eq = strchr(optarg, '=');
@@ -333,14 +333,19 @@ parse_compression(const char *optarg, Compress *compress)
len = eq - optarg;
if (strncmp(optarg, "alg", len) == 0)
{
- if (strchr(eq, ' '))
- len = strchr(eq, ' ') - eq - 1;
- else
- len = strlen(eq) - len;
- if (strncmp(1+eq, "zlib", len) == 0 ||
- strncmp(1+eq, "libz", len) == 0)
- compress->alg = COMPR_ALG_LIBZ;
- else
+ len = strlen(eq) - len;
+
+ for (int i = 0; compresslibs[i].name != NULL; ++i)
+ {
+ if (strlen(1+eq) != strlen(compresslibs[i].name))
+ continue;
+ if (strncmp(1+eq, compresslibs[i].name, len) != 0)
+ continue;
+ compress->alg = compresslibs[i].alg;
+ break;
+ }
+
+ if (compress->alg == COMPR_ALG_DEFAULT)
{
pg_log_error("unknown compression algorithm: %s", 1+eq);
exit_nicely(1);
@@ -363,7 +368,7 @@ parse_compression(const char *optarg, Compress *compress)
}
if (!compress->level_set)
- {
+ { // XXX
const int default_compress_level[] = {
0, /* COMPR_ALG_NONE */
Z_DEFAULT_COMPRESSION, /* COMPR_ALG_ZLIB */
--
2.17.0
[text/x-diff] 0005-Use-cf-abstraction-in-archiver-and-tar.patch (14.8K, ../../20210104025321.GA9712@telsasoft.com/6-0005-Use-cf-abstraction-in-archiver-and-tar.patch)
download | inline diff:
From 1bac5d02bbcc9c34dcb44b358bc27f3d204bb584 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 25 Dec 2020 00:34:01 -0600
Subject: [PATCH 05/20] Use cf* abstraction in archiver and tar
..rather than direct, conditional calls to gzopen/fopen.
See also: bf9aa490db24b2334b3595ee33653bf2fe39208c
---
src/bin/pg_dump/compress_io.c | 53 +++++++++++++
src/bin/pg_dump/compress_io.h | 5 ++
src/bin/pg_dump/pg_backup_archiver.c | 109 +++++++++------------------
src/bin/pg_dump/pg_backup_archiver.h | 16 ++--
src/bin/pg_dump/pg_backup_tar.c | 85 +++++----------------
5 files changed, 117 insertions(+), 151 deletions(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 21957d68f3..d66d6f60f5 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -540,6 +540,58 @@ cfopen(const char *path, const char *mode, Compress *compression)
}
}
+/*
+ * Open a file descriptor, with specified compression.
+ * Returns an opaque cfp object.
+ */
+cfp *
+cfdopen(int fd, const char *mode, Compress *compression)
+{
+ cfp *fp = pg_malloc0(sizeof(cfp));
+
+ switch (compression->alg)
+ {
+#ifdef HAVE_LIBZ
+ case COMPR_ALG_LIBZ:
+ if (compression->level != Z_DEFAULT_COMPRESSION)
+ {
+ /* user has specified a compression level, so tell zlib to use it */
+ char mode_compression[32];
+
+ snprintf(mode_compression, sizeof(mode_compression), "%s%d",
+ mode, compression->level);
+ fp->compressedfp = gzdopen(fd, mode_compression);
+ }
+ else
+ {
+ /* don't specify a level, just use the zlib default */
+ fp->compressedfp = gzdopen(fd, mode);
+ }
+
+ if (fp->compressedfp == NULL)
+ {
+ free_keep_errno(fp);
+ fp = NULL;
+ }
+ return fp;
+#endif
+
+ case COMPR_ALG_NONE:
+ fp->uncompressedfp = fdopen(fd, mode);
+ if (fp->uncompressedfp == NULL)
+ {
+ free_keep_errno(fp);
+ fp = NULL;
+ }
+ else
+ setvbuf(fp->uncompressedfp, NULL, _IONBF, 0);
+ return fp;
+
+ default:
+ /* Should not happen */
+ fatal("requested compression not available in this installation");
+ }
+}
int
cfread(void *ptr, int size, cfp *fp)
@@ -616,6 +668,7 @@ cfgets(cfp *fp, char *buf, int len)
return fgets(buf, len, fp->uncompressedfp);
}
+/* Close the given compressed or uncompressed stream; return 0 on success. */
int
cfclose(cfp *fp)
{
diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h
index fb9d659acc..318a6b5340 100644
--- a/src/bin/pg_dump/compress_io.h
+++ b/src/bin/pg_dump/compress_io.h
@@ -21,6 +21,10 @@
#define ZLIB_OUT_SIZE 4096
#define ZLIB_IN_SIZE 4096
+/* Forward declaration */
+struct ArchiveHandle;
+typedef struct _archiveHandle ArchiveHandle;
+
/* Prototype for callback function to WriteDataToArchive() */
typedef void (*WriteFunc) (ArchiveHandle *AH, const char *buf, size_t len);
@@ -58,6 +62,7 @@ extern const struct compressLibs compresslibs[];
typedef struct cfp cfp;
extern cfp *cfopen(const char *path, const char *mode, Compress *compression);
+extern cfp *cfdopen(int fd, const char *mode, Compress *compression);
extern cfp *cfopen_read(const char *path, const char *mode, Compress *compression);
extern cfp *cfopen_write(const char *path, const char *mode, Compress *compression);
extern int cfread(void *ptr, int size, cfp *fp);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 3eb6c55600..bd06fbb787 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -39,17 +39,11 @@
#include "pg_backup_archiver.h"
#include "pg_backup_db.h"
#include "pg_backup_utils.h"
+#include "compress_io.h"
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/* state needed to save/restore an archive's output target */
-typedef struct _outputContext
-{
- void *OF;
- int gzOut;
-} OutputContext;
-
/*
* State for tracking TocEntrys that are ready to process during a parallel
* restore. (This used to be a list, and we still call it that, though now
@@ -99,8 +93,8 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename, Compress *compress);
-static OutputContext SaveOutput(ArchiveHandle *AH);
-static void RestoreOutput(ArchiveHandle *AH, OutputContext savedContext);
+static cfp *SaveOutput(ArchiveHandle *AH);
+static void RestoreOutput(ArchiveHandle *AH, cfp *fp);
static int restore_toc_entry(ArchiveHandle *AH, TocEntry *te, bool is_parallel);
static void restore_toc_entries_prefork(ArchiveHandle *AH,
@@ -270,10 +264,8 @@ CloseArchive(Archive *AHX)
AH->ClosePtr(AH);
/* Close the output */
- if (AH->gzOut)
- res = GZCLOSE(AH->OF);
- else if (AH->OF != stdout)
- res = fclose(AH->OF);
+ if ((FILE *)AH->OF != stdout)
+ res = cfclose(AH->OF);
if (res != 0)
fatal("could not close output file: %m");
@@ -355,7 +347,7 @@ RestoreArchive(Archive *AHX)
RestoreOptions *ropt = AH->public.ropt;
bool parallel_mode;
TocEntry *te;
- OutputContext sav;
+ cfp *sav;
AH->stage = STAGE_INITIALIZING;
@@ -1120,7 +1112,7 @@ PrintTOCSummary(Archive *AHX)
RestoreOptions *ropt = AH->public.ropt;
TocEntry *te;
teSection curSection;
- OutputContext sav;
+ cfp *sav;
const char *fmtName;
char stamp_str[64];
@@ -1492,6 +1484,7 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename, Compress *compression)
{
+ char fmode[14];
int fn;
if (filename)
@@ -1511,38 +1504,22 @@ SetOutput(ArchiveHandle *AH, const char *filename, Compress *compression)
else
fn = fileno(stdout);
- /* If compression explicitly requested, use gzopen */
-#ifdef HAVE_LIBZ
- if (compression->alg != COMPR_ALG_NONE)
+ if (fn >= 0)
{
- char fmode[14];
+ /* Handle output to stdout */
+ sprintf(fmode, "%sb%d",
+ AH->mode == archModeAppend ? PG_BINARY_A : PG_BINARY_W,
+ compression->level);
- /* Don't use PG_BINARY_x since this is zlib */
- sprintf(fmode, "wb%d", compression->level);
- if (fn >= 0)
- AH->OF = gzdopen(dup(fn), fmode);
- else
- AH->OF = gzopen(filename, fmode);
- AH->gzOut = 1;
+ AH->OF = cfdopen(dup(fn), fmode, compression);
}
else
-#endif
- { /* Use fopen */
- if (AH->mode == archModeAppend)
- {
- if (fn >= 0)
- AH->OF = fdopen(dup(fn), PG_BINARY_A);
- else
- AH->OF = fopen(filename, PG_BINARY_A);
- }
- else
- {
- if (fn >= 0)
- AH->OF = fdopen(dup(fn), PG_BINARY_W);
- else
- AH->OF = fopen(filename, PG_BINARY_W);
- }
- AH->gzOut = 0;
+ {
+ Assert(filename != NULL);
+ sprintf(fmode, "%cb%d",
+ AH->mode == archModeAppend ? 'a' : 'w',
+ compression->level);
+ AH->OF = cfopen(filename, fmode, compression);
}
if (!AH->OF)
@@ -1554,32 +1531,22 @@ SetOutput(ArchiveHandle *AH, const char *filename, Compress *compression)
}
}
-static OutputContext
+/* Return a pointer to the old FP */
+static cfp *
SaveOutput(ArchiveHandle *AH)
{
- OutputContext sav;
-
- sav.OF = AH->OF;
- sav.gzOut = AH->gzOut;
-
- return sav;
+ return AH->OF;
}
static void
-RestoreOutput(ArchiveHandle *AH, OutputContext savedContext)
+RestoreOutput(ArchiveHandle *AH, cfp *savedContext)
{
int res;
-
- if (AH->gzOut)
- res = GZCLOSE(AH->OF);
- else
- res = fclose(AH->OF);
-
+ res = cfclose(AH->OF);
if (res != 0)
fatal("could not close output file: %m");
- AH->gzOut = savedContext.gzOut;
- AH->OF = savedContext.OF;
+ AH->OF = savedContext;
}
@@ -1703,22 +1670,14 @@ ahwrite(const void *ptr, size_t size, size_t nmemb, ArchiveHandle *AH)
bytes_written = size * nmemb;
}
- else if (AH->gzOut)
- bytes_written = GZWRITE(ptr, size, nmemb, AH->OF);
else if (AH->CustomOutPtr)
bytes_written = AH->CustomOutPtr(AH, ptr, size * nmemb);
-
+ else if (RestoringToDB(AH))
+ /* If we're doing a restore, and it's direct to DB, and we're
+ * connected then send it to the DB. */
+ bytes_written = ExecuteSqlCommandBuf(&AH->public, (const char *) ptr, size * nmemb);
else
- {
- /*
- * If we're doing a restore, and it's direct to DB, and we're
- * connected then send it to the DB.
- */
- if (RestoringToDB(AH))
- bytes_written = ExecuteSqlCommandBuf(&AH->public, (const char *) ptr, size * nmemb);
- else
- bytes_written = fwrite(ptr, size, nmemb, AH->OF) * size;
- }
+ bytes_written = cfwrite(ptr, size * nmemb, AH->OF);
if (bytes_written != size * nmemb)
WRITE_ERROR_EXIT;
@@ -2127,6 +2086,7 @@ _discoverArchiveFormat(ArchiveHandle *AH)
fh = stdin;
if (!fh)
fatal("could not open input file: %m");
+ setvbuf(fh, NULL, _IONBF, 0);
}
if ((cnt = fread(sig, 1, 5, fh)) != 5)
@@ -2266,6 +2226,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
SetupWorkerPtrType setupWorkerPtr)
{
ArchiveHandle *AH;
+ Compress nocompression = {0};
pg_log_debug("allocating AH for %s, format %d",
FileSpec ? FileSpec : "(stdio)", fmt);
@@ -2319,8 +2280,8 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse));
/* Open stdout with no compression for AH output handle */
- AH->gzOut = 0;
- AH->OF = stdout;
+ AH->OF = cfdopen(fileno(stdout), "w", &nocompression);
+ // AH->OF = cfdopen(STDOUT_FILENO, "w", compression); // XXX
/*
* On Windows, we need to use binary mode to read/write non-text files,
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 6e033d040e..9f511b49b9 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -30,6 +30,9 @@
#include "pg_backup.h"
#include "pqexpbuffer.h"
+/* Forward declaration XXX: CIRCULAR */
+typedef struct cfp cfp;
+
#define LOBBUFSIZE 16384
/*
@@ -38,19 +41,11 @@
*/
#ifdef HAVE_LIBZ
#include <zlib.h>
-#define GZCLOSE(fh) gzclose(fh)
-#define GZWRITE(p, s, n, fh) gzwrite(fh, p, (n) * (s))
-#define GZREAD(p, s, n, fh) gzread(fh, p, (n) * (s))
-#define GZEOF(fh) gzeof(fh)
#else
-#define GZCLOSE(fh) fclose(fh)
-#define GZWRITE(p, s, n, fh) (fwrite(p, s, n, fh) * (s))
-#define GZREAD(p, s, n, fh) fread(p, s, n, fh)
-#define GZEOF(fh) feof(fh)
+
/* this is just the redefinition of a libz constant, in case zlib isn't
* available */
#define Z_DEFAULT_COMPRESSION (-1)
-
typedef struct _z_stream
{
void *next_in;
@@ -318,8 +313,7 @@ struct _archiveHandle
char *fSpec; /* Archive File Spec */
FILE *FH; /* General purpose file handle */
- void *OF;
- int gzOut; /* Output file */
+ cfp *OF; /* Output file (compressed or not) */
struct _tocEntry *toc; /* Header of circular list of TOC entries */
int tocCount; /* Number of TOC entries */
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index 4ba79ab924..16f4e0792a 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -66,12 +66,7 @@ static void _EndBlobs(ArchiveHandle *AH, TocEntry *te);
typedef struct
{
-#ifdef HAVE_LIBZ
- gzFile zFH;
-#else
- FILE *zFH;
-#endif
- FILE *nFH;
+ cfp *FH;
FILE *tarFH;
FILE *tmpFH;
char *targetFile;
@@ -191,7 +186,7 @@ InitArchiveFmt_Tar(ArchiveHandle *AH)
* Make unbuffered since we will dup() it, and the buffers screw each
* other
*/
- /* setvbuf(ctx->tarFH, NULL, _IONBF, 0); */
+ // setvbuf(ctx->tarFH, NULL, _IONBF, 0);
ctx->hasSeek = checkSeek(ctx->tarFH);
@@ -223,7 +218,7 @@ InitArchiveFmt_Tar(ArchiveHandle *AH)
* Make unbuffered since we will dup() it, and the buffers screw each
* other
*/
- /* setvbuf(ctx->tarFH, NULL, _IONBF, 0); */
+ setvbuf(ctx->tarFH, NULL, _IONBF, 0);
ctx->tarFHpos = 0;
@@ -321,10 +316,6 @@ tarOpen(ArchiveHandle *AH, const char *filename, char mode)
lclContext *ctx = (lclContext *) AH->formatData;
TAR_MEMBER *tm;
-#ifdef HAVE_LIBZ
- char fmode[14];
-#endif
-
if (mode == 'r')
{
tm = _tarPositionTo(AH, filename);
@@ -345,16 +336,10 @@ tarOpen(ArchiveHandle *AH, const char *filename, char mode)
}
}
-#ifdef HAVE_LIBZ
-
if (AH->compression.alg == COMPR_ALG_NONE)
- tm->nFH = ctx->tarFH;
+ tm->FH = cfdopen(dup(fileno(ctx->tarFH)), "rb", &AH->compression);
else
fatal("compression is not supported by tar archive format");
- /* tm->zFH = gzdopen(dup(fileno(ctx->tarFH)), "rb"); */
-#else
- tm->nFH = ctx->tarFH;
-#endif
}
else
{
@@ -406,21 +391,11 @@ tarOpen(ArchiveHandle *AH, const char *filename, char mode)
umask(old_umask);
-#ifdef HAVE_LIBZ
-
- if (AH->compression.alg != COMPR_ALG_NONE)
- {
- sprintf(fmode, "wb%d", AH->compression.level);
- tm->zFH = gzdopen(dup(fileno(tm->tmpFH)), fmode);
- if (tm->zFH == NULL)
- fatal("could not open temporary file");
- }
- else
- tm->nFH = tm->tmpFH;
-#else
-
- tm->nFH = tm->tmpFH;
-#endif
+ tm->FH = cfdopen(dup(fileno(tm->tmpFH)),
+ mode == 'r' ? "r" : "w",
+ &AH->compression);
+ if (tm->FH == NULL)
+ fatal("could not open temporary file");
tm->AH = AH;
tm->targetFile = pg_strdup(filename);
@@ -435,12 +410,14 @@ tarOpen(ArchiveHandle *AH, const char *filename, char mode)
static void
tarClose(ArchiveHandle *AH, TAR_MEMBER *th)
{
+ int res;
+
/*
* Close the GZ file since we dup'd. This will flush the buffers.
*/
- if (AH->compression.alg != COMPR_ALG_NONE)
- if (GZCLOSE(th->zFH) != 0)
- fatal("could not close tar member");
+ res = cfclose(th->FH);
+ if (res != 0)
+ fatal("could not close tar member");
if (th->mode == 'w')
_tarAddFile(AH, th); /* This will close the temp file */
@@ -453,8 +430,7 @@ tarClose(ArchiveHandle *AH, TAR_MEMBER *th)
if (th->targetFile)
free(th->targetFile);
- th->nFH = NULL;
- th->zFH = NULL;
+ th->FH = NULL;
}
#ifdef __NOT_USED__
@@ -540,29 +516,9 @@ _tarReadRaw(ArchiveHandle *AH, void *buf, size_t len, TAR_MEMBER *th, FILE *fh)
}
else if (th)
{
- if (th->zFH)
- {
- res = GZREAD(&((char *) buf)[used], 1, len, th->zFH);
- if (res != len && !GZEOF(th->zFH))
- {
-#ifdef HAVE_LIBZ
- int errnum;
- const char *errmsg = gzerror(th->zFH, &errnum);
-
- fatal("could not read from input file: %s",
- errnum == Z_ERRNO ? strerror(errno) : errmsg);
-#else
- fatal("could not read from input file: %s",
- strerror(errno));
-#endif
- }
- }
- else
- {
- res = fread(&((char *) buf)[used], 1, len, th->nFH);
- if (res != len && !feof(th->nFH))
- READ_ERROR_EXIT(th->nFH);
- }
+ res = cfread(&((char *) buf)[used], len, th->FH);
+ if (res != len && !cfeof(th->FH))
+ fatal("could not read from input file: %m");
}
}
@@ -594,10 +550,7 @@ tarWrite(const void *buf, size_t len, TAR_MEMBER *th)
{
size_t res;
- if (th->zFH != NULL)
- res = GZWRITE(buf, 1, len, th->zFH);
- else
- res = fwrite(buf, 1, len, th->nFH);
+ res = cfwrite(buf, len, th->FH);
th->pos += res;
return res;
--
2.17.0
[text/x-diff] 0006-pg_dump-zstd-compression.patch (30.0K, ../../20210104025321.GA9712@telsasoft.com/7-0006-pg_dump-zstd-compression.patch)
download | inline diff:
From b8eb13b5bd4114a9860cfa83f8240ab09db588b4 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 22 Dec 2020 01:06:26 -0600
Subject: [PATCH 06/20] pg_dump: zstd compression
document any change in search for .gz?
docs
Maybe compress_io should be split so all the library-specific stuff are in
separate files, like compress_{zlib/zstd}.c
---
configure | 123 ++++++-
configure.ac | 22 ++
src/bin/pg_dump/compress_io.c | 480 ++++++++++++++++++++++++++
src/bin/pg_dump/pg_backup.h | 14 +
src/bin/pg_dump/pg_backup_archiver.h | 4 +
src/bin/pg_dump/pg_backup_directory.c | 8 +-
src/bin/pg_dump/pg_dump.c | 39 +++
src/include/pg_config.h.in | 3 +
src/tools/msvc/Solution.pm | 1 +
9 files changed, 686 insertions(+), 8 deletions(-)
diff --git a/configure b/configure
index 11a4284e5b..fe739879af 100755
--- a/configure
+++ b/configure
@@ -698,6 +698,7 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+with_zstd
with_zlib
with_system_tzdata
with_libxslt
@@ -798,6 +799,7 @@ infodir
docdir
oldincludedir
includedir
+runstatedir
localstatedir
sharedstatedir
sysconfdir
@@ -866,6 +868,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_zstd
with_gnu_ld
enable_largefile
'
@@ -935,6 +938,7 @@ datadir='${datarootdir}'
sysconfdir='${prefix}/etc'
sharedstatedir='${prefix}/com'
localstatedir='${prefix}/var'
+runstatedir='${localstatedir}/run'
includedir='${prefix}/include'
oldincludedir='/usr/include'
docdir='${datarootdir}/doc/${PACKAGE_TARNAME}'
@@ -1187,6 +1191,15 @@ do
| -silent | --silent | --silen | --sile | --sil)
silent=yes ;;
+ -runstatedir | --runstatedir | --runstatedi | --runstated \
+ | --runstate | --runstat | --runsta | --runst | --runs \
+ | --run | --ru | --r)
+ ac_prev=runstatedir ;;
+ -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \
+ | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \
+ | --run=* | --ru=* | --r=*)
+ runstatedir=$ac_optarg ;;
+
-sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb)
ac_prev=sbindir ;;
-sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \
@@ -1324,7 +1337,7 @@ fi
for ac_var in exec_prefix prefix bindir sbindir libexecdir datarootdir \
datadir sysconfdir sharedstatedir localstatedir includedir \
oldincludedir docdir infodir htmldir dvidir pdfdir psdir \
- libdir localedir mandir
+ libdir localedir mandir runstatedir
do
eval ac_val=\$$ac_var
# Remove trailing slashes.
@@ -1477,6 +1490,7 @@ Fine tuning of the installation directories:
--sysconfdir=DIR read-only single-machine data [PREFIX/etc]
--sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com]
--localstatedir=DIR modifiable single-machine data [PREFIX/var]
+ --runstatedir=DIR modifiable per-process data [LOCALSTATEDIR/run]
--libdir=DIR object code libraries [EPREFIX/lib]
--includedir=DIR C header files [PREFIX/include]
--oldincludedir=DIR C header files for non-gcc [/usr/include]
@@ -1570,6 +1584,7 @@ Optional Packages:
--with-system-tzdata=DIR
use system time zone data in DIR
--without-zlib do not use Zlib
+ --with-zstd use Zstd compression library
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
Some influential environment variables:
@@ -8601,6 +8616,35 @@ fi
+#
+# Zstd
+#
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+ :
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=no
+
+fi
+
+
+
+
#
# Assignments
#
@@ -12092,6 +12136,59 @@ fi
fi
+if test "$with_zstd" = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compressStream2 in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compressStream2 in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compressStream2+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compressStream2 ();
+int
+main ()
+{
+return ZSTD_compressStream2 ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_compressStream2=yes
+else
+ ac_cv_lib_zstd_ZSTD_compressStream2=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compressStream2" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compressStream2" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compressStream2" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support." "$LINENO" 5
+fi
+
+fi
+
if test "$enable_spinlocks" = yes; then
$as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h
@@ -13295,6 +13392,20 @@ Use --without-zlib to disable zlib support." "$LINENO" 5
fi
+fi
+
+if test "$with_zstd" = yes; then
+ ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+
+else
+ as_fn_error $? "zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support." "$LINENO" 5
+fi
+
+
fi
if test "$with_gssapi" = yes ; then
@@ -14689,7 +14800,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14735,7 +14846,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14759,7 +14870,7 @@ rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14804,7 +14915,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14828,7 +14939,7 @@ rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
diff --git a/configure.ac b/configure.ac
index fc523c6aeb..744836ea7f 100644
--- a/configure.ac
+++ b/configure.ac
@@ -999,6 +999,13 @@ PGAC_ARG_BOOL(with, zlib, yes,
[do not use Zlib])
AC_SUBST(with_zlib)
+#
+# Zstd
+#
+PGAC_ARG_BOOL(with, zstd, no,
+ [use Zstd compression library])
+AC_SUBST(with_zstd)
+
#
# Assignments
#
@@ -1186,6 +1193,14 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_LIB(zstd, ZSTD_compressStream2, [],
+ [AC_MSG_ERROR([zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support.])])
+fi
+
if test "$enable_spinlocks" = yes; then
AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.])
else
@@ -1400,6 +1415,13 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_HEADER(zstd.h, [], [AC_MSG_ERROR([zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support.])])
+fi
+
if test "$with_gssapi" = yes ; then
AC_CHECK_HEADERS(gssapi/gssapi.h, [],
[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index d66d6f60f5..285f554c1a 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -65,6 +65,18 @@ compresslibs[] = {
{ COMPR_ALG_LIBZ, "libz", ".gz", Z_DEFAULT_COMPRESSION },
{ COMPR_ALG_LIBZ, "zlib", ".gz", Z_DEFAULT_COMPRESSION }, /* Alternate name */
+#ifdef HAVE_LIBZSTD
+ /*
+ * ZSTD doesen't have a #define for it, but 0 means "the current default".
+ * Note that ZSTD_CLEVEL_DEFAULT is currently defined to 3.
+ *
+ * Block size should be ZSTD_DStreamOutSize(), but needs to be
+ * constant, so use ZSTD_BLOCKSIZE_MAX (128kB)
+ */
+ { COMPR_ALG_ZSTD, "zst", ".zst", 0 },
+ { COMPR_ALG_ZSTD, "zstd", ".zst", 0 }, /* Alternate name */
+#endif /* HAVE_LIBZSTD */
+
{ 0, NULL, } /* sentinel */
};
@@ -84,6 +96,18 @@ struct CompressorState
char *zlibOut;
size_t zlibOutSize;
#endif
+
+#ifdef HAVE_LIBZSTD
+ union {
+ struct {
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
+ ZSTD_CStream *cstream;
+ } zstd;
+ } u;
+#endif
+
};
/* Routines that support zlib compressed data I/O */
@@ -97,6 +121,15 @@ static void WriteDataToArchiveZlib(ArchiveHandle *AH, CompressorState *cs,
static void EndCompressorZlib(ArchiveHandle *AH, CompressorState *cs);
#endif
+#ifdef HAVE_LIBZSTD
+static ZSTD_CStream *ZstdCStreamParams(Compress *compress);
+static void InitCompressorZstd(CompressorState *cs, Compress *compress);
+static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const char *data, size_t dLen);
+static void ReadDataFromArchiveZstd(ArchiveHandle *AH, ReadFunc readF);
+#endif
+
/* Routines that support uncompressed data I/O */
static void ReadDataFromArchiveNone(ArchiveHandle *AH, ReadFunc readF);
static void WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
@@ -125,6 +158,13 @@ AllocateCompressor(Compress *compression, WriteFunc writeF)
InitCompressorZlib(cs, compression);
break;
#endif
+
+#ifdef HAVE_LIBZSTD
+ case COMPR_ALG_ZSTD:
+ InitCompressorZstd(cs, compression);
+ break;
+#endif
+
case COMPR_ALG_NONE:
/* Do nothing */
break;
@@ -153,6 +193,13 @@ ReadDataFromArchive(ArchiveHandle *AH, ReadFunc readF)
ReadDataFromArchiveZlib(AH, readF);
break;
#endif
+
+#ifdef HAVE_LIBZSTD
+ case COMPR_ALG_ZSTD:
+ ReadDataFromArchiveZstd(AH, readF);
+ break;
+#endif
+
default:
/* Should not happen */
fatal("requested compression not available in this installation");
@@ -173,6 +220,12 @@ WriteDataToArchive(ArchiveHandle *AH, CompressorState *cs,
WriteDataToArchiveZlib(AH, cs, data, dLen);
break;
#endif
+
+#ifdef HAVE_LIBZSTD
+ case COMPR_ALG_ZSTD:
+ WriteDataToArchiveZstd(AH, cs, data, dLen);
+ break;
+#endif
case COMPR_ALG_NONE:
WriteDataToArchiveNone(AH, cs, data, dLen);
break;
@@ -193,11 +246,202 @@ EndCompressor(ArchiveHandle *AH, CompressorState *cs)
if (cs->comprAlg == COMPR_ALG_LIBZ)
EndCompressorZlib(AH, cs);
#endif
+
+#ifdef HAVE_LIBZSTD
+ if (cs->comprAlg == COMPR_ALG_ZSTD)
+ EndCompressorZstd(AH, cs);
+#endif
+
free(cs);
}
/* Private routines, specific to each compression method. */
+#ifdef HAVE_LIBZSTD
+
+static void ZSTD_CCtx_setParam_or_die(ZSTD_CStream *cstream,
+ ZSTD_cParameter param, int value)
+
+{
+ size_t res;
+ res = ZSTD_CCtx_setParameter(cstream, param, value);
+ if (ZSTD_isError(res))
+ fatal("could not set compression parameter: %s",
+ ZSTD_getErrorName(res));
+}
+
+/* Return a compression stream with parameters set per argument */
+static ZSTD_CStream*
+ZstdCStreamParams(Compress *compress)
+{
+ ZSTD_CStream *cstream;
+ cstream = ZSTD_createCStream();
+ if (cstream == NULL)
+ fatal("could not initialize compression library");
+
+ if (compress->level != 0) // XXX: ZSTD_CLEVEL_DEFAULT
+ {
+ size_t res;
+ res = ZSTD_CCtx_setParameter(cstream,
+ ZSTD_c_compressionLevel, compress->level);
+ if (ZSTD_isError(res))
+ fatal("could not set compression level: %s",
+ ZSTD_getErrorName(res));
+ }
+
+ if (compress->zstd.longdistance) // XXX: ternary
+ ZSTD_CCtx_setParam_or_die(cstream,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->zstd.longdistance);
+
+ if (compress->zstd.checksum)
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_checksumFlag,
+ compress->zstd.checksum);
+
+// not supported in my library ?
+ if (compress->zstd.threads)
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
+ compress->zstd.threads);
+
+#if 0
+ /* Still marked as experimental */
+ if (compress->zstd.rsyncable)
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_rsyncable, 1);
+#endif
+
+ return cstream;
+}
+
+static void
+InitCompressorZstd(CompressorState *cs, Compress *compress)
+{
+ cs->u.zstd.cstream = ZstdCStreamParams(compress);
+ /* XXX: initialize safely like the corresponding zlib "paranoia" */
+ cs->u.zstd.output.size = ZSTD_CStreamOutSize();
+ cs->u.zstd.output.dst = pg_malloc(cs->u.zstd.output.size);
+ cs->u.zstd.output.pos = 0;
+}
+
+static void
+EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZSTD_outBuffer *output = &cs->u.zstd.output;
+
+ for (;;)
+ {
+ size_t res;
+
+ res = ZSTD_compressStream2(cs->u.zstd.cstream, output,
+ &cs->u.zstd.input, ZSTD_e_end);
+
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ if (res == 0)
+ break;
+
+ if (ZSTD_isError(res))
+ fatal("could not close compression stream: %s",
+ ZSTD_getErrorName(res));
+ }
+
+ // XXX: retval
+ ZSTD_freeCStream(cs->u.zstd.cstream);
+}
+
+static void
+WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const char *data, size_t dLen)
+{
+ ZSTD_inBuffer *input = &cs->u.zstd.input;
+ ZSTD_outBuffer *output = &cs->u.zstd.output;
+
+ input->src = (void *) unconstify(char *, data);
+ input->size = dLen;
+ input->pos = 0;
+
+ while (input->pos != input->size)
+ {
+ size_t res;
+
+ res = ZSTD_compressStream2(cs->u.zstd.cstream, output,
+ input, ZSTD_e_continue);
+
+ if (output->pos == output->size ||
+ input->pos != input->size)
+ {
+ /*
+ * Extra paranoia: avoid zero-length chunks, since a zero length
+ * chunk is the EOF marker in the custom format. This should never
+ * happen but...
+ */
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ output->pos = 0;
+ }
+
+ if (ZSTD_isError(res))
+ fatal("could not compress data: %s", ZSTD_getErrorName(res));
+ }
+}
+
+/* Read data from a compressed zstd archive */
+static void
+ReadDataFromArchiveZstd(ArchiveHandle *AH, ReadFunc readF)
+{
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ size_t res;
+ size_t input_size;
+
+ dstream = ZSTD_createDStream();
+ if (dstream == NULL)
+ fatal("could not initialize compression library");
+
+ input_size = ZSTD_DStreamInSize();
+ input.src = pg_malloc(input_size);
+
+ output.size = ZSTD_DStreamOutSize();
+ output.dst = pg_malloc(output.size);
+
+ /* read compressed data */
+ for (;;)
+ {
+ size_t cnt;
+
+ input.size = input_size; // XXX: the buffer can grow, we shouldn't keep resetting it to the original value..
+ cnt = readF(AH, (char **)unconstify(void **, &input.src), &input.size);
+ input.pos = 0;
+ input.size = cnt;
+
+ if (cnt == 0)
+ break;
+
+ while (input.pos < input.size)
+ {
+ /* decompress */
+ output.pos = 0;
+ res = ZSTD_decompressStream(dstream, &output, &input);
+
+ if (ZSTD_isError(res))
+ fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ /* write to output handle */
+ ((char *)output.dst)[output.pos] = '\0';
+ ahwrite(output.dst, 1, output.pos, AH);
+ // if (res == 0)
+ // break;
+ }
+ }
+
+ pg_free(unconstify(void *, input.src));
+ pg_free(output.dst);
+}
+
+#endif /* HAVE_LIBZSTD */
+
#ifdef HAVE_LIBZ
/*
* Functions for zlib compressed output.
@@ -411,6 +655,19 @@ struct cfp
#ifdef HAVE_LIBZ
gzFile compressedfp;
#endif
+
+#ifdef HAVE_LIBZSTD // XXX: this should be a union with a CompressionAlgorithm alg?
+ /* This is a normal file to which we read/write compressed data */
+ struct {
+ FILE *fp;
+ // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ } zstd;
+#endif
+
};
static int hasSuffix(const char *filename);
@@ -525,6 +782,31 @@ cfopen(const char *path, const char *mode, Compress *compression)
return fp;
#endif
+#ifdef HAVE_LIBZSTD
+ case COMPR_ALG_ZSTD:
+ fp->zstd.fp = fopen(path, mode);
+ if (fp->zstd.fp == NULL)
+ {
+ free_keep_errno(fp);
+ fp = NULL;
+ }
+ else if (mode[0] == 'w' || mode[0] == 'a' ||
+ strchr(mode, '+') != NULL)
+ {
+ fp->zstd.output.size = ZSTD_CStreamOutSize();
+ fp->zstd.output.dst = pg_malloc0(fp->zstd.output.size);
+ fp->zstd.cstream = ZstdCStreamParams(compression);
+ }
+ else if (strchr(mode, 'r'))
+ {
+ fp->zstd.input.src = pg_malloc0(ZSTD_DStreamInSize());
+ fp->zstd.dstream = ZSTD_createDStream();
+ if (fp->zstd.dstream == NULL)
+ fatal("could not initialize compression library");
+ } // XXX else: bad mode
+ return fp;
+#endif
+
case COMPR_ALG_NONE:
fp->uncompressedfp = fopen(path, mode);
if (fp->uncompressedfp == NULL)
@@ -576,6 +858,31 @@ cfdopen(int fd, const char *mode, Compress *compression)
return fp;
#endif
+#ifdef HAVE_LIBZSTD
+ case COMPR_ALG_ZSTD:
+ fp->zstd.fp = fdopen(fd, mode);
+ if (fp->zstd.fp == NULL)
+ {
+ free_keep_errno(fp);
+ fp = NULL;
+ }
+ else if (mode[0] == 'w' || mode[0] == 'a' ||
+ strchr(mode, '+') != NULL)
+ {
+ fp->zstd.output.size = ZSTD_CStreamOutSize();
+ fp->zstd.output.dst = pg_malloc0(fp->zstd.output.size);
+ fp->zstd.cstream = ZstdCStreamParams(compression);
+ }
+ else if (strchr(mode, 'r'))
+ {
+ fp->zstd.input.src = pg_malloc0(ZSTD_DStreamInSize());
+ fp->zstd.dstream = ZSTD_createDStream();
+ if (fp->zstd.dstream == NULL)
+ fatal("could not initialize compression library");
+ } // XXX else: bad mode
+ return fp;
+#endif
+
case COMPR_ALG_NONE:
fp->uncompressedfp = fdopen(fd, mode);
if (fp->uncompressedfp == NULL)
@@ -617,6 +924,68 @@ cfread(void *ptr, int size, cfp *fp)
}
#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ ZSTD_outBuffer *output = &fp->zstd.output;
+ ZSTD_inBuffer *input = &fp->zstd.input;
+ size_t input_size = ZSTD_DStreamInSize();
+ /* input_size is the allocated size */
+ size_t res, cnt;
+
+ output->size = size;
+ output->dst = ptr;
+ output->pos = 0;
+
+ for (;;)
+ {
+ Assert(input->pos <= input->size);
+ Assert(input->size <= input_size);
+
+ /* If the input is completely consumed, start back at the beginning */
+ if (input->pos == input->size)
+ {
+ /* input->size is size produced by "fread" */
+ input->size = 0;
+ /* input->pos is position consumed by decompress */
+ input->pos = 0;
+ }
+
+ /* read compressed data if we must produce more input */
+ if (input->pos == input->size)
+ {
+ cnt = fread(unconstify(void *, input->src), 1, input_size, fp->zstd.fp);
+ input->size = cnt;
+
+ /* If we have no input to consume, we're done */
+ if (cnt == 0)
+ break;
+ }
+
+ Assert(cnt >= 0);
+ Assert(input->size <= input_size);
+
+ /* Now consume as much as possible */
+ for ( ; input->pos < input->size; )
+ {
+ /* decompress */
+ res = ZSTD_decompressStream(fp->zstd.dstream, output, input);
+ if (res == 0)
+ break; /* End of frame */
+ if (output->pos == output->size)
+ break; /* No more room for output */
+ if (ZSTD_isError(res))
+ fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+ }
+
+ if (output->pos == output->size)
+ break; /* We read all the data that fits */
+ }
+
+ return output->pos;
+ }
+#endif
+
ret = fread(ptr, 1, size, fp->uncompressedfp);
if (ret != size && !feof(fp->uncompressedfp))
READ_ERROR_EXIT(fp->uncompressedfp);
@@ -630,6 +999,35 @@ cfwrite(const void *ptr, int size, cfp *fp)
if (fp->compressedfp)
return gzwrite(fp->compressedfp, ptr, size);
#endif
+
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ size_t res, cnt;
+ ZSTD_outBuffer *output = &fp->zstd.output;
+ ZSTD_inBuffer *input = &fp->zstd.input;
+
+ input->src = ptr;
+ input->size = size;
+ input->pos = 0;
+
+ /* Consume all input, and flush later */
+ while (input->pos != input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_continue);
+ if (ZSTD_isError(res))
+ fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ if (cnt != output->pos)
+ fatal("could not write data: %s", strerror(errno));
+ }
+
+ return size;
+ }
+#endif
+
return fwrite(ptr, 1, size, fp->uncompressedfp);
}
@@ -652,6 +1050,21 @@ cfgetc(cfp *fp)
return ret;
}
#endif
+
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ if (cfread(&ret, 1, fp) != 1)
+ {
+ if (feof(fp->zstd.fp))
+ fatal("could not read from input file: end of file");
+ else
+ fatal("could not read from input file: %s", strerror(errno));
+ }
+ return ret;
+ }
+#endif
+
ret = fgetc(fp->uncompressedfp);
if (ret == EOF)
READ_ERROR_EXIT(fp->uncompressedfp);
@@ -665,6 +1078,31 @@ cfgets(cfp *fp, char *buf, int len)
if (fp->compressedfp)
return gzgets(fp->compressedfp, buf, len);
#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ /*
+ * Read one byte at a time until newline or EOF.
+ * This is only used to read the list of blobs, and the I/O is
+ * buffered anyway.
+ */
+ int i, res;
+ for (i = 0; i < len - 1; ++i)
+ {
+ res = cfread(&buf[i], 1, fp);
+ if (res != 1)
+ break;
+ if (buf[i] == '\n')
+ {
+ ++i;
+ break;
+ }
+ }
+ buf[i] = '\0';
+ return i > 0 ? buf : 0;
+ }
+#endif
+
return fgets(buf, len, fp->uncompressedfp);
}
@@ -688,6 +1126,44 @@ cfclose(cfp *fp)
}
#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ {
+ ZSTD_outBuffer *output = &fp->zstd.output;
+ ZSTD_inBuffer *input = &fp->zstd.input;
+ size_t res, cnt;
+
+ if (fp->zstd.cstream)
+ {
+ for (;;)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_end);
+ if (ZSTD_isError(res))
+ fatal("could not compress data: %s", ZSTD_getErrorName(res));
+ cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ if (cnt != output->pos)
+ fatal("could not write data: %s", strerror(errno));
+ if (res == 0)
+ break;
+ }
+
+ ZSTD_freeCStream(fp->zstd.cstream);
+ pg_free(fp->zstd.output.dst);
+ }
+
+ if (fp->zstd.dstream)
+ {
+ ZSTD_freeDStream(fp->zstd.dstream);
+ pg_free(unconstify(void *, fp->zstd.input.src));
+ }
+
+ result = fclose(fp->zstd.fp);
+ fp->zstd.fp = NULL;
+ return result;
+ }
+#endif
+
result = fclose(fp->uncompressedfp);
fp->uncompressedfp = NULL;
free_keep_errno(fp);
@@ -702,6 +1178,10 @@ cfeof(cfp *fp)
return gzeof(fp->compressedfp);
#endif
+#ifdef HAVE_LIBZSTD
+ if (fp->zstd.fp)
+ return feof(fp->zstd.fp);
+#endif
return feof(fp->uncompressedfp);
}
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index f2390b7937..19ff6248d5 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -77,6 +77,7 @@ typedef enum
COMPR_ALG_DEFAULT = -1,
COMPR_ALG_NONE,
COMPR_ALG_LIBZ,
+ COMPR_ALG_ZSTD,
} CompressionAlgorithm;
/* Should be called "method" or "library" ? */
@@ -88,6 +89,19 @@ typedef struct Compress {
* are all integer, though.
*/
bool level_set;
+
+ /*
+ * This could be a union across all compress algorithms, but
+ * keeping as separate structs allows checking that options are
+ * not specified for a different algorithm than selected.
+ */
+
+ struct {
+ bool longdistance;
+ bool checksum;
+ bool rsyncable;
+ int threads;
+ } zstd;
} Compress;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 9f511b49b9..da2eb53277 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -56,6 +56,10 @@ typedef struct _z_stream
typedef z_stream *z_streamp;
#endif
+#ifdef HAVE_LIBZSTD
+#include <zstd.h>
+#endif /* HAVE_LIBZSTD */
+
/* Data block types */
#define BLK_DATA 1
#define BLK_BLOBS 3
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 75c1bf22e4..b8efeb8ca7 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -393,8 +393,12 @@ _PrintFileData(ArchiveHandle *AH, char *filename)
if (!cfp)
fatal("could not open input file \"%s\": %m", filename);
- buf = pg_malloc(ZLIB_OUT_SIZE);
- buflen = ZLIB_OUT_SIZE;
+ /*
+ * zstd prefers a 128kB buffer. The allocation cannot happen in
+ * cfread, since the "cfp" is an opaque type.
+ */
+ buf = pg_malloc(128*1024);
+ buflen = 128*1024;
while ((cnt = cfread(buf, buflen, cfp)))
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 75985fd4d3..7c2f7a9ca3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -356,6 +356,12 @@ parse_compression(const char *optarg, Compress *compress)
compress->level = atoi(1+eq);
compress->level_set = true;
}
+ else if (strncmp(optarg, "zstdlong", len) == 0)
+ compress->zstd.longdistance = atoi(1+eq);
+ else if (strncmp(optarg, "checksum", len) == 0)
+ compress->zstd.checksum = atoi(1+eq);
+ else if (strncmp(optarg, "threads", len) == 0)
+ compress->zstd.threads = atoi(1+eq);
else
{
pg_log_error("unknown compression setting: %s", optarg);
@@ -367,11 +373,31 @@ parse_compression(const char *optarg, Compress *compress)
break;
}
+ /* XXX: zstd will check its own compression level later */
+ if (compress->alg != COMPR_ALG_ZSTD)
+ {
+ Compress nullopts = {0};
+
+ if (compress->level < 0 || compress->level > 9)
+ {
+ pg_log_error("compression level must be in range 0..9");
+ exit_nicely(1);
+ }
+
+// XXX: needs to set default alg first
+ if (memcmp(&compress->zstd, &nullopts.zstd, sizeof(nullopts.zstd)) != 0)
+ {
+ pg_log_error("compression option not supported with this algorithm");
+ exit_nicely(1);
+ }
+ }
+
if (!compress->level_set)
{ // XXX
const int default_compress_level[] = {
0, /* COMPR_ALG_NONE */
Z_DEFAULT_COMPRESSION, /* COMPR_ALG_ZLIB */
+ 0, // XXX: ZSTD_CLEVEL_DEFAULT, /* COMPR_ALG_ZSTD */
};
compress->level = default_compress_level[compress->alg];
@@ -764,6 +790,11 @@ main(int argc, char **argv)
compress.alg = COMPR_ALG_LIBZ;
compress.level = Z_DEFAULT_COMPRESSION;
#endif
+
+#ifdef HAVE_LIBZSTD
+ compress.alg = COMPR_ALG_ZSTD; // Set default for testing purposes
+ compress.level = ZSTD_CLEVEL_DEFAULT;
+#endif
}
else
{
@@ -780,6 +811,14 @@ main(int argc, char **argv)
}
#endif
+#ifndef HAVE_LIBZSTD
+ if (compress.alg == COMPR_ALG_ZSTD)
+ {
+ pg_log_warning("requested compression not available in this installation -- archive will be uncompressed");
+ compress.alg = 0;
+ }
+#endif
+
/*
* If emitting an archive format, we always want to emit a DATABASE item,
* in case --create is specified at pg_restore time.
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index de8f838e53..da35415c72 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -346,6 +346,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if you have the `link' function. */
#undef HAVE_LINK
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index 22d6abd367..a101366b4c 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -307,6 +307,7 @@ sub GenerateFiles
HAVE_LIBXML2 => undef,
HAVE_LIBXSLT => undef,
HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef,
+ HAVE_LIBZSTD => $self->{options}->{zstd} ? 1 : undef,
HAVE_LINK => undef,
HAVE_LOCALE_T => 1,
HAVE_LONG_INT_64 => undef,
--
2.17.0
[text/x-diff] 0007-fix-comments.patch (3.5K, ../../20210104025321.GA9712@telsasoft.com/8-0007-fix-comments.patch)
download | inline diff:
From fe12ba8f0ec663f4778e6b46053739ad4ec20514 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Thu, 24 Dec 2020 00:01:43 -0600
Subject: [PATCH 07/20] fix comments
---
src/bin/pg_dump/compress_io.c | 15 +++++++--------
src/bin/pg_dump/pg_backup_directory.c | 4 ++--
2 files changed, 9 insertions(+), 10 deletions(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 285f554c1a..fa94148cdf 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -13,7 +13,7 @@
* friends, providing an interface similar to those, but abstracts away
* the possible compression. Both APIs use libz for the compression, but
* the second API uses gzip headers, so the resulting files can be easily
- * manipulated with the gzip utility.
+ * manipulated with the gzip utility. XXX
*
* Compressor API
* --------------
@@ -41,9 +41,9 @@
* libz's gzopen() APIs. It allows you to use the same functions for
* compressed and uncompressed streams. cfopen_read() first tries to open
* the file with given name, and if it fails, it tries to open the same
- * file with the .gz suffix. cfopen_write() opens a file for writing, an
+ * file with a compressed suffix. cfopen_write() opens a file for writing, an
* extra argument specifies if the file should be compressed, and adds the
- * .gz suffix to the filename if so. This allows you to easily handle both
+ * compressed suffix to the filename if so. This allows you to easily handle both
* compressed and uncompressed files.
*
* IDENTIFICATION
@@ -646,8 +646,8 @@ WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
*/
/*
- * cfp represents an open stream, wrapping the underlying FILE or gzFile
- * pointer. This is opaque to the callers.
+ * cfp represents an open stream, wrapping the underlying compressed or
+ * uncompressed file object. This is opaque to the callers.
*/
struct cfp
{
@@ -687,8 +687,7 @@ free_keep_errno(void *p)
* be either "r" or "rb".
*
* If the file at 'path' does not exist, we search with compressed suffix (if 'path'
- * doesn't already have one) and try again. So if you pass "foo" as 'path',
- * this will open either "foo" or "foo.gz".
+ * doesn't already have one) and try again.
*
* On failure, return NULL with an error code in errno.
*/
@@ -745,7 +744,7 @@ cfopen_write(const char *path, const char *mode, Compress *compression)
}
/*
- * Opens file 'path' in 'mode'. If 'compression' is non-zero, the file
+ * Opens file 'path' in 'mode'. If 'alg' is COMPR_ALG_ZLIB, the file
* is opened with libz gzopen(), otherwise with plain fopen().
*
* On failure, return NULL with an error code in errno.
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index b8efeb8ca7..f0ded2421d 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -6,8 +6,8 @@
* for the TOC, and a separate file for each data entry, named "<oid>.dat".
* Large objects (BLOBs) are stored in separate files named "blob_<oid>.dat",
* and there's a plain-text TOC file for them called "blobs.toc". If
- * compression is used, each data file is individually compressed and the
- * ".gz" suffix is added to the filenames. The TOC files are never
+ * compression is used, each data file is individually compressed with a
+ * suffix is added to the filenames. The TOC files are never
* compressed by pg_dump, however they are accepted with the .gz suffix too,
* in case the user has manually compressed them with 'gzip'.
*
--
2.17.0
[text/x-diff] 0008-union-with-a-CompressionAlgorithm-alg.patch (13.9K, ../../20210104025321.GA9712@telsasoft.com/9-0008-union-with-a-CompressionAlgorithm-alg.patch)
download | inline diff:
From 44b3ed951859072b8d814d0439565187bf960b7b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 21 Dec 2020 00:11:43 -0600
Subject: [PATCH 08/20] union{} with a CompressionAlgorithm alg
---
src/bin/pg_dump/compress_io.c | 200 ++++++++++++++++++----------------
src/bin/pg_dump/pg_dump.c | 2 +-
2 files changed, 106 insertions(+), 96 deletions(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index fa94148cdf..e07436bc21 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -651,23 +651,27 @@ WriteDataToArchiveNone(ArchiveHandle *AH, CompressorState *cs,
*/
struct cfp
{
- FILE *uncompressedfp;
+ CompressionAlgorithm alg;
+
+ union {
+ FILE *fp;
+
#ifdef HAVE_LIBZ
- gzFile compressedfp;
+ gzFile gzfp;
#endif
-#ifdef HAVE_LIBZSTD // XXX: this should be a union with a CompressionAlgorithm alg?
- /* This is a normal file to which we read/write compressed data */
- struct {
- FILE *fp;
- // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
- ZSTD_CStream *cstream;
- ZSTD_DStream *dstream;
- ZSTD_outBuffer output;
- ZSTD_inBuffer input;
- } zstd;
+#ifdef HAVE_LIBZSTD
+ struct {
+ /* This is a normal file to which we read/write compressed data */
+ FILE *fp;
+ // XXX: use one separate ZSTD_CStream per thread: disable on windows ?
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ } zstd;
#endif
-
+ } u;
};
static int hasSuffix(const char *filename);
@@ -754,6 +758,8 @@ cfopen(const char *path, const char *mode, Compress *compression)
{
cfp *fp = pg_malloc0(sizeof(cfp));
+ fp->alg = compression->alg;
+
switch (compression->alg)
{
#ifdef HAVE_LIBZ
@@ -765,15 +771,15 @@ cfopen(const char *path, const char *mode, Compress *compression)
snprintf(mode_compression, sizeof(mode_compression), "%s%d",
mode, compression->level);
- fp->compressedfp = gzopen(path, mode_compression);
+ fp->u.gzfp = gzopen(path, mode_compression);
}
else
{
/* don't specify a level, just use the zlib default */
- fp->compressedfp = gzopen(path, mode);
+ fp->u.gzfp = gzopen(path, mode);
}
- if (fp->compressedfp == NULL)
+ if (fp->u.gzfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
@@ -783,8 +789,8 @@ cfopen(const char *path, const char *mode, Compress *compression)
#ifdef HAVE_LIBZSTD
case COMPR_ALG_ZSTD:
- fp->zstd.fp = fopen(path, mode);
- if (fp->zstd.fp == NULL)
+ fp->u.zstd.fp = fopen(path, mode);
+ if (fp->u.zstd.fp == NULL)
{
free_keep_errno(fp);
fp = NULL;
@@ -792,23 +798,23 @@ cfopen(const char *path, const char *mode, Compress *compression)
else if (mode[0] == 'w' || mode[0] == 'a' ||
strchr(mode, '+') != NULL)
{
- fp->zstd.output.size = ZSTD_CStreamOutSize();
- fp->zstd.output.dst = pg_malloc0(fp->zstd.output.size);
- fp->zstd.cstream = ZstdCStreamParams(compression);
+ fp->u.zstd.output.size = ZSTD_CStreamOutSize();
+ fp->u.zstd.output.dst = pg_malloc0(fp->u.zstd.output.size);
+ fp->u.zstd.cstream = ZstdCStreamParams(compression);
}
else if (strchr(mode, 'r'))
{
- fp->zstd.input.src = pg_malloc0(ZSTD_DStreamInSize());
- fp->zstd.dstream = ZSTD_createDStream();
- if (fp->zstd.dstream == NULL)
+ fp->u.zstd.input.src = pg_malloc0(ZSTD_DStreamInSize());
+ fp->u.zstd.dstream = ZSTD_createDStream();
+ if (fp->u.zstd.dstream == NULL)
fatal("could not initialize compression library");
} // XXX else: bad mode
return fp;
#endif
case COMPR_ALG_NONE:
- fp->uncompressedfp = fopen(path, mode);
- if (fp->uncompressedfp == NULL)
+ fp->u.fp = fopen(path, mode);
+ if (fp->u.fp == NULL)
{
free_keep_errno(fp);
fp = NULL;
@@ -830,6 +836,8 @@ cfdopen(int fd, const char *mode, Compress *compression)
{
cfp *fp = pg_malloc0(sizeof(cfp));
+ fp->alg = compression->alg;
+
switch (compression->alg)
{
#ifdef HAVE_LIBZ
@@ -841,15 +849,15 @@ cfdopen(int fd, const char *mode, Compress *compression)
snprintf(mode_compression, sizeof(mode_compression), "%s%d",
mode, compression->level);
- fp->compressedfp = gzdopen(fd, mode_compression);
+ fp->u.gzfp = gzdopen(fd, mode_compression);
}
else
{
/* don't specify a level, just use the zlib default */
- fp->compressedfp = gzdopen(fd, mode);
+ fp->u.gzfp = gzdopen(fd, mode);
}
- if (fp->compressedfp == NULL)
+ if (fp->u.gzfp == NULL)
{
free_keep_errno(fp);
fp = NULL;
@@ -859,8 +867,8 @@ cfdopen(int fd, const char *mode, Compress *compression)
#ifdef HAVE_LIBZSTD
case COMPR_ALG_ZSTD:
- fp->zstd.fp = fdopen(fd, mode);
- if (fp->zstd.fp == NULL)
+ fp->u.zstd.fp = fdopen(fd, mode);
+ if (fp->u.zstd.fp == NULL)
{
free_keep_errno(fp);
fp = NULL;
@@ -868,23 +876,23 @@ cfdopen(int fd, const char *mode, Compress *compression)
else if (mode[0] == 'w' || mode[0] == 'a' ||
strchr(mode, '+') != NULL)
{
- fp->zstd.output.size = ZSTD_CStreamOutSize();
- fp->zstd.output.dst = pg_malloc0(fp->zstd.output.size);
- fp->zstd.cstream = ZstdCStreamParams(compression);
+ fp->u.zstd.output.size = ZSTD_CStreamOutSize();
+ fp->u.zstd.output.dst = pg_malloc0(fp->u.zstd.output.size);
+ fp->u.zstd.cstream = ZstdCStreamParams(compression);
}
else if (strchr(mode, 'r'))
{
- fp->zstd.input.src = pg_malloc0(ZSTD_DStreamInSize());
- fp->zstd.dstream = ZSTD_createDStream();
- if (fp->zstd.dstream == NULL)
+ fp->u.zstd.input.src = pg_malloc0(ZSTD_DStreamInSize());
+ fp->u.zstd.dstream = ZSTD_createDStream();
+ if (fp->u.zstd.dstream == NULL)
fatal("could not initialize compression library");
} // XXX else: bad mode
return fp;
#endif
case COMPR_ALG_NONE:
- fp->uncompressedfp = fdopen(fd, mode);
- if (fp->uncompressedfp == NULL)
+ fp->u.fp = fdopen(fd, mode);
+ if (fp->u.fp == NULL)
{
free_keep_errno(fp);
fp = NULL;
@@ -908,13 +916,13 @@ cfread(void *ptr, int size, cfp *fp)
return 0;
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
- ret = gzread(fp->compressedfp, ptr, size);
- if (ret != size && !gzeof(fp->compressedfp))
+ ret = gzread(fp->u.gzfp, ptr, size);
+ if (ret != size && !gzeof(fp->u.gzfp))
{
int errnum;
- const char *errmsg = gzerror(fp->compressedfp, &errnum);
+ const char *errmsg = gzerror(fp->u.gzfp, &errnum);
fatal("could not read from input file: %s",
errnum == Z_ERRNO ? strerror(errno) : errmsg);
@@ -924,10 +932,10 @@ cfread(void *ptr, int size, cfp *fp)
#endif
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
- ZSTD_outBuffer *output = &fp->zstd.output;
- ZSTD_inBuffer *input = &fp->zstd.input;
+ ZSTD_outBuffer *output = &fp->u.zstd.output;
+ ZSTD_inBuffer *input = &fp->u.zstd.input;
size_t input_size = ZSTD_DStreamInSize();
/* input_size is the allocated size */
size_t res, cnt;
@@ -953,7 +961,7 @@ cfread(void *ptr, int size, cfp *fp)
/* read compressed data if we must produce more input */
if (input->pos == input->size)
{
- cnt = fread(unconstify(void *, input->src), 1, input_size, fp->zstd.fp);
+ cnt = fread(unconstify(void *, input->src), 1, input_size, fp->u.zstd.fp);
input->size = cnt;
/* If we have no input to consume, we're done */
@@ -968,7 +976,7 @@ cfread(void *ptr, int size, cfp *fp)
for ( ; input->pos < input->size; )
{
/* decompress */
- res = ZSTD_decompressStream(fp->zstd.dstream, output, input);
+ res = ZSTD_decompressStream(fp->u.zstd.dstream, output, input);
if (res == 0)
break; /* End of frame */
if (output->pos == output->size)
@@ -985,9 +993,9 @@ cfread(void *ptr, int size, cfp *fp)
}
#endif
- ret = fread(ptr, 1, size, fp->uncompressedfp);
- if (ret != size && !feof(fp->uncompressedfp))
- READ_ERROR_EXIT(fp->uncompressedfp);
+ ret = fread(ptr, 1, size, fp->u.fp);
+ if (ret != size && !feof(fp->u.fp))
+ READ_ERROR_EXIT(fp->u.fp);
return ret;
}
@@ -995,16 +1003,16 @@ int
cfwrite(const void *ptr, int size, cfp *fp)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
- return gzwrite(fp->compressedfp, ptr, size);
+ if (fp->alg == COMPR_ALG_LIBZ)
+ return gzwrite(fp->u.gzfp, ptr, size);
#endif
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
size_t res, cnt;
- ZSTD_outBuffer *output = &fp->zstd.output;
- ZSTD_inBuffer *input = &fp->zstd.input;
+ ZSTD_outBuffer *output = &fp->u.zstd.output;
+ ZSTD_inBuffer *input = &fp->u.zstd.input;
input->src = ptr;
input->size = size;
@@ -1014,11 +1022,11 @@ cfwrite(const void *ptr, int size, cfp *fp)
while (input->pos != input->size)
{
output->pos = 0;
- res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_continue);
+ res = ZSTD_compressStream2(fp->u.zstd.cstream, output, input, ZSTD_e_continue);
if (ZSTD_isError(res))
fatal("could not compress data: %s", ZSTD_getErrorName(res));
- cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ cnt = fwrite(output->dst, 1, output->pos, fp->u.zstd.fp);
if (cnt != output->pos)
fatal("could not write data: %s", strerror(errno));
}
@@ -1027,7 +1035,7 @@ cfwrite(const void *ptr, int size, cfp *fp)
}
#endif
- return fwrite(ptr, 1, size, fp->uncompressedfp);
+ return fwrite(ptr, 1, size, fp->u.fp);
}
int
@@ -1036,12 +1044,12 @@ cfgetc(cfp *fp)
int ret;
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
- ret = gzgetc(fp->compressedfp);
+ ret = gzgetc(fp->u.gzfp);
if (ret == EOF)
{
- if (!gzeof(fp->compressedfp))
+ if (!gzeof(fp->u.gzfp))
fatal("could not read from input file: %s", strerror(errno));
else
fatal("could not read from input file: end of file");
@@ -1051,11 +1059,11 @@ cfgetc(cfp *fp)
#endif
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
if (cfread(&ret, 1, fp) != 1)
{
- if (feof(fp->zstd.fp))
+ if (feof(fp->u.zstd.fp))
fatal("could not read from input file: end of file");
else
fatal("could not read from input file: %s", strerror(errno));
@@ -1064,9 +1072,9 @@ cfgetc(cfp *fp)
}
#endif
- ret = fgetc(fp->uncompressedfp);
+ ret = fgetc(fp->u.fp);
if (ret == EOF)
- READ_ERROR_EXIT(fp->uncompressedfp);
+ READ_ERROR_EXIT(fp->u.fp);
return ret;
}
@@ -1074,11 +1082,12 @@ char *
cfgets(cfp *fp, char *buf, int len)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
- return gzgets(fp->compressedfp, buf, len);
+ if (fp->alg == COMPR_ALG_LIBZ)
+ return gzgets(fp->u.gzfp, buf, len);
#endif
+
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
/*
* Read one byte at a time until newline or EOF.
@@ -1102,7 +1111,7 @@ cfgets(cfp *fp, char *buf, int len)
}
#endif
- return fgets(buf, len, fp->uncompressedfp);
+ return fgets(buf, len, fp->u.fp);
}
/* Close the given compressed or uncompressed stream; return 0 on success. */
@@ -1117,54 +1126,54 @@ cfclose(cfp *fp)
return EOF;
}
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
- result = gzclose(fp->compressedfp);
- fp->compressedfp = NULL;
+ result = gzclose(fp->u.gzfp);
+ fp->u.gzfp = NULL;
return result;
}
#endif
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
+ if (fp->alg == COMPR_ALG_ZSTD)
{
- ZSTD_outBuffer *output = &fp->zstd.output;
- ZSTD_inBuffer *input = &fp->zstd.input;
+ ZSTD_outBuffer *output = &fp->u.zstd.output;
+ ZSTD_inBuffer *input = &fp->u.zstd.input;
size_t res, cnt;
- if (fp->zstd.cstream)
+ if (fp->u.zstd.cstream)
{
for (;;)
{
output->pos = 0;
- res = ZSTD_compressStream2(fp->zstd.cstream, output, input, ZSTD_e_end);
+ res = ZSTD_compressStream2(fp->u.zstd.cstream, output, input, ZSTD_e_end);
if (ZSTD_isError(res))
fatal("could not compress data: %s", ZSTD_getErrorName(res));
- cnt = fwrite(output->dst, 1, output->pos, fp->zstd.fp);
+ cnt = fwrite(output->dst, 1, output->pos, fp->u.zstd.fp);
if (cnt != output->pos)
fatal("could not write data: %s", strerror(errno));
if (res == 0)
break;
}
- ZSTD_freeCStream(fp->zstd.cstream);
- pg_free(fp->zstd.output.dst);
+ ZSTD_freeCStream(fp->u.zstd.cstream);
+ pg_free(fp->u.zstd.output.dst);
}
- if (fp->zstd.dstream)
+ if (fp->u.zstd.dstream)
{
- ZSTD_freeDStream(fp->zstd.dstream);
- pg_free(unconstify(void *, fp->zstd.input.src));
+ ZSTD_freeDStream(fp->u.zstd.dstream);
+ pg_free(unconstify(void *, fp->u.zstd.input.src));
}
- result = fclose(fp->zstd.fp);
- fp->zstd.fp = NULL;
+ result = fclose(fp->u.zstd.fp);
+ fp->u.zstd.fp = NULL;
return result;
}
#endif
- result = fclose(fp->uncompressedfp);
- fp->uncompressedfp = NULL;
+ result = fclose(fp->u.fp);
+ fp->u.fp = NULL;
free_keep_errno(fp);
return result;
}
@@ -1173,25 +1182,26 @@ int
cfeof(cfp *fp)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
- return gzeof(fp->compressedfp);
+ if (fp->alg == COMPR_ALG_LIBZ)
+ return gzeof(fp->u.gzfp);
#endif
#ifdef HAVE_LIBZSTD
- if (fp->zstd.fp)
- return feof(fp->zstd.fp);
+ if (fp->alg == COMPR_ALG_ZSTD)
+ return feof(fp->u.zstd.fp);
#endif
- return feof(fp->uncompressedfp);
+
+ return feof(fp->u.fp);
}
const char *
get_cfp_error(cfp *fp)
{
#ifdef HAVE_LIBZ
- if (fp->compressedfp)
+ if (fp->alg == COMPR_ALG_LIBZ)
{
int errnum;
- const char *errmsg = gzerror(fp->compressedfp, &errnum);
+ const char *errmsg = gzerror(fp->u.gzfp, &errnum);
if (errnum != Z_ERRNO)
return errmsg;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7c2f7a9ca3..5e009e5854 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -397,7 +397,7 @@ parse_compression(const char *optarg, Compress *compress)
const int default_compress_level[] = {
0, /* COMPR_ALG_NONE */
Z_DEFAULT_COMPRESSION, /* COMPR_ALG_ZLIB */
- 0, // XXX: ZSTD_CLEVEL_DEFAULT, /* COMPR_ALG_ZSTD */
+ 0, // #ifdef LIBZSTD ZSTD_CLEVEL_DEFAULT, /* COMPR_ALG_ZSTD */
};
compress->level = default_compress_level[compress->alg];
--
2.17.0
[text/x-diff] 0009-Move-zlib-into-the-union.patch (4.6K, ../../20210104025321.GA9712@telsasoft.com/10-0009-Move-zlib-into-the-union.patch)
download | inline diff:
From c43384ebc1ff47536b207374ed472182ff4ae0e8 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 11 Dec 2020 22:22:31 -0600
Subject: [PATCH 09/20] Move zlib into the union{}
---
src/bin/pg_dump/compress_io.c | 56 ++++++++++++++++++-----------------
1 file changed, 29 insertions(+), 27 deletions(-)
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index e07436bc21..ad085a7d54 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -91,23 +91,25 @@ struct CompressorState
CompressionAlgorithm comprAlg;
WriteFunc writeF;
+ union {
#ifdef HAVE_LIBZ
- z_streamp zp;
- char *zlibOut;
- size_t zlibOutSize;
+ struct {
+ z_streamp zp;
+ char *zlibOut;
+ size_t zlibOutSize;
+ } zlib;
#endif
#ifdef HAVE_LIBZSTD
- union {
+ /* This is used for compression but not decompression */
struct {
- ZSTD_outBuffer output;
- ZSTD_inBuffer input;
// XXX: use one separate ZSTD_CStream per thread: disable on windows ?
- ZSTD_CStream *cstream;
+ ZSTD_CStream *cstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
} zstd;
- } u;
#endif
-
+ } u;
};
/* Routines that support zlib compressed data I/O */
@@ -452,7 +454,7 @@ InitCompressorZlib(CompressorState *cs, Compress *compress)
{
z_streamp zp;
- zp = cs->zp = (z_streamp) pg_malloc(sizeof(z_stream));
+ zp = cs->u.zlib.zp = (z_streamp) pg_malloc(sizeof(z_stream));
zp->zalloc = Z_NULL;
zp->zfree = Z_NULL;
zp->opaque = Z_NULL;
@@ -462,22 +464,22 @@ InitCompressorZlib(CompressorState *cs, Compress *compress)
* actually allocate one extra byte because some routines want to append a
* trailing zero byte to the zlib output.
*/
- cs->zlibOut = (char *) pg_malloc(ZLIB_OUT_SIZE + 1);
- cs->zlibOutSize = ZLIB_OUT_SIZE;
+ cs->u.zlib.zlibOut = (char *) pg_malloc(ZLIB_OUT_SIZE + 1);
+ cs->u.zlib.zlibOutSize = ZLIB_OUT_SIZE;
if (deflateInit(zp, compress->level) != Z_OK)
fatal("could not initialize compression library: %s",
zp->msg);
/* Just be paranoid - maybe End is called after Start, with no Write */
- zp->next_out = (void *) cs->zlibOut;
- zp->avail_out = cs->zlibOutSize;
+ zp->next_out = (void *) cs->u.zlib.zlibOut;
+ zp->avail_out = cs->u.zlib.zlibOutSize;
}
static void
EndCompressorZlib(ArchiveHandle *AH, CompressorState *cs)
{
- z_streamp zp = cs->zp;
+ z_streamp zp = cs->u.zlib.zp;
zp->next_in = NULL;
zp->avail_in = 0;
@@ -488,23 +490,23 @@ EndCompressorZlib(ArchiveHandle *AH, CompressorState *cs)
if (deflateEnd(zp) != Z_OK)
fatal("could not close compression stream: %s", zp->msg);
- free(cs->zlibOut);
- free(cs->zp);
+ free(cs->u.zlib.zlibOut);
+ free(cs->u.zlib.zp);
}
static void
DeflateCompressorZlib(ArchiveHandle *AH, CompressorState *cs, bool flush)
{
- z_streamp zp = cs->zp;
- char *out = cs->zlibOut;
+ z_streamp zp = cs->u.zlib.zp;
+ char *out = cs->u.zlib.zlibOut;
int res = Z_OK;
- while (cs->zp->avail_in != 0 || flush)
+ while (cs->u.zlib.zp->avail_in != 0 || flush)
{
res = deflate(zp, flush ? Z_FINISH : Z_NO_FLUSH);
if (res == Z_STREAM_ERROR)
fatal("could not compress data: %s", zp->msg);
- if ((flush && (zp->avail_out < cs->zlibOutSize))
+ if ((flush && (zp->avail_out < cs->u.zlib.zlibOutSize))
|| (zp->avail_out == 0)
|| (zp->avail_in != 0)
)
@@ -514,18 +516,18 @@ DeflateCompressorZlib(ArchiveHandle *AH, CompressorState *cs, bool flush)
* chunk is the EOF marker in the custom format. This should never
* happen but...
*/
- if (zp->avail_out < cs->zlibOutSize)
+ if (zp->avail_out < cs->u.zlib.zlibOutSize)
{
/*
* Any write function should do its own error checking but to
* make sure we do a check here as well...
*/
- size_t len = cs->zlibOutSize - zp->avail_out;
+ size_t len = cs->u.zlib.zlibOutSize - zp->avail_out;
cs->writeF(AH, out, len);
}
zp->next_out = (void *) out;
- zp->avail_out = cs->zlibOutSize;
+ zp->avail_out = cs->u.zlib.zlibOutSize;
}
if (res == Z_STREAM_END)
@@ -537,8 +539,8 @@ static void
WriteDataToArchiveZlib(ArchiveHandle *AH, CompressorState *cs,
const char *data, size_t dLen)
{
- cs->zp->next_in = (void *) unconstify(char *, data);
- cs->zp->avail_in = dLen;
+ cs->u.zlib.zp->next_in = (void *) unconstify(char *, data);
+ cs->u.zlib.zp->avail_in = dLen;
DeflateCompressorZlib(AH, cs, false);
}
@@ -898,7 +900,7 @@ cfdopen(int fd, const char *mode, Compress *compression)
fp = NULL;
}
else
- setvbuf(fp->uncompressedfp, NULL, _IONBF, 0);
+ setvbuf(fp->u.fp, NULL, _IONBF, 0);
return fp;
default:
--
2.17.0
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2020-12-21 19:49 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2020-12-21 20:02 ` Re: zstd compression for pg_dump Tom Lane <tgl@sss.pgh.pa.us>
2020-12-22 02:32 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 02:53 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2021-01-04 06:04 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-04 07:06 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 10:17 ` Re: zstd compression for pg_dump Daniil Zakhlystov <usernamedt@yandex-team.ru>
2 siblings, 2 replies; 46+ messages in thread
From: Andrey Borodin @ 2021-01-04 06:04 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers; Daniil Zakhlystov <usernamedt@yandex-team.ru>
> 4 янв. 2021 г., в 07:53, Justin Pryzby <pryzby@telsasoft.com> написал(а):
>
> Note, there's currently several "compression" patches in CF app. This patch
> seems to be independent of the others, but probably shouldn't be totally
> uncoordinated (like adding lz4 in one and ztsd in another might be poor
> execution).
>
> https://commitfest.postgresql.org/31/2897/
> - Faster pglz compression
> https://commitfest.postgresql.org/31/2813/
> - custom compression methods for toast
> https://commitfest.postgresql.org/31/2773/
> - libpq compression
I think that's downside of our development system: patch authors do not want to create dependencies on other patches.
I'd say that both lz4 and zstd should be supported in TOAST, FPIs, libpq, and pg_dump. As to pglz - I think we should not proliferate it any further.
Lz4 and Zstd represent a different tradeoff actually. Basically, lz4 is so CPU-cheap that one should use it whenever they write to disk or network interface. Zstd represent an actual bandwith\CPU tradeoff.
Also, all patchsets do not touch important possibility - preexisting dictionary could radically improve compression of small data (event in pglz).
Some minor notes on patchset at this thread.
Libpq compression encountered some problems with memory consumption which required some extra config efforts. Did you measure memory usage for this patchset?
[PATCH 03/20] Support multiple compression algs/levels/opts..
abtracts -> abstracts
enum CompressionAlgorithm actually represent the very same thing as in "Custom compression methods"
Daniil, is levels definition compatible with libpq compression patch?
+typedef struct Compress {
+ CompressionAlgorithm alg;
+ int level;
+ /* Is a nondefault level set ? This is useful since different compression
+ * methods have different "default" levels. For now we assume the levels
+ * are all integer, though.
+ */
+ bool level_set;
+} Compress;
[PATCH 04/20] struct compressLibs
I think this directive would be correct.
+// #ifdef HAVE_LIBZ?
Here's extra comment
// && errno == ENOENT)
[PATCH 06/20] pg_dump: zstd compression
I'd propose to build with Zstd by default. It seems other patches do it this way. Though, I there are possible downsides.
Thanks for working on this! We will have very IO-efficient Postgres :)
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2020-12-21 19:49 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2020-12-21 20:02 ` Re: zstd compression for pg_dump Tom Lane <tgl@sss.pgh.pa.us>
2020-12-22 02:32 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 02:53 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 06:04 ` Re: zstd compression for pg_dump Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-01-04 07:06 ` Justin Pryzby <pryzby@telsasoft.com>
1 sibling, 0 replies; 46+ messages in thread
From: Justin Pryzby @ 2021-01-04 07:06 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers; Daniil Zakhlystov <usernamedt@yandex-team.ru>
On Mon, Jan 04, 2021 at 11:04:57AM +0500, Andrey Borodin wrote:
> > 4 янв. 2021 г., в 07:53, Justin Pryzby <pryzby@telsasoft.com> написал(а):
> > Note, there's currently several "compression" patches in CF app. This patch
> > seems to be independent of the others, but probably shouldn't be totally
> > uncoordinated (like adding lz4 in one and ztsd in another might be poor
> > execution).
> >
> > https://commitfest.postgresql.org/31/2897/
> > - Faster pglz compression
> > https://commitfest.postgresql.org/31/2813/
> > - custom compression methods for toast
> > https://commitfest.postgresql.org/31/2773/
> > - libpq compression
>
> I think that's downside of our development system: patch authors do not want to create dependencies on other patches.
I think in these cases, someone who notices common/overlapping patches should
suggest that the authors review each other's work. In some cases, I think it's
appropriate to come up with a "shared" preliminary patch(es), which both (all)
patch authors can include as 0001 until its finalized and merged. That might
be true for some things like the tableam work, or the two "online checksum"
patches.
> I'd say that both lz4 and zstd should be supported in TOAST, FPIs, libpq, and pg_dump. As to pglz - I think we should not proliferate it any further.
pg_basebackup came up as another use on another thread, I think related to
libpq protocol compression.
> Libpq compression encountered some problems with memory consumption which
> required some extra config efforts. Did you measure memory usage for this
> patchset?
RAM use is not significantly different from zlib, except that zstd --long adds
more memory.
$ command time -v pg_dump -d ts -t ... -Fc -Z0 |wc -c
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:28.77
Maximum resident set size (kbytes): 40504
1397288924 # no compression: 1400MB
$ command time -v pg_dump -d ts -t ... -Fc |wc -c
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:37.17
Maximum resident set size (kbytes): 40504
132932415 # default (zlib) compression: 132 MB
$ command time -v ./pg_dump -d ts -t ... -Fc |wc -c
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:29.28
Maximum resident set size (kbytes): 40568
86048139 # zstd: 86MB
$ command time -v ./pg_dump -d ts -t ... -Fc -Z 'alg=zstd opt=zstdlong' |wc -c
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:30.49
Maximum resident set size (kbytes): 180332
72202937 # zstd long: 180MB
> [PATCH 04/20] struct compressLibs
> I think this directive would be correct.
> +// #ifdef HAVE_LIBZ?
I'm not sure .. I'm thinking of making the COMPR_ALG_* always defined, and then
fail later if an operation is unsupported. There's an excessive number of
#ifdefs already, so the early commits are intended to minimize as far as
possible what's needed for each additional compression
algorithm(lib/method/whatever it's called). I haven't tested much with
pg_restore of files with unsupported compression libs.
> [PATCH 06/20] pg_dump: zstd compression
> I'd propose to build with Zstd by default. It seems other patches do it this way. Though, I there are possible downsides.
Yes...but the cfbot turns red if the patch require zstd, so it defaults to
off until it's included in the build environments (but for now, the main patch
isn't being tested).
Thanks for looking.
--
Justin
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2020-12-21 19:49 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2020-12-21 20:02 ` Re: zstd compression for pg_dump Tom Lane <tgl@sss.pgh.pa.us>
2020-12-22 02:32 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 02:53 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 06:04 ` Re: zstd compression for pg_dump Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-01-04 10:17 ` Daniil Zakhlystov <usernamedt@yandex-team.ru>
2021-03-12 22:31 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
1 sibling, 1 reply; 46+ messages in thread
From: Daniil Zakhlystov @ 2021-01-04 10:17 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi!
> On Jan 4, 2021, at 11:04 AM, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>
> Daniil, is levels definition compatible with libpq compression patch?
> +typedef struct Compress {
> + CompressionAlgorithm alg;
> + int level;
> + /* Is a nondefault level set ? This is useful since different compression
> + * methods have different "default" levels. For now we assume the levels
> + * are all integer, though.
> + */
> + bool level_set;
> +} Compress;
Similarly to this patch, it is also possible to define the compression level at the initialization stage in libpq compression patch.
The difference is that in libpq compression patch the default compression level always equal to 1, independently of the chosen compression algorithm.
> On Jan 4, 2021, at 11:04 AM, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>
> Libpq compression encountered some problems with memory consumption which required some extra config efforts.
> On Jan 4, 2021, at 12:06 PM, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> RAM use is not significantly different from zlib, except that zstd --long adds
> more memory.
Regarding ZSTD memory usage:
Recently I’ve made a couple of tests of libpq compression with different ZLIB/ZSTD compression levels which shown that compressing/decompressing ZSTD w/ high compression levels
require to allocate more virtual (Commited_AS) memory, which may be exploited by malicious clients:
https://www.postgresql.org/message-id/62527092-16BD-479F-B503-FA527AF3B0C2%40yandex-team.ru
We can avoid high memory usage by limiting the max window size to 8MB. This should effectively disable the support of compression levels above 19:
https://www.postgresql.org/message-id/6A45DFAA-1682-4EF2-B835-C5F46615EC49%40yandex-team.ru
So maybe it is worthwhile to use similar restrictions in this patch.
—
Daniil Zakhlystov
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2020-12-21 19:49 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2020-12-21 20:02 ` Re: zstd compression for pg_dump Tom Lane <tgl@sss.pgh.pa.us>
2020-12-22 02:32 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 02:53 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 06:04 ` Re: zstd compression for pg_dump Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-04 10:17 ` Re: zstd compression for pg_dump Daniil Zakhlystov <usernamedt@yandex-team.ru>
@ 2021-03-12 22:31 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
0 siblings, 0 replies; 46+ messages in thread
From: Tomas Vondra @ 2021-03-12 22:31 UTC (permalink / raw)
To: Daniil Zakhlystov <usernamedt@yandex-team.ru>; Justin Pryzby <pryzby@telsasoft.com>; Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 1/4/21 11:17 AM, Daniil Zakhlystov wrote:
> Hi!
>
>> On Jan 4, 2021, at 11:04 AM, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>>
>> Daniil, is levels definition compatible with libpq compression patch?
>> +typedef struct Compress {
>> + CompressionAlgorithm alg;
>> + int level;
>> + /* Is a nondefault level set ? This is useful since different compression
>> + * methods have different "default" levels. For now we assume the levels
>> + * are all integer, though.
>> + */
>> + bool level_set;
>> +} Compress;
>
> Similarly to this patch, it is also possible to define the compression level at the initialization stage in libpq compression patch.
>
> The difference is that in libpq compression patch the default compression level always equal to 1, independently of the chosen compression algorithm.
>
>> On Jan 4, 2021, at 11:04 AM, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>>
>> Libpq compression encountered some problems with memory consumption which required some extra config efforts.
>
>
>> On Jan 4, 2021, at 12:06 PM, Justin Pryzby <pryzby@telsasoft.com> wrote:
>>
>> RAM use is not significantly different from zlib, except that zstd --long adds
>> more memory.
>
> Regarding ZSTD memory usage:
>
> Recently I’ve made a couple of tests of libpq compression with different ZLIB/ZSTD compression levels which shown that compressing/decompressing ZSTD w/ high compression levels
> require to allocate more virtual (Commited_AS) memory, which may be exploited by malicious clients:
>
> https://www.postgresql.org/message-id/62527092-16BD-479F-B503-FA527AF3B0C2%40yandex-team.ru
>
> We can avoid high memory usage by limiting the max window size to 8MB. This should effectively disable the support of compression levels above 19:
> https://www.postgresql.org/message-id/6A45DFAA-1682-4EF2-B835-C5F46615EC49%40yandex-team.ru
>
> So maybe it is worthwhile to use similar restrictions in this patch.
>
I think there's a big difference between those two patches. In the libpq
case, the danger is that the client requests the server to compress the
data in a way that requires a lot of memory. I.e. the memory is consumed
on the server.
With this pg_dump patch, the compression is done by the pg_dump process,
not the server. So if the attacker configures the compression in a way
that requires a lot of memory, so what? He'll just allocate memory on
the client machine, where he could also just run a custom binary that
does a huge malloc().
So I don't think we need to worry about this too much.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2020-12-21 19:49 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2020-12-21 20:02 ` Re: zstd compression for pg_dump Tom Lane <tgl@sss.pgh.pa.us>
2020-12-22 02:32 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 02:53 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2021-01-10 21:06 ` Andreas Karlsson <andreas@proxel.se>
2 siblings, 0 replies; 46+ messages in thread
From: Andreas Karlsson @ 2021-01-10 21:06 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers
On 1/4/21 3:53 AM, Justin Pryzby wrote:
>> About 89% smaller.
Did a quick code review of the patch. I have not yet taken it for a spin
yet and there are parts of the code I have not read yet.
## Is there any reason for this diff?
- cfp *fp = pg_malloc(sizeof(cfp));
+ cfp *fp = pg_malloc0(sizeof(cfp));
## Since we know have multiple returns in cfopen() I am not sure that
setting fp to NULL is still clearer than just returning NULL.
## I do not like that this pretends to support r+, w+ and a+ but does
not actually do so since it does not create an input stream in those cases.
else if (mode[0] == 'w' || mode[0] == 'a' ||
strchr(mode, '+') != NULL)
[...]
else if (strchr(mode, 'r'))
## Wouldn't cfread(), cfwrite(), cfgetc(), cfgets(), cfclose() and
cfeof() be cleaner with sitch statments similar to cfopen()?
## "/* Should be called "method" or "library" ? */"
Maybe, but personally I think algorithm is fine too.
## "Is a nondefault level set ?"
The PostgreSQL project does not use space before question mark (at least
not in English).
## Why isn't level_set just a local variable in parse_compression()? It
does not seem to be used elsewhere.
## Shouldn't we call the Compression variable in OpenArchive()
nocompress to match with the naming convention in other places.
And in general I wonder if we should not write "nocompression =
{COMPR_ALG_NONE}" rather than "nocompression = {0}".
## Why not use const on the pointers to Compression for functions like
cfopen()? As far as I can see several of them could be const.
## Shouldn't "AH->compression.alg = Z_DEFAULT_COMPRESSION" in ReadHead()
be "AH->compression.alg = COMPR_ALG_DEFAULT"?
Additionally I am not convinced that returning COMPR_ALG_DEFAULT will
even work but I have not had the time to test that theory yet. And in
general I am quite sceptical of that we really need of COMPR_ALG_DEFAULT.
## Some white space issues
Add spaces around plus in "atoi(1+eq)" and "pg_log_error("unknown
compression algorithm: %s", 1+eq)".
Add spaces around plus in parse_compression(), e.g. in "strlen(1+eq)".
## Shouldn't hasSuffix() take the current compression algorithm as a
parameter? Or alternatively look up which compression algorithm to use
from the suffix?
## Why support multiple ways to write zlib on the command line? I do not
see any advatange of being able to write it as libz.
## I feel renaming SaveOutput() to GetOutput() would make it more clear
what it does now that you have changed the return type.
## You have accidentally committed "-runstatedir" in configure. I have
no idea why we do not have it (maybe it is something Debian specific)
but even if we are going to add it it should not be in this patch. Same
with the parenthesis changes to LARGE_OFF_T.
## This is probably out of scope of your patch but I am not a fan of the
fallback logic in cfopen_read(). I feel ideally we should always know if
there is a suffix or not and not try to guess file names and do
pointless syscalls.
## COMPR_ALG_DEFAULT looks like it would error out for archive and
directory if someone has neither zlib nor zstandard. It feels like it
should default to uncompressed if we have neither. Or at least give a
better error message.
> Note, there's currently several "compression" patches in CF app. This patch
> seems to be independent of the others, but probably shouldn't be totally
> uncoordinated (like adding lz4 in one and ztsd in another might be poor
> execution).
A thought here is that maybe we want to use the same values for the
enums in all patches. Especially if we write the numeric value to pg
dump files.
Andreas
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2020-12-21 19:49 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2020-12-21 20:02 ` Re: zstd compression for pg_dump Tom Lane <tgl@sss.pgh.pa.us>
2020-12-22 02:32 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 02:53 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2021-04-08 15:19 ` David Steele <david@pgmasters.net>
2 siblings, 0 replies; 46+ messages in thread
From: David Steele @ 2021-04-08 15:19 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers
On 1/3/21 9:53 PM, Justin Pryzby wrote:
> I rebased so the "typedef struct compression" patch is first and zstd on top of
> that (say, in case someone wants to bikeshed about which compression algorithm
> to support). And made a central struct with all the compression-specific info
> to further isolate the compress-specific changes.
It has been a few months since there was a new patch and the current one
no longer applies, so marking Returned with Feedback.
Please resubmit to the next CF when you have a new patch.
Regards,
--
-David
david@pgmasters.net
^ permalink raw reply [nested|flat] 46+ messages in thread
* zstd compression for pg_dump
@ 2023-02-24 19:18 Justin Pryzby <pryzby@telsasoft.com>
2023-02-25 04:44 ` Re: zstd compression for pg_dump Michael Paquier <michael@paquier.xyz>
2023-02-25 10:31 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 3 replies; 46+ messages in thread
From: Justin Pryzby @ 2023-02-24 19:18 UTC (permalink / raw)
To: pgsql-hackers; +Cc: gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
This is a draft patch - review is welcome and would help to get this
ready to be considererd for v16, if desired.
I'm going to add this thread to the old CF entry.
https://commitfest.postgresql.org/31/2888/
--
Justin
Attachments:
[text/x-diff] 0001-WIP-pg_dump-zstd-compression.patch (23.6K, ../../20230224191840.GD1653@telsasoft.com/2-0001-WIP-pg_dump-zstd-compression.patch)
download | inline diff:
From 2486417b7c3586e150e806a1fbc3b873c2a4a0f9 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 7 Jan 2023 15:45:06 -0600
Subject: [PATCH 1/3] WIP: pg_dump: zstd compression
Previously proposed at: 20201221194924.GI30237@telsasoft.com
note-to-self: see also: private commit 36ab001fb
---
src/bin/pg_dump/Makefile | 1 +
src/bin/pg_dump/compress_io.c | 54 +--
src/bin/pg_dump/compress_zstd.c | 520 ++++++++++++++++++++++++++
src/bin/pg_dump/compress_zstd.h | 9 +
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/pg_backup_archiver.c | 9 +-
src/bin/pg_dump/pg_backup_directory.c | 2 +
src/bin/pg_dump/pg_dump.c | 13 -
src/bin/pg_dump/t/002_pg_dump.pl | 71 ++++
9 files changed, 640 insertions(+), 40 deletions(-)
create mode 100644 src/bin/pg_dump/compress_zstd.c
create mode 100644 src/bin/pg_dump/compress_zstd.h
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index eb8f59459a1..76574298faf 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -29,6 +29,7 @@ OBJS = \
compress_io.o \
compress_lz4.o \
compress_none.o \
+ compress_zstd.o \
dumputils.o \
parallel.o \
pg_backup_archiver.o \
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index ce06f1eac9c..061e3d9ce1c 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -52,8 +52,8 @@
*
* InitDiscoverCompressFileHandle tries to infer the compression by the
* filename suffix. If the suffix is not yet known then it tries to simply
- * open the file and if it fails, it tries to open the same file with the .gz
- * suffix, and then again with the .lz4 suffix.
+ * open the file and if it fails, it tries to open the same file with
+ * compressed suffixes.
*
* IDENTIFICATION
* src/bin/pg_dump/compress_io.c
@@ -69,6 +69,7 @@
#include "compress_io.h"
#include "compress_lz4.h"
#include "compress_none.h"
+#include "compress_zstd.h"
#include "pg_backup_utils.h"
/*----------------------
@@ -98,6 +99,10 @@ supports_compression(const pg_compress_specification compression_spec)
if (algorithm == PG_COMPRESSION_LZ4)
supported = true;
#endif
+#ifdef USE_ZSTD
+ if (algorithm == PG_COMPRESSION_ZSTD)
+ supported = true;
+#endif
if (!supported)
return psprintf("this build does not support compression with %s",
@@ -130,6 +135,8 @@ AllocateCompressor(const pg_compress_specification compression_spec,
InitCompressorGzip(cs, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressorLZ4(cs, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressorZstd(cs, compression_spec);
return cs;
}
@@ -196,25 +203,36 @@ InitCompressFileHandle(const pg_compress_specification compression_spec)
InitCompressFileHandleGzip(CFH, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressFileHandleLZ4(CFH, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressFileHandleZstd(CFH, compression_spec);
return CFH;
}
+static bool
+check_compressed_file(const char *path, char **fname, char *ext)
+{
+ free_keep_errno(*fname);
+ *fname = psprintf("%s.%s", path, ext);
+ return (access(*fname, F_OK) == 0);
+}
+
/*
* Open a file for reading. 'path' is the file to open, and 'mode' should
* be either "r" or "rb".
*
* If the file at 'path' contains the suffix of a supported compression method,
- * currently this includes ".gz" and ".lz4", then this compression will be used
+ * currently this includes ".gz", ".lz4" and ".zst", then this compression will be used
* throughout. Otherwise the compression will be inferred by iteratively trying
* to open the file at 'path', first as is, then by appending known compression
* suffixes. So if you pass "foo" as 'path', this will open either "foo" or
- * "foo.gz" or "foo.lz4", trying in that order.
+ * "foo.{gz,lz4,zst}", trying in that order.
*
* On failure, return NULL with an error code in errno.
*/
CompressFileHandle *
InitDiscoverCompressFileHandle(const char *path, const char *mode)
+// pg_compress_algorithm alg
{
CompressFileHandle *CFH = NULL;
struct stat st;
@@ -237,28 +255,12 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode)
/* avoid unused warning if it is not built with compression */
if (exists)
compression_spec.algorithm = PG_COMPRESSION_NONE;
-#ifdef HAVE_LIBZ
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.gz", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_GZIP;
- }
-#endif
-#ifdef USE_LZ4
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.lz4", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_LZ4;
- }
-#endif
+ else if (check_compressed_file(path, &fname, "gz")) // alg == PG_COMPRESSION_GZIP &&
+ compression_spec.algorithm = PG_COMPRESSION_GZIP;
+ else if (check_compressed_file(path, &fname, "lz4")) // alg == PG_COMPRESSION_LZ4 &&
+ compression_spec.algorithm = PG_COMPRESSION_LZ4;
+ else if (check_compressed_file(path, &fname, "zst")) // alg == PG_COMPRESSION_ZSTD &&
+ compression_spec.algorithm = PG_COMPRESSION_ZSTD;
}
CFH = InitCompressFileHandle(compression_spec);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
new file mode 100644
index 00000000000..659d08533ae
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -0,0 +1,520 @@
+#include "postgres_fe.h"
+
+#include "pg_backup_utils.h"
+#include "compress_zstd.h"
+
+#ifndef USE_ZSTD
+
+void
+InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+#else
+
+#include <zstd.h>
+
+typedef struct ZstdCompressorState
+{
+ /* This is a normal file to which we read/write compressed data */
+ FILE *fp;
+ /* XXX: use one separate ZSTD_CStream per thread: disable on windows ? */
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+} ZstdCompressorState;
+
+static ZSTD_CStream *ZstdCStreamParams(pg_compress_specification compress);
+static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen);
+static void ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs);
+
+static void
+ZSTD_CCtx_setParam_or_die(ZSTD_CStream * cstream,
+ ZSTD_cParameter param, int value)
+{
+ size_t res;
+
+ res = ZSTD_CCtx_setParameter(cstream, param, value);
+ if (ZSTD_isError(res))
+ pg_fatal("could not set compression parameter: %s",
+ ZSTD_getErrorName(res));
+}
+
+/* Return a compression stream with parameters set per argument */
+static ZSTD_CStream *
+ZstdCStreamParams(pg_compress_specification compress)
+{
+ ZSTD_CStream *cstream;
+
+ cstream = ZSTD_createCStream();
+ if (cstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_compressionLevel,
+ compress.level);
+
+ if (compress.options & PG_COMPRESSION_OPTION_WORKERS)
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
+ compress.workers);
+
+#if 0
+ if (compress.options & PG_COMPRESSION_OPTION_CHECKSUM)
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_checksumFlag,
+ compress.checksum);
+
+ /* Still marked as experimental */
+ if (compress.options & PG_COMPRESSION_OPTION_RSYNCABLE)
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_rsyncable, 1);
+#endif
+
+ return cstream;
+}
+
+void
+EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ if (cs->writeF == NULL)
+ return;
+
+ for (;;)
+ {
+ size_t res;
+
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output,
+ &zstdcs->input, ZSTD_e_end);
+
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+ /* TODO check that we wrote "pos" bytes */
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not close compression stream: %s",
+ ZSTD_getErrorName(res));
+
+ if (res == 0)
+ break;
+ }
+
+ /* XXX: retval */
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ pg_free(zstdcs);
+}
+
+static void
+WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ input->src = data;
+ input->size = dLen;
+ input->pos = 0;
+
+#if 0
+ ZSTD_CCtx_reset(zstdcs->cstream, ZSTD_reset_session_only); // XXX */
+ res = ZSTD_CCtx_setPledgedSrcSize(cs->zstd.cstream, dLen);
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+#endif
+
+ while (input->pos != input->size)
+ {
+ size_t res;
+
+ res = ZSTD_compressStream2(zstdcs->cstream, output,
+ input, ZSTD_e_continue);
+
+ if (output->pos == output->size ||
+ input->pos != input->size)
+ {
+ /*
+ * Extra paranoia: avoid zero-length chunks, since a zero length
+ * chunk is the EOF marker in the custom format. This should never
+ * happen but...
+ */
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ output->pos = 0;
+ }
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+ }
+}
+
+/* Read data from a compressed zstd archive */
+static void
+ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+ size_t res;
+ size_t input_size;
+
+ dstream = ZSTD_createDStream();
+ if (dstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ input_size = ZSTD_DStreamInSize();
+ input.src = pg_malloc(input_size);
+
+ output.size = ZSTD_DStreamOutSize();
+ output.dst = pg_malloc(output.size);
+
+ /* read compressed data */
+ for (;;)
+ {
+ size_t cnt;
+
+ /*
+ * XXX: the buffer can grow, we shouldn't keep resetting it to the
+ * original value..
+ */
+ input.size = input_size;
+
+ cnt = cs->readF(AH, (char **) unconstify(void **, &input.src), &input.size);
+ input.pos = 0;
+ input.size = cnt;
+
+ if (cnt == 0)
+ break;
+
+ while (input.pos < input.size)
+ {
+ /* decompress */
+ output.pos = 0;
+ res = ZSTD_decompressStream(dstream, &output, &input);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ /* write to output handle */
+ ((char *) output.dst)[output.pos] = '\0';
+ ahwrite(output.dst, 1, output.pos, AH);
+ /* if (res == 0) break; */
+ }
+ }
+
+ pg_free(unconstify(void *, input.src));
+ pg_free(output.dst);
+}
+
+/* Public routines that support Zstd compressed data I/O */
+void
+InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec)
+{
+ ZstdCompressorState *zstdcs;
+
+ cs->readData = ReadDataFromArchiveZstd;
+ cs->writeData = WriteDataToArchiveZstd;
+ cs->end = EndCompressorZstd;
+
+ cs->compression_spec = compression_spec;
+
+ cs->private_data = pg_malloc0(sizeof(ZstdCompressorState));
+ zstdcs = cs->private_data;
+ /* XXX: initialize safely like the corresponding zlib "paranoia" */
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size);
+ zstdcs->output.pos = 0;
+ zstdcs->cstream = ZstdCStreamParams(cs->compression_spec);
+}
+
+/*----------------------
+ * Compress File API
+ *----------------------
+ */
+
+static size_t
+Zstd_read(void *ptr, size_t size, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t input_size = ZSTD_DStreamInSize();
+
+ /* input_size is the allocated size */
+ size_t res,
+ cnt;
+
+ output->size = size;
+ output->dst = ptr;
+ output->pos = 0;
+
+ for (;;)
+ {
+ Assert(input->pos <= input->size);
+ Assert(input->size <= input_size);
+
+ /* If the input is completely consumed, start back at the beginning */
+ if (input->pos == input->size)
+ {
+ /* input->size is size produced by "fread" */
+ input->size = 0;
+ /* input->pos is position consumed by decompress */
+ input->pos = 0;
+ }
+
+ /* read compressed data if we must produce more input */
+ if (input->pos == input->size)
+ {
+ cnt = fread(unconstify(void *, input->src), 1, input_size, zstdcs->fp);
+ input->size = cnt;
+
+ /* If we have no input to consume, we're done */
+ if (cnt == 0)
+ break;
+ }
+
+ Assert(cnt >= 0);
+ Assert(input->size <= input_size);
+
+ /* Now consume as much as possible */
+ for (; input->pos < input->size;)
+ {
+ /* decompress */
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+ if (output->pos == output->size)
+ break; /* No more room for output */
+ if (res == 0)
+ break; /* End of frame */
+ }
+
+ if (output->pos == output->size)
+ break; /* We read all the data that fits */
+ }
+
+ return output->pos;
+}
+
+static size_t
+Zstd_write(const void *ptr, size_t size, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t res,
+ cnt;
+
+ input->src = ptr;
+ input->size = size;
+ input->pos = 0;
+
+#if 0
+ ZSTD_CCtx_reset(fp->zstd.cstream, ZSTD_reset_session_only);
+ res = ZSTD_CCtx_setPledgedSrcSize(fp->zstd.cstream, size);
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+#endif
+
+ /* Consume all input, and flush later */
+ while (input->pos != input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_continue);
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ pg_fatal("could not write data: %m");
+ }
+
+ return size;
+}
+
+static int
+Zstd_getc(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ int ret;
+
+ if (CFH->read_func(&ret, 1, CFH) != 1)
+ {
+ if (feof(zstdcs->fp))
+ pg_fatal("could not read from input file: end of file");
+ else
+ pg_fatal("could not read from input file: %m");
+ }
+ return ret;
+}
+
+static char *
+Zstd_gets(char *buf, int len, CompressFileHandle *CFH)
+{
+ /*
+ * Read one byte at a time until newline or EOF. This is only used to read
+ * the list of blobs, and the I/O is buffered anyway.
+ */
+ int i,
+ res;
+
+ for (i = 0; i < len - 1; ++i)
+ {
+ res = CFH->read_func(&buf[i], 1, CFH);
+ if (res != 1)
+ break;
+ if (buf[i] == '\n')
+ {
+ ++i;
+ break;
+ }
+ }
+ buf[i] = '\0';
+ return i > 0 ? buf : 0;
+}
+
+static int
+Zstd_close(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ int result;
+
+ if (zstdcs->cstream)
+ {
+ size_t res,
+ cnt;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ for (;;)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_end);
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ pg_fatal("could not write data: %m");
+
+ if (res == 0)
+ break;
+ }
+
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ if (zstdcs->dstream)
+ {
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ }
+
+ result = fclose(zstdcs->fp);
+ pg_free(zstdcs);
+ return result;
+}
+
+static int
+Zstd_eof(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ return feof(zstdcs->fp);
+}
+
+static int
+Zstd_open(const char *path, int fd, const char *mode,
+ CompressFileHandle *CFH)
+{
+ FILE *fp;
+ ZstdCompressorState *zstdcs;
+
+ if (fd >= 0)
+ fp = fdopen(fd, mode);
+ else
+ fp = fopen(path, mode);
+
+ if (fp == NULL)
+ {
+ /* XXX zstdcs->errcode = errno; */
+ return 1;
+ }
+
+ CFH->private_data = pg_malloc0(sizeof(ZstdCompressorState));
+ zstdcs = (ZstdCompressorState *) CFH->private_data;
+ zstdcs->fp = fp;
+
+ if (mode[0] == 'w' || mode[0] == 'a')
+ {
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc0(zstdcs->output.size);
+ zstdcs->cstream = ZstdCStreamParams(CFH->compression_spec);
+ }
+ else if (strchr(mode, 'r'))
+ {
+ zstdcs->input.src = pg_malloc0(ZSTD_DStreamInSize());
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+ }
+ /* XXX else: bad mode */
+
+ return 0;
+}
+
+static int
+Zstd_open_write(const char *path, const char *mode, CompressFileHandle *CFH)
+{
+ char fname[MAXPGPATH];
+
+ sprintf(fname, "%s.zst", path);
+ return CFH->open_func(fname, -1, mode, CFH);
+}
+
+static const char *
+Zstd_get_error(CompressFileHandle *CFH)
+{
+#if 0
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ if (ZSTD_isError(res))
+ return ZSTD_getErrorName(res)
+ else
+#endif
+
+ return strerror(errno);
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec)
+{
+ CFH->open_func = Zstd_open;
+ CFH->open_write_func = Zstd_open_write;
+ CFH->read_func = Zstd_read;
+ CFH->write_func = Zstd_write;
+ CFH->gets_func = Zstd_gets;
+ CFH->getc_func = Zstd_getc;
+ CFH->close_func = Zstd_close;
+ CFH->eof_func = Zstd_eof;
+ CFH->get_error_func = Zstd_get_error;
+
+ CFH->compression_spec = compression_spec;
+
+ CFH->private_data = NULL;
+}
+
+#endif /* USE_ZSTD */
diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h
new file mode 100644
index 00000000000..f36698b4c26
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.h
@@ -0,0 +1,9 @@
+#ifndef COMPRESS_ZSTD_H
+#define COMPRESS_ZSTD_H
+
+#include "compress_io.h"
+
+extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec);
+extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec);
+
+#endif /* COMPRESS_ZSTD_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 0da476a4c34..334d449091d 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -5,6 +5,7 @@ pg_dump_common_sources = files(
'compress_io.c',
'compress_lz4.c',
'compress_none.c',
+ 'compress_zstd.c',
'dumputils.c',
'parallel.c',
'pg_backup_archiver.c',
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 61ebb8fe85d..6e97d1f5894 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -2075,7 +2075,7 @@ _discoverArchiveFormat(ArchiveHandle *AH)
/*
* Check if the specified archive is a directory. If so, check if
- * there's a "toc.dat" (or "toc.dat.{gz,lz4}") file in it.
+ * there's a "toc.dat" (or "toc.dat.{gz,lz4,zst}") file in it.
*/
if (stat(AH->fSpec, &st) == 0 && S_ISDIR(st.st_mode))
{
@@ -2086,10 +2086,17 @@ _discoverArchiveFormat(ArchiveHandle *AH)
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.gz"))
return AH->format;
#endif
+
#ifdef USE_LZ4
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.lz4"))
return AH->format;
#endif
+
+#ifdef USE_ZSTD
+ if (_fileExistsInDirectory(AH->fSpec, "toc.dat.zst"))
+ return AH->format;
+#endif
+
pg_fatal("directory \"%s\" does not appear to be a valid archive (\"toc.dat\" does not exist)",
AH->fSpec);
fh = NULL; /* keep compiler quiet */
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 41c2b733e3e..29845340859 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -785,6 +785,8 @@ _PrepParallelRestore(ArchiveHandle *AH)
strlcat(fname, ".gz", sizeof(fname));
else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4)
strlcat(fname, ".lz4", sizeof(fname));
+ else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ strlcat(fname, ".zst", sizeof(fname));
if (stat(fname, &st) == 0)
te->dataLength = st.st_size;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 24ba936332d..72f6126a1fb 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -710,19 +710,6 @@ main(int argc, char **argv)
pg_fatal("invalid compression specification: %s",
error_detail);
- switch (compression_algorithm)
- {
- case PG_COMPRESSION_NONE:
- /* fallthrough */
- case PG_COMPRESSION_GZIP:
- /* fallthrough */
- case PG_COMPRESSION_LZ4:
- break;
- case PG_COMPRESSION_ZSTD:
- pg_fatal("compression with %s is not yet supported", "ZSTD");
- break;
- }
-
/*
* Custom and directory formats are compressed by default with gzip when
* available, not the others.
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 72b19ee6cde..160cd1124a2 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -213,6 +213,77 @@ my %pgdump_runs = (
},
},
+ compression_zstd_custom => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=custom',
+ '--compress=zstd', "--file=$tempdir/compression_zstd_custom.dump",
+ 'postgres',
+ ],
+ restore_cmd => [
+ 'pg_restore',
+ "--file=$tempdir/compression_zstd_custom.sql",
+ "$tempdir/compression_zstd_custom.dump",
+ ],
+ command_like => {
+ command => [
+ 'pg_restore',
+ '-l', "$tempdir/compression_zstd_custom.dump",
+ ],
+ expected => qr/Compression: zstd/,
+ name => 'data content is zstd compressed'
+ },
+ },
+
+ compression_zstd_dir => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--jobs=2',
+ '--format=directory', '--compress=zstd:1',
+ "--file=$tempdir/compression_zstd_dir", 'postgres',
+ ],
+ # Give coverage for manually compressed blob.toc files during
+ # restore.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-z', '-f', '--rm',
+ "$tempdir/compression_zstd_dir/blobs.toc",
+ "$tempdir/compression_zstd_dir/blobs.toc.zst",
+ ],
+ },
+ # Verify that data files were compressed
+ glob_patterns => [
+ "$tempdir/compression_zstd_dir/toc.dat",
+ "$tempdir/compression_zstd_dir/*.dat.zst",
+ ],
+ restore_cmd => [
+ 'pg_restore', '--jobs=2',
+ "--file=$tempdir/compression_zstd_dir.sql",
+ "$tempdir/compression_zstd_dir",
+ ],
+ },
+
+ compression_zstd_plain => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=plain', '--compress=zstd',
+ "--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
+ ],
+ # Decompress the generated file to run through the tests.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-d', '-f',
+ "$tempdir/compression_zstd_plain.sql.zst",
+ "$tempdir/compression_zstd_plain.sql",
+ ],
+ },
+ },
+
clean => {
dump_cmd => [
'pg_dump',
--
2.34.1
[text/x-diff] 0002-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch (2.6K, ../../20230224191840.GD1653@telsasoft.com/3-0002-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch)
download | inline diff:
From 6121c2bf52253c016c1222e56dbff86a2d2bddc5 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 4 Jan 2023 21:21:53 -0600
Subject: [PATCH 2/3] TMP: pg_dump: use Zstd by default, for CI only
---
src/bin/pg_dump/pg_dump.c | 4 ++--
src/bin/pg_dump/t/002_pg_dump.pl | 15 ++++++++-------
2 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 72f6126a1fb..0c61194f960 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -717,8 +717,8 @@ main(int argc, char **argv)
if ((archiveFormat == archCustom || archiveFormat == archDirectory) &&
!user_compression_defined)
{
-#ifdef HAVE_LIBZ
- parse_compress_specification(PG_COMPRESSION_GZIP, NULL,
+#ifdef USE_ZSTD
+ parse_compress_specification(PG_COMPRESSION_ZSTD, NULL,
&compression_spec);
#else
/* Nothing to do in the default case */
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 160cd1124a2..a69c87fca25 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -54,8 +54,9 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir;
# those lines) to validate that part of the process.
my $supports_icu = ($ENV{with_icu} eq 'yes');
-my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1");
+my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
+my $supports_zstd = check_pg_config("#define USE_ZSTD 1");
my %pgdump_runs = (
binary_upgrade => {
@@ -383,10 +384,10 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_custom_format.dump", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
- name => 'data content is gzip-compressed by default if available',
+ name => 'data content is zstd-compressed by default if available',
},
},
@@ -408,8 +409,8 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_dir_format", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
name => 'data content is gzip-compressed by default',
},
@@ -417,7 +418,7 @@ my %pgdump_runs = (
"$tempdir/defaults_dir_format/toc.dat",
"$tempdir/defaults_dir_format/blobs.toc",
$supports_gzip ?
- "$tempdir/defaults_dir_format/*.dat.gz" :
+ "$tempdir/defaults_dir_format/*.dat.zst" :
"$tempdir/defaults_dir_format/*.dat",
],
},
--
2.34.1
[text/x-diff] 0003-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch (13.3K, ../../20230224191840.GD1653@telsasoft.com/4-0003-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch)
download | inline diff:
From 854db16b74f527b096cff646b5311c9a84029265 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 27 Mar 2022 11:55:01 -0500
Subject: [PATCH 3/3] zstd: support long distance mode in pg_dump/basebackup
First proposed here:
20220327205020.GM28503@telsasoft.com
ci-os-only: freebsd
---
doc/src/sgml/protocol.sgml | 10 +++-
doc/src/sgml/ref/pg_basebackup.sgml | 4 +-
doc/src/sgml/ref/pg_dump.sgml | 8 ++-
src/backend/backup/basebackup_zstd.c | 12 ++++
src/bin/pg_basebackup/bbstreamer_zstd.c | 13 +++++
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 9 ++-
src/bin/pg_dump/compress_zstd.c | 5 ++
src/bin/pg_dump/t/002_pg_dump.pl | 3 +-
src/bin/pg_verifybackup/t/008_untar.pl | 8 +++
src/bin/pg_verifybackup/t/010_client_untar.pl | 8 +++
src/common/compression.c | 57 ++++++++++++++++++-
src/include/common/compression.h | 2 +
12 files changed, 131 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 93fc7167d4a..ee8bb7c2445 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2747,7 +2747,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
level. Otherwise, it should be a comma-separated list of items,
each of the form <replaceable>keyword</replaceable> or
<replaceable>keyword=value</replaceable>. Currently, the supported
- keywords are <literal>level</literal> and <literal>workers</literal>.
+ keywords are <literal>level</literal>, <literal>long</literal> and
+ <literal>workers</literal>.
</para>
<para>
@@ -2764,6 +2765,13 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<literal>3</literal>).
</para>
+ <para>
+ The <literal>long</literal> keyword enables long-distance matching
+ mode, for improved compression ratio, at the expense of higher memory
+ use. Long-distance mode is supported only for
+ <literal>zstd</literal>.
+ </para>
+
<para>
The <literal>workers</literal> keyword sets the number of threads
that should be used for parallel compression. Parallel compression
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index db3ad9cd5eb..79d3e657c32 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -424,8 +424,8 @@ PostgreSQL documentation
level. Otherwise, it should be a comma-separated list of items,
each of the form <literal>keyword</literal> or
<literal>keyword=value</literal>.
- Currently, the supported keywords are <literal>level</literal>
- and <literal>workers</literal>.
+ Currently, the supported keywords are <literal>level</literal>,
+ <literal>long</literal>, and <literal>workers</literal>.
The detail string cannot be used when the compression method
is specified as a plain integer.
</para>
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 49d218905fb..46eb219812f 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -676,8 +676,12 @@ PostgreSQL documentation
individual table-data segments, and the default is to compress using
<literal>gzip</literal> at a moderate level. For plain text output,
setting a nonzero compression level causes the entire output file to be compressed,
- as though it had been fed through <application>gzip</application> or
- <application>lz4</application>; but the default is not to compress.
+ as though it had been fed through <application>gzip</application>, or
+ <application>lz4</application>; or <application>zstd</application>,
+ but the default is not to compress.
+ With zstd compression, <literal>long</literal> mode may allow dumps
+ to be significantly smaller, but it might not reduce the size of
+ custom or directory format dumps, whose fields are separately compressed.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/backend/backup/basebackup_zstd.c b/src/backend/backup/basebackup_zstd.c
index ac6cac178a0..1bb5820c884 100644
--- a/src/backend/backup/basebackup_zstd.c
+++ b/src/backend/backup/basebackup_zstd.c
@@ -118,6 +118,18 @@ bbsink_zstd_begin_backup(bbsink *sink)
compress->workers, ZSTD_getErrorName(ret)));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(mysink->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret)));
+ }
+
/*
* We need our own buffer, because we're going to pass different data to
* the next sink than what gets passed to us.
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index fe17d6df4ef..fba391e2a0f 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -106,6 +106,19 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, pg_compress_specification *comp
compress->workers, ZSTD_getErrorName(ret));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(streamer->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ {
+ pg_log_error("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret));
+ exit(1);
+ }
+ }
+
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
streamer->zstd_outBuf.size = streamer->base.bbs_buffer.maxlen;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index b60cb78a0d5..4d130a7f944 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -139,7 +139,14 @@ SKIP:
'gzip:workers=3',
'invalid compression specification: compression algorithm "gzip" does not accept a worker count',
'failure on worker count for gzip'
- ],);
+ ],
+ [
+ 'gzip:long',
+ 'invalid compression specification: compression algorithm "gzip" does not support long-distance mode',
+ 'failure on long mode for gzip'
+ ],
+ );
+
for my $cft (@compression_failure_tests)
{
my $cfail = quotemeta($client_fails . $cft->[1]);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
index 659d08533ae..e32dd73da64 100644
--- a/src/bin/pg_dump/compress_zstd.c
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -67,6 +67,11 @@ ZstdCStreamParams(pg_compress_specification compress)
ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
compress.workers);
+ if (compress.options & PG_COMPRESSION_OPTION_LONG_DISTANCE)
+ ZSTD_CCtx_setParam_or_die(cstream,
+ ZSTD_c_enableLongDistanceMatching,
+ compress.long_distance);
+
#if 0
if (compress.options & PG_COMPRESSION_OPTION_CHECKSUM)
ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_checksumFlag,
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index a69c87fca25..a09dc205188 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -267,11 +267,12 @@ my %pgdump_runs = (
],
},
+ # Exercise long mode for test coverage
compression_zstd_plain => {
test_key => 'compression',
compile_option => 'zstd',
dump_cmd => [
- 'pg_dump', '--format=plain', '--compress=zstd',
+ 'pg_dump', '--format=plain', '--compress=zstd:long',
"--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
],
# Decompress the generated file to run through the tests.
diff --git a/src/bin/pg_verifybackup/t/008_untar.pl b/src/bin/pg_verifybackup/t/008_untar.pl
index 3007bbe8556..05754bc8ec7 100644
--- a/src/bin/pg_verifybackup/t/008_untar.pl
+++ b/src/bin/pg_verifybackup/t/008_untar.pl
@@ -49,6 +49,14 @@ my @test_configuration = (
'decompress_program' => $ENV{'ZSTD'},
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => [ '--compress', 'server-zstd:level=1,long' ],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => ['-d'],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
});
for my $tc (@test_configuration)
diff --git a/src/bin/pg_verifybackup/t/010_client_untar.pl b/src/bin/pg_verifybackup/t/010_client_untar.pl
index f3aa0f59e29..ac51a174d14 100644
--- a/src/bin/pg_verifybackup/t/010_client_untar.pl
+++ b/src/bin/pg_verifybackup/t/010_client_untar.pl
@@ -50,6 +50,14 @@ my @test_configuration = (
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
},
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => ['--compress', 'client-zstd:level=1,long'],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => [ '-d' ],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
{
'compression_method' => 'parallel zstd',
'backup_flags' => [ '--compress', 'client-zstd:workers=3' ],
diff --git a/src/common/compression.c b/src/common/compression.c
index 2d3e56b4d62..713a77c292d 100644
--- a/src/common/compression.c
+++ b/src/common/compression.c
@@ -12,7 +12,7 @@
* Otherwise, a compression specification is a comma-separated list of items,
* each having the form keyword or keyword=value.
*
- * Currently, the only supported keywords are "level" and "workers".
+ * Currently, the supported keywords are "level", "long", and "workers".
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
*
@@ -38,6 +38,8 @@
static int expect_integer_value(char *keyword, char *value,
pg_compress_specification *result);
+static bool expect_boolean_value(char *keyword, char *value,
+ pg_compress_specification *result);
/*
* Look up a compression algorithm by name. Returns true and sets *algorithm
@@ -232,6 +234,11 @@ parse_compress_specification(pg_compress_algorithm algorithm, char *specificatio
result->workers = expect_integer_value(keyword, value, result);
result->options |= PG_COMPRESSION_OPTION_WORKERS;
}
+ else if (strcmp(keyword, "long") == 0)
+ {
+ result->long_distance = expect_boolean_value(keyword, value, result);
+ result->options |= PG_COMPRESSION_OPTION_LONG_DISTANCE;
+ }
else
result->parse_error =
psprintf(_("unrecognized compression option: \"%s\""), keyword);
@@ -289,6 +296,43 @@ expect_integer_value(char *keyword, char *value, pg_compress_specification *resu
return ivalue;
}
+/*
+ * Parse 'value' as an boolean and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1. The caller must check result->parse_error to determine if
+ * the call was successful.
+ *
+ * Valid values are: yes, no, on, off, 1, 0.
+ *
+ * Inspired by ParseVariableBool().
+ */
+static bool
+expect_boolean_value(char *keyword, char *value, pg_compress_specification *result)
+{
+ if (value == NULL)
+ return true;
+
+ if (pg_strcasecmp(value, "yes") == 0)
+ return true;
+ if (pg_strcasecmp(value, "on") == 0)
+ return true;
+ if (pg_strcasecmp(value, "1") == 0)
+ return true;
+
+ if (pg_strcasecmp(value, "no") == 0)
+ return false;
+ if (pg_strcasecmp(value, "off") == 0)
+ return false;
+ if (pg_strcasecmp(value, "0") == 0)
+ return false;
+
+ result->parse_error =
+ psprintf(_("value for compression option \"%s\" must be a boolean"),
+ keyword);
+ return false;
+}
+
/*
* Returns NULL if the compression specification string was syntactically
* valid and semantically sensible. Otherwise, returns an error message.
@@ -354,6 +398,17 @@ validate_compress_specification(pg_compress_specification *spec)
get_compress_algorithm_name(spec->algorithm));
}
+ /*
+ * Of the compression algorithms that we currently support, only zstd
+ * supports long-distance mode.
+ */
+ if ((spec->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0 &&
+ (spec->algorithm != PG_COMPRESSION_ZSTD))
+ {
+ return psprintf(_("compression algorithm \"%s\" does not support long-distance mode"),
+ get_compress_algorithm_name(spec->algorithm));
+ }
+
return NULL;
}
diff --git a/src/include/common/compression.h b/src/include/common/compression.h
index b48c173022e..6cf8cf396a8 100644
--- a/src/include/common/compression.h
+++ b/src/include/common/compression.h
@@ -27,6 +27,7 @@ typedef enum pg_compress_algorithm
} pg_compress_algorithm;
#define PG_COMPRESSION_OPTION_WORKERS (1 << 0)
+#define PG_COMPRESSION_OPTION_LONG_DISTANCE (1 << 1)
typedef struct pg_compress_specification
{
@@ -34,6 +35,7 @@ typedef struct pg_compress_specification
unsigned options; /* OR of PG_COMPRESSION_OPTION constants */
int level;
int workers;
+ int long_distance;
char *parse_error; /* NULL if parsing was OK, else message */
} pg_compress_specification;
--
2.34.1
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-02-25 04:44 ` Michael Paquier <michael@paquier.xyz>
2023-02-25 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2 siblings, 1 reply; 46+ messages in thread
From: Michael Paquier @ 2023-02-25 04:44 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Fri, Feb 24, 2023 at 01:18:40PM -0600, Justin Pryzby wrote:
> This is a draft patch - review is welcome and would help to get this
> ready to be considererd for v16, if desired.
>
> I'm going to add this thread to the old CF entry.
> https://commitfest.postgresql.org/31/2888/
Patch 0003 adds support for the --long option of zstd, meaning that it
"enables long distance matching with #windowLog". What's the benefit
of that when it is applied to dumps and base backup contents?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../Y%2FmSNGOZGUTHkhec@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-25 04:44 ` Re: zstd compression for pg_dump Michael Paquier <michael@paquier.xyz>
@ 2023-02-25 04:50 ` Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 0 replies; 46+ messages in thread
From: Justin Pryzby @ 2023-02-25 04:50 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Sat, Feb 25, 2023 at 01:44:36PM +0900, Michael Paquier wrote:
> On Fri, Feb 24, 2023 at 01:18:40PM -0600, Justin Pryzby wrote:
> > This is a draft patch - review is welcome and would help to get this
> > ready to be considererd for v16, if desired.
> >
> > I'm going to add this thread to the old CF entry.
> > https://commitfest.postgresql.org/31/2888/
>
> Patch 0003 adds support for the --long option of zstd, meaning that it
> "enables long distance matching with #windowLog". What's the benefit
> of that when it is applied to dumps and base backup contents?
It (can) makes it smaller.
+ The <literal>long</literal> keyword enables long-distance matching
+ mode, for improved compression ratio, at the expense of higher memory
+ use. Long-distance mode is supported only for
+ With zstd compression, <literal>long</literal> mode may allow dumps
+ to be significantly smaller, but it might not reduce the size of
+ custom or directory format dumps, whose fields are separately compressed.
Note that I included that here as 003, but I also have an pre-existing
patch for adding that just to basebackup.
--
Justin
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-02-25 10:31 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-02-25 14:47 ` Re: zstd compression for pg_dump Euler Taveira <euler@eulerto.com>
2 siblings, 1 reply; 46+ messages in thread
From: Tomas Vondra @ 2023-02-25 10:31 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; pgsql-hackers; +Cc: gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On 2/24/23 20:18, Justin Pryzby wrote:
> This is a draft patch - review is welcome and would help to get this
> ready to be considererd for v16, if desired.
>
> I'm going to add this thread to the old CF entry.
> https://commitfest.postgresql.org/31/2888/
>
Thanks. Sadly cfbot is unhappy - the windows and cplusplus builds failed
because of some issue in pg_backup_archiver.h. But it's a bit bizarre
because the patch does not modify that file at all ...
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-25 10:31 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2023-02-25 14:47 ` Euler Taveira <euler@eulerto.com>
0 siblings, 0 replies; 46+ messages in thread
From: Euler Taveira @ 2023-02-25 14:47 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; Justin Pryzby <pryzby@telsasoft.com>; pgsql-hackers; +Cc: gkokolatos@pm.me, "Michael Paquier" <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Sat, Feb 25, 2023, at 7:31 AM, Tomas Vondra wrote:
> On 2/24/23 20:18, Justin Pryzby wrote:
> > This is a draft patch - review is welcome and would help to get this
> > ready to be considererd for v16, if desired.
> >
> > I'm going to add this thread to the old CF entry.
> > https://commitfest.postgresql.org/31/2888/
> >
>
> Thanks. Sadly cfbot is unhappy - the windows and cplusplus builds failed
> because of some issue in pg_backup_archiver.h. But it's a bit bizarre
> because the patch does not modify that file at all ...
cpluspluscheck says
# pg_dump is not C++-clean because it uses "public" and "namespace"
# as field names, which is unfortunate but we won't change it now.
Hence, the patch should exclude the new header file from it.
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -153,6 +153,7 @@ do
test "$f" = src/bin/pg_dump/compress_gzip.h && continue
test "$f" = src/bin/pg_dump/compress_io.h && continue
test "$f" = src/bin/pg_dump/compress_lz4.h && continue
+ test "$f" = src/bin/pg_dump/compress_zstd.h && continue
test "$f" = src/bin/pg_dump/compress_none.h && continue
test "$f" = src/bin/pg_dump/parallel.h && continue
test "$f" = src/bin/pg_dump/pg_backup_archiver.h && continue
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-02-26 01:22 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-05 17:47 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2 siblings, 2 replies; 46+ messages in thread
From: Justin Pryzby @ 2023-02-26 01:22 UTC (permalink / raw)
To: pgsql-hackers; +Cc: gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Fri, Feb 24, 2023 at 01:18:40PM -0600, Justin Pryzby wrote:
> This is a draft patch - review is welcome and would help to get this
> ready to be considererd for v16, if desired.
>
> I'm going to add this thread to the old CF entry.
> https://commitfest.postgresql.org/31/2888/
This resolves cfbot warnings: windows and cppcheck.
And refactors zstd routines.
And updates docs.
And includes some fixes for earlier patches that these patches conflicts
with/depends on.
Attachments:
[text/x-diff] 0001-f-fixes-for-LZ4.patch (6.9K, ../../20230226012227.GK1653@telsasoft.com/2-0001-f-fixes-for-LZ4.patch)
download | inline diff:
From ea9b67d09fe1b51e7946cf34fca5795e57dd3858 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 24 Feb 2023 14:07:09 -0600
Subject: [PATCH 1/4] f!fixes for LZ4
---
doc/src/sgml/ref/pg_dump.sgml | 4 ++--
src/bin/pg_dump/compress_gzip.c | 8 --------
src/bin/pg_dump/compress_io.c | 2 +-
src/bin/pg_dump/compress_io.h | 4 ++--
src/bin/pg_dump/compress_lz4.c | 4 ++--
src/bin/pg_dump/pg_backup_archiver.c | 4 ++--
src/bin/pg_dump/pg_dump.c | 2 --
src/bin/pg_dump/t/002_pg_dump.pl | 6 +++---
8 files changed, 12 insertions(+), 22 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 49d218905fb..6fbe49f7ede 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -331,7 +331,7 @@ PostgreSQL documentation
can read. A directory format archive can be manipulated with
standard Unix tools; for example, files in an uncompressed archive
can be compressed with the <application>gzip</application> or
- <application>lz4</application>tool.
+ <application>lz4</application> tools.
This format is compressed by default using <literal>gzip</literal>
and also supports parallel dumps.
</para>
@@ -655,7 +655,7 @@ PostgreSQL documentation
<para>
Specify the compression method and/or the compression level to use.
The compression method can be set to <literal>gzip</literal> or
- <literal>lz4</literal> or <literal>none</literal> for no compression.
+ <literal>lz4</literal>, or <literal>none</literal> for no compression.
A compression detail string can optionally be specified. If the
detail string is an integer, it specifies the compression level.
Otherwise, it should be a comma-separated list of items, each of the
diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c
index 0af65afeb4e..52f41c2e58c 100644
--- a/src/bin/pg_dump/compress_gzip.c
+++ b/src/bin/pg_dump/compress_gzip.c
@@ -123,17 +123,9 @@ WriteDataToArchiveGzip(ArchiveHandle *AH, CompressorState *cs,
gzipcs->outbuf = pg_malloc(ZLIB_OUT_SIZE + 1);
gzipcs->outsize = ZLIB_OUT_SIZE;
- /*
- * A level of zero simply copies the input one block at the time. This
- * is probably not what the user wanted when calling this interface.
- */
- if (cs->compression_spec.level == 0)
- pg_fatal("requested to compress the archive yet no level was specified");
-
if (deflateInit(zp, cs->compression_spec.level) != Z_OK)
pg_fatal("could not initialize compression library: %s", zp->msg);
- /* Just be paranoid - maybe End is called after Start, with no Write */
zp->next_out = gzipcs->outbuf;
zp->avail_out = gzipcs->outsize;
}
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index ce06f1eac9c..9239dbb2755 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -83,7 +83,7 @@
* used by the caller in an error message.
*/
char *
-supports_compression(const pg_compress_specification compression_spec)
+pgdump_supports_compression(const pg_compress_specification compression_spec)
{
const pg_compress_algorithm algorithm = compression_spec.algorithm;
bool supported = false;
diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h
index bbde2693915..46815fa2ebe 100644
--- a/src/bin/pg_dump/compress_io.h
+++ b/src/bin/pg_dump/compress_io.h
@@ -21,7 +21,7 @@
#define ZLIB_OUT_SIZE 4096
#define ZLIB_IN_SIZE 4096
-extern char *supports_compression(const pg_compress_specification compression_spec);
+extern char *pgdump_supports_compression(const pg_compress_specification compression_spec);
/*
* Prototype for callback function used in writeData()
@@ -172,7 +172,7 @@ struct CompressFileHandle
extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec);
/*
- * Initialize a compress file stream. Deffer the compression algorithm
+ * Initialize a compress file stream. Infer the compression algorithm
* from 'path', either by examining its suffix or by appending the supported
* suffixes in 'path'.
*/
diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c
index fe1014e6e77..63e794cdc68 100644
--- a/src/bin/pg_dump/compress_lz4.c
+++ b/src/bin/pg_dump/compress_lz4.c
@@ -161,8 +161,8 @@ typedef struct LZ4File
} LZ4File;
/*
- * LZ4 equivalent to feof() or gzeof(). The end of file is reached if there
- * is no decompressed output in the overflow buffer and the end of the file
+ * LZ4 equivalent to feof() or gzeof(). Return true iff there is no
+ * decompressed output in the overflow buffer and the end of the backing file
* is reached.
*/
static int
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 61ebb8fe85d..2063d6f239d 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -388,7 +388,7 @@ RestoreArchive(Archive *AHX)
{
if (te->hadDumper && (te->reqs & REQ_DATA) != 0)
{
- char *errmsg = supports_compression(AH->compression_spec);
+ char *errmsg = pgdump_supports_compression(AH->compression_spec);
if (errmsg)
pg_fatal("cannot restore from compressed archive (%s)",
errmsg);
@@ -3745,7 +3745,7 @@ ReadHead(ArchiveHandle *AH)
else
AH->compression_spec.algorithm = PG_COMPRESSION_GZIP;
- errmsg = supports_compression(AH->compression_spec);
+ errmsg = pgdump_supports_compression(AH->compression_spec);
if (errmsg)
{
pg_log_warning("archive is compressed, but this installation does not support compression (%s) -- no data will be available",
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 24ba936332d..ce2242195f3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -733,8 +733,6 @@ main(int argc, char **argv)
#ifdef HAVE_LIBZ
parse_compress_specification(PG_COMPRESSION_GZIP, NULL,
&compression_spec);
-#else
- /* Nothing to do in the default case */
#endif
}
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 72b19ee6cde..ad7bc5c194b 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -4248,10 +4248,10 @@ foreach my $run (sort keys %pgdump_runs)
my $test_key = $run;
my $run_db = 'postgres';
- # Skip command-level tests for gzip if there is no support for it.
+ # Skip command-level tests for gzip/lz4 if they're not supported.
if ($pgdump_runs{$run}->{compile_option} &&
- ($pgdump_runs{$run}->{compile_option} eq 'gzip' && !$supports_gzip) ||
- ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4))
+ (($pgdump_runs{$run}->{compile_option} eq 'gzip' && !$supports_gzip) ||
+ ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4)))
{
note "$run: skipped due to no $pgdump_runs{$run}->{compile_option} support";
next;
--
2.34.1
[text/x-diff] 0002-pg_dump-zstd-compression.patch (26.6K, ../../20230226012227.GK1653@telsasoft.com/3-0002-pg_dump-zstd-compression.patch)
download | inline diff:
From d9eca9692f11650be1bfb5635ae833e9ba94a43a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 7 Jan 2023 15:45:06 -0600
Subject: [PATCH 2/4] pg_dump: zstd compression
Previously proposed at: 20201221194924.GI30237@telsasoft.com
---
doc/src/sgml/ref/pg_dump.sgml | 10 +-
src/bin/pg_dump/Makefile | 2 +
src/bin/pg_dump/compress_io.c | 53 +--
src/bin/pg_dump/compress_zstd.c | 505 ++++++++++++++++++++++++++
src/bin/pg_dump/compress_zstd.h | 9 +
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/pg_backup_archiver.c | 9 +-
src/bin/pg_dump/pg_backup_directory.c | 2 +
src/bin/pg_dump/pg_dump.c | 13 -
src/bin/pg_dump/t/002_pg_dump.pl | 79 +++-
src/tools/pginclude/cpluspluscheck | 1 +
11 files changed, 637 insertions(+), 47 deletions(-)
create mode 100644 src/bin/pg_dump/compress_zstd.c
create mode 100644 src/bin/pg_dump/compress_zstd.h
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 6fbe49f7ede..0df86a245f0 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -330,8 +330,9 @@ PostgreSQL documentation
machine-readable format that <application>pg_restore</application>
can read. A directory format archive can be manipulated with
standard Unix tools; for example, files in an uncompressed archive
- can be compressed with the <application>gzip</application> or
- <application>lz4</application> tools.
+ can be compressed with the <application>gzip</application>,
+ <application>lz4</application>, or
+ <application>zstd</application> tools.
This format is compressed by default using <literal>gzip</literal>
and also supports parallel dumps.
</para>
@@ -654,8 +655,9 @@ PostgreSQL documentation
<listitem>
<para>
Specify the compression method and/or the compression level to use.
- The compression method can be set to <literal>gzip</literal> or
- <literal>lz4</literal>, or <literal>none</literal> for no compression.
+ The compression method can be set to <literal>gzip</literal>,
+ <literal>lz4</literal> or <literal>zstd</literal>,
+ or <literal>none</literal> for no compression.
A compression detail string can optionally be specified. If the
detail string is an integer, it specifies the compression level.
Otherwise, it should be a comma-separated list of items, each of the
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index eb8f59459a1..bf540fee2ba 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -18,6 +18,7 @@ include $(top_builddir)/src/Makefile.global
export GZIP_PROGRAM=$(GZIP)
export LZ4
+#XXX export ZSTD
export with_icu
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
@@ -29,6 +30,7 @@ OBJS = \
compress_io.o \
compress_lz4.o \
compress_none.o \
+ compress_zstd.o \
dumputils.o \
parallel.o \
pg_backup_archiver.o \
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 9239dbb2755..121c69ed48f 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -52,8 +52,8 @@
*
* InitDiscoverCompressFileHandle tries to infer the compression by the
* filename suffix. If the suffix is not yet known then it tries to simply
- * open the file and if it fails, it tries to open the same file with the .gz
- * suffix, and then again with the .lz4 suffix.
+ * open the file and if it fails, it tries to open the same file with
+ * compressed suffixes.
*
* IDENTIFICATION
* src/bin/pg_dump/compress_io.c
@@ -69,6 +69,7 @@
#include "compress_io.h"
#include "compress_lz4.h"
#include "compress_none.h"
+#include "compress_zstd.h"
#include "pg_backup_utils.h"
/*----------------------
@@ -98,6 +99,10 @@ pgdump_supports_compression(const pg_compress_specification compression_spec)
if (algorithm == PG_COMPRESSION_LZ4)
supported = true;
#endif
+#ifdef USE_ZSTD
+ if (algorithm == PG_COMPRESSION_ZSTD)
+ supported = true;
+#endif
if (!supported)
return psprintf("this build does not support compression with %s",
@@ -130,6 +135,8 @@ AllocateCompressor(const pg_compress_specification compression_spec,
InitCompressorGzip(cs, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressorLZ4(cs, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressorZstd(cs, compression_spec);
return cs;
}
@@ -196,20 +203,30 @@ InitCompressFileHandle(const pg_compress_specification compression_spec)
InitCompressFileHandleGzip(CFH, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressFileHandleLZ4(CFH, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressFileHandleZstd(CFH, compression_spec);
return CFH;
}
+static bool
+check_compressed_file(const char *path, char **fname, char *ext)
+{
+ free_keep_errno(*fname);
+ *fname = psprintf("%s.%s", path, ext);
+ return (access(*fname, F_OK) == 0);
+}
+
/*
* Open a file for reading. 'path' is the file to open, and 'mode' should
* be either "r" or "rb".
*
* If the file at 'path' contains the suffix of a supported compression method,
- * currently this includes ".gz" and ".lz4", then this compression will be used
+ * currently this includes ".gz", ".lz4" and ".zst", then this compression will be used
* throughout. Otherwise the compression will be inferred by iteratively trying
* to open the file at 'path', first as is, then by appending known compression
* suffixes. So if you pass "foo" as 'path', this will open either "foo" or
- * "foo.gz" or "foo.lz4", trying in that order.
+ * "foo.{gz,lz4,zst}", trying in that order.
*
* On failure, return NULL with an error code in errno.
*/
@@ -237,28 +254,12 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode)
/* avoid unused warning if it is not built with compression */
if (exists)
compression_spec.algorithm = PG_COMPRESSION_NONE;
-#ifdef HAVE_LIBZ
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.gz", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_GZIP;
- }
-#endif
-#ifdef USE_LZ4
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.lz4", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_LZ4;
- }
-#endif
+ else if (check_compressed_file(path, &fname, "gz"))
+ compression_spec.algorithm = PG_COMPRESSION_GZIP;
+ else if (check_compressed_file(path, &fname, "lz4"))
+ compression_spec.algorithm = PG_COMPRESSION_LZ4;
+ else if (check_compressed_file(path, &fname, "zst"))
+ compression_spec.algorithm = PG_COMPRESSION_ZSTD;
}
CFH = InitCompressFileHandle(compression_spec);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
new file mode 100644
index 00000000000..5d2f2bf2a2d
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -0,0 +1,505 @@
+#include "postgres_fe.h"
+
+#include "pg_backup_utils.h"
+#include "compress_zstd.h"
+
+#ifndef USE_ZSTD
+
+void
+InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+#else
+
+#include <zstd.h>
+
+typedef struct ZstdCompressorState
+{
+ /* This is a normal file to which we read/write compressed data */
+ FILE *fp;
+ /* XXX: use one separate ZSTD_CStream per thread: disable on windows ? */
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+} ZstdCompressorState;
+
+static ZSTD_CStream *ZstdCStreamParams(pg_compress_specification compress);
+static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen);
+static void ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs);
+
+static void
+ZSTD_CCtx_setParam_or_die(ZSTD_CStream * cstream,
+ ZSTD_cParameter param, int value)
+{
+ size_t res;
+
+ res = ZSTD_CCtx_setParameter(cstream, param, value);
+ if (ZSTD_isError(res))
+ pg_fatal("could not set compression parameter: %s",
+ ZSTD_getErrorName(res));
+}
+
+/* Return a compression stream with parameters set per argument */
+static ZSTD_CStream *
+ZstdCStreamParams(pg_compress_specification compress)
+{
+ ZSTD_CStream *cstream;
+
+ cstream = ZSTD_createCStream();
+ if (cstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_compressionLevel,
+ compress.level);
+
+ if (compress.options & PG_COMPRESSION_OPTION_WORKERS)
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
+ compress.workers);
+
+#if 0
+ if (compress.options & PG_COMPRESSION_OPTION_CHECKSUM)
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_checksumFlag,
+ compress.checksum);
+
+ /* Still marked as experimental */
+ if (compress.options & PG_COMPRESSION_OPTION_RSYNCABLE)
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_rsyncable, 1);
+#endif
+
+ return cstream;
+}
+
+/* Helper function for EndCompressorZstd and WriteDataToArchiveZstd */
+static void
+ZstdWriteCommon(ArchiveHandle *AH, CompressorState *cs, bool flush)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ /* Loop while there's any input or until flushed */
+ while (input->pos != input->size || flush)
+ {
+ size_t res;
+
+ res = ZSTD_compressStream2(zstdcs->cstream, output,
+ input, flush ? ZSTD_e_end : ZSTD_e_continue);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ /*
+ * Extra paranoia: avoid zero-length chunks, since a zero length
+ * chunk is the EOF marker in the custom format. This should never
+ * happen but...
+ */
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ output->pos = 0;
+
+ if (res == 0)
+ break;
+ }
+}
+
+void
+EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ if (zstdcs->cstream)
+ {
+ ZstdWriteCommon(AH, cs, true);
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ zstdcs->cstream = NULL;
+ }
+ pg_free(zstdcs);
+}
+
+static void
+WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ if (zstdcs->cstream == NULL)
+ {
+ zstdcs->cstream = ZstdCStreamParams(cs->compression_spec);
+
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size);
+ zstdcs->output.pos = 0;
+ }
+
+ zstdcs->input.src = data;
+ zstdcs->input.size = dLen;
+ zstdcs->input.pos = 0;
+
+ ZstdWriteCommon(AH, cs, false);
+}
+
+/* Read data from a compressed zstd archive */
+static void
+ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ size_t res;
+
+ if (zstdcs->dstream == NULL)
+ {
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ input->size = ZSTD_DStreamInSize();
+ input->src = pg_malloc(input->size);
+
+ output->size = ZSTD_DStreamOutSize();
+ output->dst = pg_malloc(output->size);
+ }
+
+ /* read compressed data */
+ for (;;)
+ {
+ size_t cnt;
+
+ /*
+ * XXX: the buffer can grow, we shouldn't keep resetting it to the
+ * original value..
+ */
+ input->size = ZSTD_DStreamInSize();
+
+ cnt = cs->readF(AH, (char **) unconstify(void **, &input->src), &input->size);
+ input->pos = 0;
+ input->size = cnt;
+
+ if (cnt == 0)
+ break;
+
+ /* Now consume as much as possible */
+ while (input->pos < input->size)
+ {
+ /* decompress */
+ output->pos = 0;
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ /* write to output handle */
+ ((char *) output->dst)[output->pos] = '\0';
+ ahwrite(output->dst, 1, output->pos, AH);
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+ }
+
+ pg_free(unconstify(void *, input->src));
+ pg_free(output->dst);
+}
+
+/* Public routines that support Zstd compressed data I/O */
+void
+InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec)
+{
+ cs->readData = ReadDataFromArchiveZstd;
+ cs->writeData = WriteDataToArchiveZstd;
+ cs->end = EndCompressorZstd;
+
+ cs->compression_spec = compression_spec;
+
+ cs->private_data = pg_malloc0(sizeof(ZstdCompressorState));
+}
+
+/*----------------------
+ * Compressed stream API
+ *----------------------
+ */
+
+static size_t
+Zstd_read(void *ptr, size_t size, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ /* input_size is the allocated size */
+ size_t input_size = ZSTD_DStreamInSize();
+ size_t res,
+ cnt;
+
+ output->size = size;
+ output->dst = ptr;
+ output->pos = 0;
+
+ for (;;)
+ {
+ Assert(input->pos <= input->size);
+ Assert(input->size <= input_size);
+
+ /* If the input is completely consumed, start back at the beginning */
+ if (input->pos == input->size)
+ {
+ /* input->size is size produced by "fread" */
+ input->size = 0;
+ /* input->pos is position consumed by decompress */
+ input->pos = 0;
+ }
+
+ /* read compressed data if we must produce more input */
+ if (input->pos == input->size)
+ {
+ cnt = fread(unconstify(void *, input->src), 1, input_size, zstdcs->fp);
+ input->size = cnt;
+
+ /* If we have no input to consume, we're done */
+ if (cnt == 0)
+ break;
+ }
+
+ Assert(cnt >= 0);
+ Assert(input->size <= input_size);
+
+ while (input->pos < input->size)
+ {
+ /* decompress */
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ if (output->pos == output->size)
+ break; /* No more room for output */
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+
+ if (output->pos == output->size)
+ break; /* We read all the data that fits */
+ }
+
+ return output->pos;
+}
+
+static size_t
+Zstd_write(const void *ptr, size_t size, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t res,
+ cnt;
+
+ input->src = ptr;
+ input->size = size;
+ input->pos = 0;
+
+#if 0
+ ZSTD_CCtx_reset(fp->zstd.cstream, ZSTD_reset_session_only);
+ res = ZSTD_CCtx_setPledgedSrcSize(fp->zstd.cstream, size);
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+#endif
+
+ /* Consume all input, and flush later */
+ while (input->pos != input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_continue);
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ pg_fatal("could not write data: %m");
+ }
+
+ return size;
+}
+
+static int
+Zstd_getc(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ int ret;
+
+ if (CFH->read_func(&ret, 1, CFH) != 1)
+ {
+ if (feof(zstdcs->fp))
+ pg_fatal("could not read from input file: end of file");
+ else
+ pg_fatal("could not read from input file: %m");
+ }
+ return ret;
+}
+
+static char *
+Zstd_gets(char *buf, int len, CompressFileHandle *CFH)
+{
+ int i,
+ res;
+
+ /*
+ * Read one byte at a time until newline or EOF. This is only used to read
+ * the list of LOs, and the I/O is buffered anyway.
+ */
+ for (i = 0; i < len - 1; ++i)
+ {
+ res = CFH->read_func(&buf[i], 1, CFH);
+ if (res != 1)
+ break;
+ if (buf[i] == '\n')
+ {
+ ++i;
+ break;
+ }
+ }
+ buf[i] = '\0';
+ return i > 0 ? buf : NULL;
+}
+
+static int
+Zstd_close(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ int result;
+
+ if (zstdcs->cstream)
+ {
+ size_t res,
+ cnt;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ for (;;)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_end);
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ pg_fatal("could not write data: %m");
+
+ if (res == 0)
+ break;
+ }
+
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ if (zstdcs->dstream)
+ {
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ }
+
+ result = fclose(zstdcs->fp);
+ pg_free(zstdcs);
+ return result;
+}
+
+static int
+Zstd_eof(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ return feof(zstdcs->fp);
+}
+
+static int
+Zstd_open(const char *path, int fd, const char *mode,
+ CompressFileHandle *CFH)
+{
+ FILE *fp;
+ ZstdCompressorState *zstdcs;
+
+ if (fd >= 0)
+ fp = fdopen(fd, mode);
+ else
+ fp = fopen(path, mode);
+
+ if (fp == NULL)
+ return 1;
+
+ CFH->private_data = pg_malloc0(sizeof(ZstdCompressorState));
+ zstdcs = (ZstdCompressorState *) CFH->private_data;
+ zstdcs->fp = fp;
+
+ if (mode[0] == 'w' || mode[0] == 'a')
+ {
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc0(zstdcs->output.size);
+ zstdcs->cstream = ZstdCStreamParams(CFH->compression_spec);
+ }
+ else if (strchr(mode, 'r'))
+ {
+ zstdcs->input.src = pg_malloc0(ZSTD_DStreamInSize());
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+ }
+ /* XXX else: bad mode */
+
+ return 0;
+}
+
+static int
+Zstd_open_write(const char *path, const char *mode, CompressFileHandle *CFH)
+{
+ char fname[MAXPGPATH];
+
+ sprintf(fname, "%s.zst", path);
+ return CFH->open_func(fname, -1, mode, CFH);
+}
+
+static const char *
+Zstd_get_error(CompressFileHandle *CFH)
+{
+#if 0
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ if (ZSTD_isError(res))
+ return ZSTD_getErrorName(res)
+ else
+#endif
+
+ return strerror(errno);
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec)
+{
+ CFH->open_func = Zstd_open;
+ CFH->open_write_func = Zstd_open_write;
+ CFH->read_func = Zstd_read;
+ CFH->write_func = Zstd_write;
+ CFH->gets_func = Zstd_gets;
+ CFH->getc_func = Zstd_getc;
+ CFH->close_func = Zstd_close;
+ CFH->eof_func = Zstd_eof;
+ CFH->get_error_func = Zstd_get_error;
+
+ CFH->compression_spec = compression_spec;
+
+ CFH->private_data = NULL;
+}
+
+#endif /* USE_ZSTD */
diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h
new file mode 100644
index 00000000000..f36698b4c26
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.h
@@ -0,0 +1,9 @@
+#ifndef COMPRESS_ZSTD_H
+#define COMPRESS_ZSTD_H
+
+#include "compress_io.h"
+
+extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec);
+extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec);
+
+#endif /* COMPRESS_ZSTD_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 0da476a4c34..334d449091d 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -5,6 +5,7 @@ pg_dump_common_sources = files(
'compress_io.c',
'compress_lz4.c',
'compress_none.c',
+ 'compress_zstd.c',
'dumputils.c',
'parallel.c',
'pg_backup_archiver.c',
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 2063d6f239d..fc739db4355 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -2075,7 +2075,7 @@ _discoverArchiveFormat(ArchiveHandle *AH)
/*
* Check if the specified archive is a directory. If so, check if
- * there's a "toc.dat" (or "toc.dat.{gz,lz4}") file in it.
+ * there's a "toc.dat" (or "toc.dat.{gz,lz4,zst}") file in it.
*/
if (stat(AH->fSpec, &st) == 0 && S_ISDIR(st.st_mode))
{
@@ -2086,10 +2086,17 @@ _discoverArchiveFormat(ArchiveHandle *AH)
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.gz"))
return AH->format;
#endif
+
#ifdef USE_LZ4
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.lz4"))
return AH->format;
#endif
+
+#ifdef USE_ZSTD
+ if (_fileExistsInDirectory(AH->fSpec, "toc.dat.zst"))
+ return AH->format;
+#endif
+
pg_fatal("directory \"%s\" does not appear to be a valid archive (\"toc.dat\" does not exist)",
AH->fSpec);
fh = NULL; /* keep compiler quiet */
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 41c2b733e3e..29845340859 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -785,6 +785,8 @@ _PrepParallelRestore(ArchiveHandle *AH)
strlcat(fname, ".gz", sizeof(fname));
else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4)
strlcat(fname, ".lz4", sizeof(fname));
+ else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ strlcat(fname, ".zst", sizeof(fname));
if (stat(fname, &st) == 0)
te->dataLength = st.st_size;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index ce2242195f3..6495a88ff99 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -710,19 +710,6 @@ main(int argc, char **argv)
pg_fatal("invalid compression specification: %s",
error_detail);
- switch (compression_algorithm)
- {
- case PG_COMPRESSION_NONE:
- /* fallthrough */
- case PG_COMPRESSION_GZIP:
- /* fallthrough */
- case PG_COMPRESSION_LZ4:
- break;
- case PG_COMPRESSION_ZSTD:
- pg_fatal("compression with %s is not yet supported", "ZSTD");
- break;
- }
-
/*
* Custom and directory formats are compressed by default with gzip when
* available, not the others.
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index ad7bc5c194b..eb5e09f55c8 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -54,8 +54,9 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir;
# those lines) to validate that part of the process.
my $supports_icu = ($ENV{with_icu} eq 'yes');
-my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1");
+my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
+my $supports_zstd = check_pg_config("#define USE_ZSTD 1");
my %pgdump_runs = (
binary_upgrade => {
@@ -213,6 +214,77 @@ my %pgdump_runs = (
},
},
+ compression_zstd_custom => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=custom',
+ '--compress=zstd', "--file=$tempdir/compression_zstd_custom.dump",
+ 'postgres',
+ ],
+ restore_cmd => [
+ 'pg_restore',
+ "--file=$tempdir/compression_zstd_custom.sql",
+ "$tempdir/compression_zstd_custom.dump",
+ ],
+ command_like => {
+ command => [
+ 'pg_restore',
+ '-l', "$tempdir/compression_zstd_custom.dump",
+ ],
+ expected => qr/Compression: zstd/,
+ name => 'data content is zstd compressed'
+ },
+ },
+
+ compression_zstd_dir => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--jobs=2',
+ '--format=directory', '--compress=zstd:1',
+ "--file=$tempdir/compression_zstd_dir", 'postgres',
+ ],
+ # Give coverage for manually compressed blob.toc files during
+ # restore.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-z', '-f', '--rm',
+ "$tempdir/compression_zstd_dir/blobs.toc",
+ "$tempdir/compression_zstd_dir/blobs.toc.zst",
+ ],
+ },
+ # Verify that data files were compressed
+ glob_patterns => [
+ "$tempdir/compression_zstd_dir/toc.dat",
+ "$tempdir/compression_zstd_dir/*.dat.zst",
+ ],
+ restore_cmd => [
+ 'pg_restore', '--jobs=2',
+ "--file=$tempdir/compression_zstd_dir.sql",
+ "$tempdir/compression_zstd_dir",
+ ],
+ },
+
+ compression_zstd_plain => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=plain', '--compress=zstd',
+ "--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
+ ],
+ # Decompress the generated file to run through the tests.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-d', '-f',
+ "$tempdir/compression_zstd_plain.sql.zst",
+ "$tempdir/compression_zstd_plain.sql",
+ ],
+ },
+ },
+
clean => {
dump_cmd => [
'pg_dump',
@@ -4248,10 +4320,11 @@ foreach my $run (sort keys %pgdump_runs)
my $test_key = $run;
my $run_db = 'postgres';
- # Skip command-level tests for gzip/lz4 if they're not supported.
+ # Skip command-level tests for gzip/lz4/zstd if they're not supported.
if ($pgdump_runs{$run}->{compile_option} &&
(($pgdump_runs{$run}->{compile_option} eq 'gzip' && !$supports_gzip) ||
- ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4)))
+ ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4) ||
+ ($pgdump_runs{$run}->{compile_option} eq 'zstd' && !$supports_zstd)))
{
note "$run: skipped due to no $pgdump_runs{$run}->{compile_option} support";
next;
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index 135633fb19b..71d19683817 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -154,6 +154,7 @@ do
test "$f" = src/bin/pg_dump/compress_io.h && continue
test "$f" = src/bin/pg_dump/compress_lz4.h && continue
test "$f" = src/bin/pg_dump/compress_none.h && continue
+ test "$f" = src/bin/pg_dump/compress_zstd.h && continue
test "$f" = src/bin/pg_dump/parallel.h && continue
test "$f" = src/bin/pg_dump/pg_backup_archiver.h && continue
test "$f" = src/bin/pg_dump/pg_dump.h && continue
--
2.34.1
[text/x-diff] 0003-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch (2.2K, ../../20230226012227.GK1653@telsasoft.com/4-0003-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch)
download | inline diff:
From 7b9bff6e458afad750e026a20a7ebf96c7767dde Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 4 Jan 2023 21:21:53 -0600
Subject: [PATCH 3/4] TMP: pg_dump: use Zstd by default, for CI only
ci-os-only: warnings
---
src/bin/pg_dump/pg_dump.c | 4 ++--
src/bin/pg_dump/t/002_pg_dump.pl | 14 +++++++-------
2 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 6495a88ff99..27eb9f4e63d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -717,8 +717,8 @@ main(int argc, char **argv)
if ((archiveFormat == archCustom || archiveFormat == archDirectory) &&
!user_compression_defined)
{
-#ifdef HAVE_LIBZ
- parse_compress_specification(PG_COMPRESSION_GZIP, NULL,
+#ifdef USE_ZSTD
+ parse_compress_specification(PG_COMPRESSION_ZSTD, NULL,
&compression_spec);
#endif
}
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index eb5e09f55c8..8b58507703d 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -384,10 +384,10 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_custom_format.dump", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
- name => 'data content is gzip-compressed by default if available',
+ name => 'data content is zstd-compressed by default if available',
},
},
@@ -409,16 +409,16 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_dir_format", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
name => 'data content is gzip-compressed by default',
},
glob_patterns => [
"$tempdir/defaults_dir_format/toc.dat",
"$tempdir/defaults_dir_format/blobs.toc",
- $supports_gzip ?
- "$tempdir/defaults_dir_format/*.dat.gz" :
+ $supports_zstd ?
+ "$tempdir/defaults_dir_format/*.dat.zst" :
"$tempdir/defaults_dir_format/*.dat",
],
},
--
2.34.1
[text/x-diff] 0004-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch (13.3K, ../../20230226012227.GK1653@telsasoft.com/5-0004-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch)
download | inline diff:
From 2fe26e632f68d7ffdaa977b6110739a85ee33a60 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 27 Mar 2022 11:55:01 -0500
Subject: [PATCH 4/4] zstd: support long distance mode in pg_dump/basebackup
First proposed here:
20220327205020.GM28503@telsasoft.com
//-os-only: freebsd
---
doc/src/sgml/protocol.sgml | 10 +++-
doc/src/sgml/ref/pg_basebackup.sgml | 4 +-
doc/src/sgml/ref/pg_dump.sgml | 8 ++-
src/backend/backup/basebackup_zstd.c | 12 ++++
src/bin/pg_basebackup/bbstreamer_zstd.c | 13 +++++
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 9 ++-
src/bin/pg_dump/compress_zstd.c | 5 ++
src/bin/pg_dump/t/002_pg_dump.pl | 3 +-
src/bin/pg_verifybackup/t/008_untar.pl | 8 +++
src/bin/pg_verifybackup/t/010_client_untar.pl | 8 +++
src/common/compression.c | 57 ++++++++++++++++++-
src/include/common/compression.h | 2 +
12 files changed, 131 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 93fc7167d4a..ee8bb7c2445 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2747,7 +2747,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
level. Otherwise, it should be a comma-separated list of items,
each of the form <replaceable>keyword</replaceable> or
<replaceable>keyword=value</replaceable>. Currently, the supported
- keywords are <literal>level</literal> and <literal>workers</literal>.
+ keywords are <literal>level</literal>, <literal>long</literal> and
+ <literal>workers</literal>.
</para>
<para>
@@ -2764,6 +2765,13 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<literal>3</literal>).
</para>
+ <para>
+ The <literal>long</literal> keyword enables long-distance matching
+ mode, for improved compression ratio, at the expense of higher memory
+ use. Long-distance mode is supported only for
+ <literal>zstd</literal>.
+ </para>
+
<para>
The <literal>workers</literal> keyword sets the number of threads
that should be used for parallel compression. Parallel compression
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index db3ad9cd5eb..79d3e657c32 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -424,8 +424,8 @@ PostgreSQL documentation
level. Otherwise, it should be a comma-separated list of items,
each of the form <literal>keyword</literal> or
<literal>keyword=value</literal>.
- Currently, the supported keywords are <literal>level</literal>
- and <literal>workers</literal>.
+ Currently, the supported keywords are <literal>level</literal>,
+ <literal>long</literal>, and <literal>workers</literal>.
The detail string cannot be used when the compression method
is specified as a plain integer.
</para>
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0df86a245f0..253518b7a41 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -678,8 +678,12 @@ PostgreSQL documentation
individual table-data segments, and the default is to compress using
<literal>gzip</literal> at a moderate level. For plain text output,
setting a nonzero compression level causes the entire output file to be compressed,
- as though it had been fed through <application>gzip</application> or
- <application>lz4</application>; but the default is not to compress.
+ as though it had been fed through <application>gzip</application>, or
+ <application>lz4</application>; or <application>zstd</application>,
+ but the default is not to compress.
+ With zstd compression, <literal>long</literal> mode may allow dumps
+ to be significantly smaller, but it might not reduce the size of
+ custom or directory format dumps, whose fields are separately compressed.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/backend/backup/basebackup_zstd.c b/src/backend/backup/basebackup_zstd.c
index ac6cac178a0..1bb5820c884 100644
--- a/src/backend/backup/basebackup_zstd.c
+++ b/src/backend/backup/basebackup_zstd.c
@@ -118,6 +118,18 @@ bbsink_zstd_begin_backup(bbsink *sink)
compress->workers, ZSTD_getErrorName(ret)));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(mysink->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret)));
+ }
+
/*
* We need our own buffer, because we're going to pass different data to
* the next sink than what gets passed to us.
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index fe17d6df4ef..fba391e2a0f 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -106,6 +106,19 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, pg_compress_specification *comp
compress->workers, ZSTD_getErrorName(ret));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(streamer->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ {
+ pg_log_error("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret));
+ exit(1);
+ }
+ }
+
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
streamer->zstd_outBuf.size = streamer->base.bbs_buffer.maxlen;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index b60cb78a0d5..4d130a7f944 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -139,7 +139,14 @@ SKIP:
'gzip:workers=3',
'invalid compression specification: compression algorithm "gzip" does not accept a worker count',
'failure on worker count for gzip'
- ],);
+ ],
+ [
+ 'gzip:long',
+ 'invalid compression specification: compression algorithm "gzip" does not support long-distance mode',
+ 'failure on long mode for gzip'
+ ],
+ );
+
for my $cft (@compression_failure_tests)
{
my $cfail = quotemeta($client_fails . $cft->[1]);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
index 5d2f2bf2a2d..65c14a7cb67 100644
--- a/src/bin/pg_dump/compress_zstd.c
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -67,6 +67,11 @@ ZstdCStreamParams(pg_compress_specification compress)
ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
compress.workers);
+ if (compress.options & PG_COMPRESSION_OPTION_LONG_DISTANCE)
+ ZSTD_CCtx_setParam_or_die(cstream,
+ ZSTD_c_enableLongDistanceMatching,
+ compress.long_distance);
+
#if 0
if (compress.options & PG_COMPRESSION_OPTION_CHECKSUM)
ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_checksumFlag,
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 8b58507703d..15e8f606a86 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -267,11 +267,12 @@ my %pgdump_runs = (
],
},
+ # Exercise long mode for test coverage
compression_zstd_plain => {
test_key => 'compression',
compile_option => 'zstd',
dump_cmd => [
- 'pg_dump', '--format=plain', '--compress=zstd',
+ 'pg_dump', '--format=plain', '--compress=zstd:long',
"--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
],
# Decompress the generated file to run through the tests.
diff --git a/src/bin/pg_verifybackup/t/008_untar.pl b/src/bin/pg_verifybackup/t/008_untar.pl
index 3007bbe8556..05754bc8ec7 100644
--- a/src/bin/pg_verifybackup/t/008_untar.pl
+++ b/src/bin/pg_verifybackup/t/008_untar.pl
@@ -49,6 +49,14 @@ my @test_configuration = (
'decompress_program' => $ENV{'ZSTD'},
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => [ '--compress', 'server-zstd:level=1,long' ],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => ['-d'],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
});
for my $tc (@test_configuration)
diff --git a/src/bin/pg_verifybackup/t/010_client_untar.pl b/src/bin/pg_verifybackup/t/010_client_untar.pl
index f3aa0f59e29..ac51a174d14 100644
--- a/src/bin/pg_verifybackup/t/010_client_untar.pl
+++ b/src/bin/pg_verifybackup/t/010_client_untar.pl
@@ -50,6 +50,14 @@ my @test_configuration = (
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
},
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => ['--compress', 'client-zstd:level=1,long'],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => [ '-d' ],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
{
'compression_method' => 'parallel zstd',
'backup_flags' => [ '--compress', 'client-zstd:workers=3' ],
diff --git a/src/common/compression.c b/src/common/compression.c
index 2d3e56b4d62..713a77c292d 100644
--- a/src/common/compression.c
+++ b/src/common/compression.c
@@ -12,7 +12,7 @@
* Otherwise, a compression specification is a comma-separated list of items,
* each having the form keyword or keyword=value.
*
- * Currently, the only supported keywords are "level" and "workers".
+ * Currently, the supported keywords are "level", "long", and "workers".
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
*
@@ -38,6 +38,8 @@
static int expect_integer_value(char *keyword, char *value,
pg_compress_specification *result);
+static bool expect_boolean_value(char *keyword, char *value,
+ pg_compress_specification *result);
/*
* Look up a compression algorithm by name. Returns true and sets *algorithm
@@ -232,6 +234,11 @@ parse_compress_specification(pg_compress_algorithm algorithm, char *specificatio
result->workers = expect_integer_value(keyword, value, result);
result->options |= PG_COMPRESSION_OPTION_WORKERS;
}
+ else if (strcmp(keyword, "long") == 0)
+ {
+ result->long_distance = expect_boolean_value(keyword, value, result);
+ result->options |= PG_COMPRESSION_OPTION_LONG_DISTANCE;
+ }
else
result->parse_error =
psprintf(_("unrecognized compression option: \"%s\""), keyword);
@@ -289,6 +296,43 @@ expect_integer_value(char *keyword, char *value, pg_compress_specification *resu
return ivalue;
}
+/*
+ * Parse 'value' as an boolean and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1. The caller must check result->parse_error to determine if
+ * the call was successful.
+ *
+ * Valid values are: yes, no, on, off, 1, 0.
+ *
+ * Inspired by ParseVariableBool().
+ */
+static bool
+expect_boolean_value(char *keyword, char *value, pg_compress_specification *result)
+{
+ if (value == NULL)
+ return true;
+
+ if (pg_strcasecmp(value, "yes") == 0)
+ return true;
+ if (pg_strcasecmp(value, "on") == 0)
+ return true;
+ if (pg_strcasecmp(value, "1") == 0)
+ return true;
+
+ if (pg_strcasecmp(value, "no") == 0)
+ return false;
+ if (pg_strcasecmp(value, "off") == 0)
+ return false;
+ if (pg_strcasecmp(value, "0") == 0)
+ return false;
+
+ result->parse_error =
+ psprintf(_("value for compression option \"%s\" must be a boolean"),
+ keyword);
+ return false;
+}
+
/*
* Returns NULL if the compression specification string was syntactically
* valid and semantically sensible. Otherwise, returns an error message.
@@ -354,6 +398,17 @@ validate_compress_specification(pg_compress_specification *spec)
get_compress_algorithm_name(spec->algorithm));
}
+ /*
+ * Of the compression algorithms that we currently support, only zstd
+ * supports long-distance mode.
+ */
+ if ((spec->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0 &&
+ (spec->algorithm != PG_COMPRESSION_ZSTD))
+ {
+ return psprintf(_("compression algorithm \"%s\" does not support long-distance mode"),
+ get_compress_algorithm_name(spec->algorithm));
+ }
+
return NULL;
}
diff --git a/src/include/common/compression.h b/src/include/common/compression.h
index b48c173022e..6cf8cf396a8 100644
--- a/src/include/common/compression.h
+++ b/src/include/common/compression.h
@@ -27,6 +27,7 @@ typedef enum pg_compress_algorithm
} pg_compress_algorithm;
#define PG_COMPRESSION_OPTION_WORKERS (1 << 0)
+#define PG_COMPRESSION_OPTION_LONG_DISTANCE (1 << 1)
typedef struct pg_compress_specification
{
@@ -34,6 +35,7 @@ typedef struct pg_compress_specification
unsigned options; /* OR of PG_COMPRESSION_OPTION constants */
int level;
int workers;
+ int long_distance;
char *parse_error; /* NULL if parsing was OK, else message */
} pg_compress_specification;
--
2.34.1
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-03-03 18:32 ` Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
1 sibling, 1 reply; 46+ messages in thread
From: Jacob Champion @ 2023-03-03 18:32 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Sat, Feb 25, 2023 at 5:22 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> This resolves cfbot warnings: windows and cppcheck.
> And refactors zstd routines.
> And updates docs.
> And includes some fixes for earlier patches that these patches conflicts
> with/depends on.
This'll need a rebase (cfbot took a while to catch up). The patchset
includes basebackup modifications, which are part of a different CF
entry; was that intended?
I tried this on a local, 3.5GB, mostly-text table (from the UK Price
Paid dataset [1]) and the comparison against the other methods was
impressive. (I'm no good at constructing compression benchmarks, so
this is a super naive setup. Client's on the same laptop as the
server.)
$ time ./src/bin/pg_dump/pg_dump -d postgres -t pp_complete -Z
zstd > /tmp/zstd.dump
real 1m17.632s
user 0m35.521s
sys 0m2.683s
$ time ./\src/bin/pg_dump/pg_dump -d postgres -t pp_complete -Z
lz4 > /tmp/lz4.dump
real 1m13.125s
user 0m19.795s
sys 0m3.370s
$ time ./\src/bin/pg_dump/pg_dump -d postgres -t pp_complete -Z
gzip > /tmp/gzip.dump
real 2m24.523s
user 2m22.114s
sys 0m1.848s
$ ls -l /tmp/*.dump
-rw-rw-r-- 1 jacob jacob 1331493925 Mar 3 09:45 /tmp/gzip.dump
-rw-rw-r-- 1 jacob jacob 2125998939 Mar 3 09:42 /tmp/lz4.dump
-rw-rw-r-- 1 jacob jacob 1215834718 Mar 3 09:40 /tmp/zstd.dump
Default gzip was the only method that bottlenecked on pg_dump rather
than the server, and default zstd outcompressed it at a fraction of
the CPU time. So, naively, this looks really good.
With this particular dataset, I don't see much improvement with
zstd:long. (At nearly double the CPU time, I get a <1% improvement in
compression size.) I assume it's heavily data dependent, but from the
notes on --long [2] it seems like they expect you to play around with
the window size to further tailor it to your data. Does it make sense
to provide the long option without the windowLog parameter?
Thanks,
--Jacob
[1] https://landregistry.data.gov.uk/
[2] https://github.com/facebook/zstd/releases/tag/v1.3.2
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
@ 2023-03-03 18:55 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
0 siblings, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2023-03-03 18:55 UTC (permalink / raw)
To: Jacob Champion <jchampion@timescale.com>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Fri, Mar 03, 2023 at 10:32:53AM -0800, Jacob Champion wrote:
> On Sat, Feb 25, 2023 at 5:22 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> > This resolves cfbot warnings: windows and cppcheck.
> > And refactors zstd routines.
> > And updates docs.
> > And includes some fixes for earlier patches that these patches conflicts
> > with/depends on.
>
> This'll need a rebase (cfbot took a while to catch up).
Soon.
> The patchset includes basebackup modifications, which are part of a
> different CF entry; was that intended?
Yes, it's intentional - if zstd:long mode were to be merged first, then
this patch should include long mode from the start.
Or, if pgdump+zstd were merged first, then long mode could be added to
both places.
> I tried this on a local, 3.5GB, mostly-text table (from the UK Price
Thanks for looking. If your zstd library is compiled with thread
support, could you also try with :workers=N ? I believe this is working
correctly, but I'm going to ask for help verifying that...
It'd be especially useful to test under windows, where pgdump/restore
use threads instead of forking... If you have a windows environment but
not set up for development, I think it's possible to get cirrusci to
compile a patch for you and then retrieve the binaries provided as an
"artifact" (credit/blame for this idea should be directed to Thomas
Munro).
> With this particular dataset, I don't see much improvement with
> zstd:long.
Yeah. I this could be because either 1) you already got very good
comprssion without looking at more data; and/or 2) the neighboring data
is already very similar, maybe equally or more similar, than the further
data, from which there's nothing to gain.
> (At nearly double the CPU time, I get a <1% improvement in
> compression size.) I assume it's heavily data dependent, but from the
> notes on --long [2] it seems like they expect you to play around with
> the window size to further tailor it to your data. Does it make sense
> to provide the long option without the windowLog parameter?
I don't want to start exposing lots of fine-granined parameters at this
point. In the immediate case, it looks like it may require more than
just adding another parameter:
Note: If windowLog is set to larger than 27,
--long=windowLog or --memory=windowSize needs to be passed to the
decompressor.
--
Justin
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-03-03 21:38 ` Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 46+ messages in thread
From: Jacob Champion @ 2023-03-03 21:38 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Fri, Mar 3, 2023 at 10:55 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> Thanks for looking. If your zstd library is compiled with thread
> support, could you also try with :workers=N ? I believe this is working
> correctly, but I'm going to ask for help verifying that...
Unfortunately not (Ubuntu 20.04):
pg_dump: error: could not set compression parameter: Unsupported parameter
But that lets me review the error! I think these error messages should
say which options caused them.
> It'd be especially useful to test under windows, where pgdump/restore
> use threads instead of forking... If you have a windows environment but
> not set up for development, I think it's possible to get cirrusci to
> compile a patch for you and then retrieve the binaries provided as an
> "artifact" (credit/blame for this idea should be directed to Thomas
> Munro).
I should be able to do that next week.
> > With this particular dataset, I don't see much improvement with
> > zstd:long.
>
> Yeah. I this could be because either 1) you already got very good
> comprssion without looking at more data; and/or 2) the neighboring data
> is already very similar, maybe equally or more similar, than the further
> data, from which there's nothing to gain.
What kinds of improvements do you see with your setup? I'm wondering
when we would suggest that people use it.
> I don't want to start exposing lots of fine-granined parameters at this
> point. In the immediate case, it looks like it may require more than
> just adding another parameter:
>
> Note: If windowLog is set to larger than 27,
> --long=windowLog or --memory=windowSize needs to be passed to the
> decompressor.
Hm. That would complicate things.
Thanks,
--Jacob
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
@ 2023-03-04 16:57 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
0 siblings, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2023-03-04 16:57 UTC (permalink / raw)
To: Jacob Champion <jchampion@timescale.com>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Fri, Mar 03, 2023 at 01:38:05PM -0800, Jacob Champion wrote:
> > > With this particular dataset, I don't see much improvement with
> > > zstd:long.
> >
> > Yeah. I this could be because either 1) you already got very good
> > comprssion without looking at more data; and/or 2) the neighboring data
> > is already very similar, maybe equally or more similar, than the further
> > data, from which there's nothing to gain.
>
> What kinds of improvements do you see with your setup? I'm wondering
> when we would suggest that people use it.
On customer data, I see small improvements - below 10%.
But on my first two tries, I made synthetic data sets where it's a lot:
$ ./src/bin/pg_dump/pg_dump -d pryzbyj -Fp -Z zstd:long |wc -c
286107
$ ./src/bin/pg_dump/pg_dump -d pryzbyj -Fp -Z zstd:long=0 |wc -c
1709695
That's just 6 identical tables like:
pryzbyj=# CREATE TABLE t1 AS SELECT generate_series(1,999999);
In this case, "custom" format doesn't see that benefit, because the
greatest similarity is across tables, which don't share compressor
state. But I think the note that I wrote in the docs about that should
be removed - custom format could see a big benefit, as long as the table
is big enough, and there's more similarity/repetition at longer
distances.
Here's one where custom format *does* benefit, due to long-distance
repetition within a single table. The data is contrived, but the schema
of ID => data is not. What's notable isn't how compressible the data
is, but how much *more* compressible it is with long-distance matching.
pryzbyj=# CREATE TABLE t1 AS SELECT i,array_agg(j) FROM generate_series(1,444)i,generate_series(1,99999)j GROUP BY 1;
$ ./src/bin/pg_dump/pg_dump -d pryzbyj -Fc -Z zstd:long=1 |wc -c
82023
$ ./src/bin/pg_dump/pg_dump -d pryzbyj -Fc -Z zstd:long=0 |wc -c
1048267
--
Justin
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-03-08 18:59 ` Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
0 siblings, 1 reply; 46+ messages in thread
From: Jacob Champion @ 2023-03-08 18:59 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Sat, Mar 4, 2023 at 8:57 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> pryzbyj=# CREATE TABLE t1 AS SELECT i,array_agg(j) FROM generate_series(1,444)i,generate_series(1,99999)j GROUP BY 1;
> $ ./src/bin/pg_dump/pg_dump -d pryzbyj -Fc -Z zstd:long=1 |wc -c
> 82023
> $ ./src/bin/pg_dump/pg_dump -d pryzbyj -Fc -Z zstd:long=0 |wc -c
> 1048267
Nice!
I did some smoke testing against zstd's GitHub release on Windows. To
build against it, I had to construct an import library, and put that
and the DLL into the `lib` folder expected by the MSVC scripts...
which makes me wonder if I've chosen a harder way than necessary?
Parallel zstd dumps seem to work as expected, in that the resulting
pg_restore output is identical to uncompressed dumps and nothing
explodes. I haven't inspected the threading implementation for safety
yet, as you mentioned. And I still wasn't able to test :workers, since
it looks like the official libzstd for Windows isn't built for
multithreading. That'll be another day's project.
--Jacob
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
@ 2023-03-10 20:48 ` Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 46+ messages in thread
From: Jacob Champion @ 2023-03-10 20:48 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
Hi,
This'll need another rebase over the meson ICU changes.
On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <jchampion@timescale.com>
wrote:
> I did some smoke testing against zstd's GitHub release on Windows. To
> build against it, I had to construct an import library, and put that
> and the DLL into the `lib` folder expected by the MSVC scripts...
> which makes me wonder if I've chosen a harder way than necessary?
A meson wrap made this much easier! It looks like pg_dump's meson.build
is missing dependencies on zstd (meson couldn't find the headers in the
subproject without them).
> Parallel zstd dumps seem to work as expected, in that the resulting
> pg_restore output is identical to uncompressed dumps and nothing
> explodes. I haven't inspected the threading implementation for safety
> yet, as you mentioned.
Hm. Best I can tell, the CloneArchive() machinery is supposed to be
handling safety for this, by isolating each thread's state. I don't feel
comfortable pronouncing this new addition safe or not, because I'm not
sure I understand what the comments in the format-specific _Clone()
callbacks are saying yet.
> And I still wasn't able to test :workers, since
> it looks like the official libzstd for Windows isn't built for
> multithreading. That'll be another day's project.
The wrapped installation enabled threading too, so I was able to try
with :workers=8. Everything seems to work, but I didn't have a dataset
that showed speed improvements at the time. It did seem to affect the
overall compressibility negatively -- which makes sense, I think,
assuming each thread is looking at a separate and/or smaller window.
On to code (not a complete review):
> if (hasSuffix(fname, ".gz"))
> compression_spec.algorithm = PG_COMPRESSION_GZIP;
> else
> {
> bool exists;
>
> exists = (stat(path, &st) == 0);
> /* avoid unused warning if it is not built with compression */
> if (exists)
> compression_spec.algorithm = PG_COMPRESSION_NONE;
> -#ifdef HAVE_LIBZ
> - if (!exists)
> - {
> - free_keep_errno(fname);
> - fname = psprintf("%s.gz", path);
> - exists = (stat(fname, &st) == 0);
> -
> - if (exists)
> - compression_spec.algorithm = PG_COMPRESSION_GZIP;
> - }
> -#endif
> -#ifdef USE_LZ4
> - if (!exists)
> - {
> - free_keep_errno(fname);
> - fname = psprintf("%s.lz4", path);
> - exists = (stat(fname, &st) == 0);
> -
> - if (exists)
> - compression_spec.algorithm = PG_COMPRESSION_LZ4;
> - }
> -#endif
> + else if (check_compressed_file(path, &fname, "gz"))
> + compression_spec.algorithm = PG_COMPRESSION_GZIP;
> + else if (check_compressed_file(path, &fname, "lz4"))
> + compression_spec.algorithm = PG_COMPRESSION_LZ4;
> + else if (check_compressed_file(path, &fname, "zst"))
> + compression_spec.algorithm = PG_COMPRESSION_ZSTD;
> }
This function lost some coherence, I think. Should there be a hasSuffix
check at the top for ".zstd" (and, for that matter, ".lz4")? And the
comment references an unused warning, which is only possible with the
#ifdef blocks that were removed.
I'm a little suspicious of the replacement of supports_compression()
with parse_compress_specification(). For example:
> - errmsg = supports_compression(AH->compression_spec);
> - if (errmsg)
> + parse_compress_specification(AH->compression_spec.algorithm,
> + NULL, &compress_spec);
> + if (compress_spec.parse_error != NULL)
> {
> pg_log_warning("archive is compressed, but this installation does not support compression (%s
> - errmsg);
> - pg_free(errmsg);
> + compress_spec.parse_error);
> + pg_free(compress_spec.parse_error);
> }
The top-level error here is "does not support compression", but wouldn't
a bad specification option with a supported compression method trip this
path too?
> +static void
> +ZSTD_CCtx_setParam_or_die(ZSTD_CStream *cstream,
> + ZSTD_cParameter param, int value, char *paramname)
IMO we should avoid stepping on the ZSTD_ namespace with our own
internal function names.
> + if (cs->readF != NULL)
> + {
> + zstdcs->dstream = ZSTD_createDStream();
> + if (zstdcs->dstream == NULL)
> + pg_fatal("could not initialize compression library");
> +
> + zstdcs->input.size = ZSTD_DStreamInSize();
> + zstdcs->input.src = pg_malloc(zstdcs->input.size);
> +
> + zstdcs->output.size = ZSTD_DStreamOutSize();
> + zstdcs->output.dst = pg_malloc(zstdcs->output.size + 1);
> + }
> +
> + if (cs->writeF != NULL)
> + {
> + zstdcs->cstream = ZstdCStreamParams(cs->compression_spec);
> +
> + zstdcs->output.size = ZSTD_CStreamOutSize();
> + zstdcs->output.dst = pg_malloc(zstdcs->output.size);
> + zstdcs->output.pos = 0;
> + }
This seems to suggest that both cs->readF and cs->writeF could be set,
but in that case, the output buffer gets reallocated.
I was curious about the extra byte allocated in the decompression case.
I see that ReadDataFromArchiveZstd() is null-terminating the buffer
before handing it to ahwrite(), but why does it need to do that?
> +static const char *
> +Zstd_get_error(CompressFileHandle *CFH)
> +{
> + return strerror(errno);
> +}
Seems like this should be using the zstderror stored in the handle?
In ReadDataFromArchiveZstd():
> + size_t input_allocated_size = ZSTD_DStreamInSize();
> + size_t res;
> +
> + for (;;)
> + {
> + size_t cnt;
> +
> + /*
> + * Read compressed data. Note that readF can resize the buffer; the
> + * new size is tracked and used for future loops.
> + */
> + input->size = input_allocated_size;
> + cnt = cs->readF(AH, (char **) unconstify(void **, &input->src), &input->size);
> + input_allocated_size = input->size;
> + input->size = cnt;
This is pretty complex for what it's doing. I'm a little worried that we
let the reallocated buffer escape to the caller while losing track of
how big it is. I think that works today, since there's only ever one
call per handle, but any future refactoring that allowed cs->readData()
to be called more than once would subtly break this code.
In ZstdWriteCommon():
> + /*
> + * Extra paranoia: avoid zero-length chunks, since a zero length chunk
> + * is the EOF marker in the custom format. This should never happen
> + * but...
> + */
> + if (output->pos > 0)
> + cs->writeF(AH, output->dst, output->pos);
> +
> + output->pos = 0;
Elsewhere, output->pos is set to zero before compressing, but here we do
it after, which I think leads to subtle differences in the function
preconditions. If that's an intentional difference, can the reason be
called out in a comment?
--Jacob
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
@ 2023-03-16 04:50 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-28 22:33 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
0 siblings, 2 replies; 46+ messages in thread
From: Justin Pryzby @ 2023-03-16 04:50 UTC (permalink / raw)
To: Jacob Champion <jchampion@timescale.com>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <jchampion@timescale.com> wrote:
> > I did some smoke testing against zstd's GitHub release on Windows. To
> > build against it, I had to construct an import library, and put that
> > and the DLL into the `lib` folder expected by the MSVC scripts...
> > which makes me wonder if I've chosen a harder way than necessary?
>
> It looks like pg_dump's meson.build is missing dependencies on zstd
> (meson couldn't find the headers in the subproject without them).
I saw that this was added for LZ4, but I hadn't added it for zstd since
I didn't run into an issue without it. Could you check that what I've
added works for your case ?
> > Parallel zstd dumps seem to work as expected, in that the resulting
> > pg_restore output is identical to uncompressed dumps and nothing
> > explodes. I haven't inspected the threading implementation for safety
> > yet, as you mentioned.
>
> Hm. Best I can tell, the CloneArchive() machinery is supposed to be
> handling safety for this, by isolating each thread's state. I don't feel
> comfortable pronouncing this new addition safe or not, because I'm not
> sure I understand what the comments in the format-specific _Clone()
> callbacks are saying yet.
My line of reasoning for unix is that pg_dump forks before any calls to
zstd. Nothing zstd does ought to affect the pg_dump layer. But that
doesn't apply to pg_dump under windows. This is an opened question. If
there's no solid answer, I could disable/ignore the option (maybe only
under windows).
> On to code (not a complete review):
>
> > if (hasSuffix(fname, ".gz"))
> > compression_spec.algorithm = PG_COMPRESSION_GZIP;
> > else
> > {
> > bool exists;
> >
> > exists = (stat(path, &st) == 0);
> > /* avoid unused warning if it is not built with compression */
> > if (exists)
> > compression_spec.algorithm = PG_COMPRESSION_NONE;
> > -#ifdef HAVE_LIBZ
> > - if (!exists)
> > - {
> > - free_keep_errno(fname);
> > - fname = psprintf("%s.gz", path);
> > - exists = (stat(fname, &st) == 0);
> > -
> > - if (exists)
> > - compression_spec.algorithm = PG_COMPRESSION_GZIP;
> > - }
> > -#endif
> > -#ifdef USE_LZ4
> > - if (!exists)
> > - {
> > - free_keep_errno(fname);
> > - fname = psprintf("%s.lz4", path);
> > - exists = (stat(fname, &st) == 0);
> > -
> > - if (exists)
> > - compression_spec.algorithm = PG_COMPRESSION_LZ4;
> > - }
> > -#endif
> > + else if (check_compressed_file(path, &fname, "gz"))
> > + compression_spec.algorithm = PG_COMPRESSION_GZIP;
> > + else if (check_compressed_file(path, &fname, "lz4"))
> > + compression_spec.algorithm = PG_COMPRESSION_LZ4;
> > + else if (check_compressed_file(path, &fname, "zst"))
> > + compression_spec.algorithm = PG_COMPRESSION_ZSTD;
> > }
>
> This function lost some coherence, I think. Should there be a hasSuffix
> check at the top for ".zstd" (and, for that matter, ".lz4")?
The function is first checking if it was passed a filename which already
has a suffix. And if not, it searches through a list of suffixes,
testing for an existing file with each suffix. The search with stat()
doesn't happen if it has a suffix. I'm having trouble seeing how the
hasSuffix() branch isn't dead code. Another opened question.
> I'm a little suspicious of the replacement of supports_compression()
> with parse_compress_specification(). For example:
>
> > - errmsg = supports_compression(AH->compression_spec);
> > - if (errmsg)
> > + parse_compress_specification(AH->compression_spec.algorithm,
> > + NULL, &compress_spec);
> > + if (compress_spec.parse_error != NULL)
> > {
> > pg_log_warning("archive is compressed, but this installation does not support compression (%s
> > - errmsg);
> > - pg_free(errmsg);
> > + compress_spec.parse_error);
> > + pg_free(compress_spec.parse_error);
> > }
>
> The top-level error here is "does not support compression", but wouldn't
> a bad specification option with a supported compression method trip this
> path too?
No - since the 2nd argument is passed as NULL, it just checks whether
the compression is supported. Maybe there ought to be a more
direct/clean way to do it. But up to now evidently nobody needed to do
that.
> > +static void
> > +ZSTD_CCtx_setParam_or_die(ZSTD_CStream *cstream,
> > + ZSTD_cParameter param, int value, char *paramname)
>
> IMO we should avoid stepping on the ZSTD_ namespace with our own
> internal function names.
done
> > + if (cs->readF != NULL)
> > +
> > + if (cs->writeF != NULL)
>
> This seems to suggest that both cs->readF and cs->writeF could be set,
> but in that case, the output buffer gets reallocated.
I put back an assertion that exactly one of them was set, since that's
true of how it currently works.
> I was curious about the extra byte allocated in the decompression case.
> I see that ReadDataFromArchiveZstd() is null-terminating the buffer
> before handing it to ahwrite(), but why does it need to do that?
I was trying to figure that out, too. I think the unterminated case
might be for ExecuteSqlCommandBuf(), and that may only (have) been
needed to allow pg_restore to handle ancient/development versions of
pg_dump... It's not currently hit.
https://coverage.postgresql.org/src/bin/pg_dump/pg_backup_db.c.gcov.html#470
I found that the terminator was added for the uncompressed case was
added at e8f69be05 and removed in bf9aa490d.
> > +Zstd_get_error(CompressFileHandle *CFH)
>
> Seems like this should be using the zstderror stored in the handle?
Yes - I'd already addressed that locally.
> In ReadDataFromArchiveZstd():
>
> > + * Read compressed data. Note that readF can resize the buffer; the
> > + * new size is tracked and used for future loops.
> This is pretty complex for what it's doing. I'm a little worried that we
> let the reallocated buffer escape to the caller while losing track of
> how big it is. I think that works today, since there's only ever one
> call per handle, but any future refactoring that allowed cs->readData()
> to be called more than once would subtly break this code.
Note that nothing bad happens if we lose track of how big it is (well,
assuming that readF doesn't *shrink* the buffer).
The previous patch version didn't keep track of its new size, and the only
consequence is that readF() might re-resize it again on a future iteration,
even if it was already sufficiently large.
When I originally wrote it (and up until that patch version), I left
this as an XXX comment about reusing the resized buffer. But it seemed
easy enough to fix so I did.
> In ZstdWriteCommon():
>
> Elsewhere, output->pos is set to zero before compressing, but here we do
> it after, which I think leads to subtle differences in the function
> preconditions. If that's an intentional difference, can the reason be
> called out in a comment?
It's not deliberate. I think it had no effect, but changed - thanks.
--
Justin
Attachments:
[text/x-diff] 0001-pg_dump-zstd-compression.patch (30.9K, ../../ZBKgBz+4blcKru6x@telsasoft.com/2-0001-pg_dump-zstd-compression.patch)
download | inline diff:
From 605bfb6974503bb71bdc09c9539313ef92d19ff3 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 7 Jan 2023 15:45:06 -0600
Subject: [PATCH 1/3] pg_dump: zstd compression
Previously proposed at: 20201221194924.GI30237@telsasoft.com
---
doc/src/sgml/ref/pg_dump.sgml | 8 +-
src/bin/pg_dump/Makefile | 2 +
src/bin/pg_dump/compress_io.c | 85 ++---
src/bin/pg_dump/compress_zstd.c | 523 ++++++++++++++++++++++++++
src/bin/pg_dump/compress_zstd.h | 25 ++
src/bin/pg_dump/meson.build | 10 +-
src/bin/pg_dump/pg_backup_archiver.c | 28 +-
src/bin/pg_dump/pg_backup_directory.c | 2 +
src/bin/pg_dump/pg_dump.c | 13 -
src/bin/pg_dump/t/002_pg_dump.pl | 79 +++-
src/tools/pginclude/cpluspluscheck | 1 +
11 files changed, 683 insertions(+), 93 deletions(-)
create mode 100644 src/bin/pg_dump/compress_zstd.c
create mode 100644 src/bin/pg_dump/compress_zstd.h
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index e6b003bf104..4652087144f 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -330,8 +330,9 @@ PostgreSQL documentation
machine-readable format that <application>pg_restore</application>
can read. A directory format archive can be manipulated with
standard Unix tools; for example, files in an uncompressed archive
- can be compressed with the <application>gzip</application> or
- <application>lz4</application> tools.
+ can be compressed with the <application>gzip</application>,
+ <application>lz4</application>, or
+ <application>zstd</application> tools.
This format is compressed by default using <literal>gzip</literal>
and also supports parallel dumps.
</para>
@@ -655,7 +656,8 @@ PostgreSQL documentation
<para>
Specify the compression method and/or the compression level to use.
The compression method can be set to <literal>gzip</literal>,
- <literal>lz4</literal>, or <literal>none</literal> for no compression.
+ <literal>lz4</literal>, <literal>zstd</literal>,
+ or <literal>none</literal> for no compression.
A compression detail string can optionally be specified. If the
detail string is an integer, it specifies the compression level.
Otherwise, it should be a comma-separated list of items, each of the
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index eb8f59459a1..24de7593a6a 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -18,6 +18,7 @@ include $(top_builddir)/src/Makefile.global
export GZIP_PROGRAM=$(GZIP)
export LZ4
+export ZSTD
export with_icu
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
@@ -29,6 +30,7 @@ OBJS = \
compress_io.o \
compress_lz4.o \
compress_none.o \
+ compress_zstd.o \
dumputils.o \
parallel.o \
pg_backup_archiver.o \
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index ce06f1eac9c..21972933d63 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -52,8 +52,8 @@
*
* InitDiscoverCompressFileHandle tries to infer the compression by the
* filename suffix. If the suffix is not yet known then it tries to simply
- * open the file and if it fails, it tries to open the same file with the .gz
- * suffix, and then again with the .lz4 suffix.
+ * open the file and if it fails, it tries to open the same file with
+ * compressed suffixes.
*
* IDENTIFICATION
* src/bin/pg_dump/compress_io.c
@@ -69,6 +69,7 @@
#include "compress_io.h"
#include "compress_lz4.h"
#include "compress_none.h"
+#include "compress_zstd.h"
#include "pg_backup_utils.h"
/*----------------------
@@ -76,36 +77,6 @@
*----------------------
*/
-/*
- * Checks whether a compression algorithm is supported.
- *
- * On success returns NULL, otherwise returns a malloc'ed string which can be
- * used by the caller in an error message.
- */
-char *
-supports_compression(const pg_compress_specification compression_spec)
-{
- const pg_compress_algorithm algorithm = compression_spec.algorithm;
- bool supported = false;
-
- if (algorithm == PG_COMPRESSION_NONE)
- supported = true;
-#ifdef HAVE_LIBZ
- if (algorithm == PG_COMPRESSION_GZIP)
- supported = true;
-#endif
-#ifdef USE_LZ4
- if (algorithm == PG_COMPRESSION_LZ4)
- supported = true;
-#endif
-
- if (!supported)
- return psprintf("this build does not support compression with %s",
- get_compress_algorithm_name(algorithm));
-
- return NULL;
-}
-
/*----------------------
* Compressor API
*----------------------
@@ -130,6 +101,8 @@ AllocateCompressor(const pg_compress_specification compression_spec,
InitCompressorGzip(cs, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressorLZ4(cs, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressorZstd(cs, compression_spec);
return cs;
}
@@ -196,20 +169,30 @@ InitCompressFileHandle(const pg_compress_specification compression_spec)
InitCompressFileHandleGzip(CFH, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressFileHandleLZ4(CFH, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressFileHandleZstd(CFH, compression_spec);
return CFH;
}
+static bool
+check_compressed_file(const char *path, char **fname, char *ext)
+{
+ free_keep_errno(*fname);
+ *fname = psprintf("%s.%s", path, ext);
+ return (access(*fname, F_OK) == 0);
+}
+
/*
* Open a file for reading. 'path' is the file to open, and 'mode' should
* be either "r" or "rb".
*
* If the file at 'path' contains the suffix of a supported compression method,
- * currently this includes ".gz" and ".lz4", then this compression will be used
+ * currently this includes ".gz", ".lz4" and ".zst", then this compression will be used
* throughout. Otherwise the compression will be inferred by iteratively trying
* to open the file at 'path', first as is, then by appending known compression
* suffixes. So if you pass "foo" as 'path', this will open either "foo" or
- * "foo.gz" or "foo.lz4", trying in that order.
+ * "foo.{gz,lz4,zst}", trying in that order.
*
* On failure, return NULL with an error code in errno.
*/
@@ -231,34 +214,14 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode)
compression_spec.algorithm = PG_COMPRESSION_GZIP;
else
{
- bool exists;
-
- exists = (stat(path, &st) == 0);
- /* avoid unused warning if it is not built with compression */
- if (exists)
+ if (stat(path, &st) == 0)
compression_spec.algorithm = PG_COMPRESSION_NONE;
-#ifdef HAVE_LIBZ
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.gz", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_GZIP;
- }
-#endif
-#ifdef USE_LZ4
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.lz4", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_LZ4;
- }
-#endif
+ else if (check_compressed_file(path, &fname, "gz"))
+ compression_spec.algorithm = PG_COMPRESSION_GZIP;
+ else if (check_compressed_file(path, &fname, "lz4"))
+ compression_spec.algorithm = PG_COMPRESSION_LZ4;
+ else if (check_compressed_file(path, &fname, "zst"))
+ compression_spec.algorithm = PG_COMPRESSION_ZSTD;
}
CFH = InitCompressFileHandle(compression_spec);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
new file mode 100644
index 00000000000..dc610a64de5
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -0,0 +1,523 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_zstd.c
+ * Routines for archivers to write a Zstd compressed data stream.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/compress_zstd.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "pg_backup_utils.h"
+#include "compress_zstd.h"
+
+#ifndef USE_ZSTD
+
+void
+InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+#else
+
+#include <zstd.h>
+
+typedef struct ZstdCompressorState
+{
+ /* This is a normal file to which we read/write compressed data */
+ FILE *fp;
+
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+
+ /* pointer to a static string like from strerror(), for Zstd_write() */
+ const char *zstderror;
+} ZstdCompressorState;
+
+static ZSTD_CStream *ZstdCStreamParams(pg_compress_specification compress);
+static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen);
+static void ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs);
+
+static void
+Zstd_CCtx_setParam_or_die(ZSTD_CStream *cstream,
+ ZSTD_cParameter param, int value, char *paramname)
+{
+ size_t res;
+
+ res = ZSTD_CCtx_setParameter(cstream, param, value);
+ if (ZSTD_isError(res))
+ pg_fatal("could not set compression parameter: \"%s\": %s",
+ paramname, ZSTD_getErrorName(res));
+}
+
+/* Return a compression stream with parameters set per argument */
+static ZSTD_CStream *
+ZstdCStreamParams(pg_compress_specification compress)
+{
+ ZSTD_CStream *cstream;
+
+ cstream = ZSTD_createCStream();
+ if (cstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_compressionLevel,
+ compress.level, "level");
+
+ if (compress.options & PG_COMPRESSION_OPTION_WORKERS)
+ Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
+ compress.workers, "workers");
+
+ return cstream;
+}
+
+/* Helper function for WriteDataToArchiveZstd and EndCompressorZstd */
+static void
+ZstdWriteCommon(ArchiveHandle *AH, CompressorState *cs, bool flush)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ /* Loop while there's any input or until flushed */
+ while (input->pos != input->size || flush)
+ {
+ size_t res;
+
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output,
+ input, flush ? ZSTD_e_end : ZSTD_e_continue);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ /*
+ * Extra paranoia: avoid zero-length chunks, since a zero length chunk
+ * is the EOF marker in the custom format. This should never happen
+ * but...
+ */
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ if (res == 0)
+ break;
+ }
+}
+
+void
+EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ /* We expect that exactly one of readF/writeF is specified */
+ Assert((cs->readF == NULL) != (cs->writeF == NULL));
+
+ if (cs->readF != NULL)
+ {
+ Assert(zstdcs->cstream == NULL);
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ }
+ else if (cs->writeF != NULL)
+ {
+ Assert(zstdcs->dstream == NULL);
+ ZstdWriteCommon(AH, cs, true);
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ pg_free(zstdcs);
+}
+
+static void
+WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ zstdcs->input.src = data;
+ zstdcs->input.size = dLen;
+ zstdcs->input.pos = 0;
+
+ ZstdWriteCommon(AH, cs, false);
+}
+
+static void
+ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ size_t input_allocated_size = ZSTD_DStreamInSize();
+ size_t res;
+
+ for (;;)
+ {
+ size_t cnt;
+
+ /*
+ * Read compressed data. Note that readF can resize the buffer; the
+ * new size is tracked and used for future loops.
+ */
+ input->size = input_allocated_size;
+ cnt = cs->readF(AH, (char **) unconstify(void **, &input->src), &input->size);
+ input_allocated_size = input->size;
+ input->size = cnt;
+ input->pos = 0;
+
+ if (cnt == 0)
+ break;
+
+ /* Now decompress */
+ while (input->pos < input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ /* then write the decompressed data to the output handle */
+ ((char *) output->dst)[output->pos] = '\0';
+ ahwrite(output->dst, 1, output->pos, AH);
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+ }
+}
+
+/* Public routine that supports Zstd compressed data I/O */
+void
+InitCompressorZstd(CompressorState *cs,
+ const pg_compress_specification compression_spec)
+{
+ ZstdCompressorState *zstdcs;
+
+ cs->readData = ReadDataFromArchiveZstd;
+ cs->writeData = WriteDataToArchiveZstd;
+ cs->end = EndCompressorZstd;
+
+ cs->compression_spec = compression_spec;
+
+ zstdcs = (ZstdCompressorState *) pg_malloc0(sizeof(*zstdcs));
+ cs->private_data = zstdcs;
+
+ /* We expect that exactly one of readF/writeF is specified */
+ Assert((cs->readF == NULL) != (cs->writeF == NULL));
+
+ if (cs->readF != NULL)
+ {
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ zstdcs->input.size = ZSTD_DStreamInSize();
+ zstdcs->input.src = pg_malloc(zstdcs->input.size);
+
+ zstdcs->output.size = ZSTD_DStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size + 1);
+ }
+ else if (cs->writeF != NULL)
+ {
+ zstdcs->cstream = ZstdCStreamParams(cs->compression_spec);
+
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size);
+ zstdcs->output.pos = 0;
+ }
+}
+
+/*
+ * Compressed stream API
+ */
+
+static size_t
+Zstd_read(void *ptr, size_t size, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t input_allocated_size = ZSTD_DStreamInSize();
+ size_t res,
+ cnt;
+
+ output->size = size;
+ output->dst = ptr;
+ output->pos = 0;
+
+ for (;;)
+ {
+ Assert(input->pos <= input->size);
+ Assert(input->size <= input_allocated_size);
+
+ /* If the input is completely consumed, start back at the beginning */
+ if (input->pos == input->size)
+ {
+ /* input->size is size produced by "fread" */
+ input->size = 0;
+ /* input->pos is position consumed by decompress */
+ input->pos = 0;
+ }
+
+ /* read compressed data if we must produce more input */
+ if (input->pos == input->size)
+ {
+ cnt = fread(unconstify(void *, input->src), 1, input_allocated_size, zstdcs->fp);
+ input->size = cnt;
+
+ Assert(cnt >= 0);
+ Assert(cnt <= input_allocated_size);
+
+ /* If we have no more input to consume, we're done */
+ if (cnt == 0)
+ break;
+ }
+
+ while (input->pos < input->size)
+ {
+ /* now decompress */
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ if (output->pos == output->size)
+ break; /* No more room for output */
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+
+ if (output->pos == output->size)
+ break; /* We read all the data that fits */
+ }
+
+ return output->pos;
+}
+
+static size_t
+Zstd_write(const void *ptr, size_t size, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t res,
+ cnt;
+
+ input->src = ptr;
+ input->size = size;
+ input->pos = 0;
+
+ /* Consume all input, to be flushed later */
+ while (input->pos != input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_continue);
+ if (ZSTD_isError(res))
+ {
+ zstdcs->zstderror = ZSTD_getErrorName(res);
+ return -1;
+ }
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ {
+ zstdcs->zstderror = strerror(errno);
+ return -1;
+ }
+ }
+
+ return size;
+}
+
+static int
+Zstd_getc(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ int ret;
+
+ if (CFH->read_func(&ret, 1, CFH) != 1)
+ {
+ if (feof(zstdcs->fp))
+ pg_fatal("could not read from input file: end of file");
+ else
+ pg_fatal("could not read from input file: %m");
+ }
+ return ret;
+}
+
+static char *
+Zstd_gets(char *buf, int len, CompressFileHandle *CFH)
+{
+ int i,
+ res;
+
+ /*
+ * Read one byte at a time until newline or EOF. This is only used to read
+ * the list of LOs, and the I/O is buffered anyway.
+ */
+ for (i = 0; i < len - 1; ++i)
+ {
+ res = CFH->read_func(&buf[i], 1, CFH);
+ if (res != 1)
+ break;
+ if (buf[i] == '\n')
+ {
+ ++i;
+ break;
+ }
+ }
+ buf[i] = '\0';
+ return i > 0 ? buf : NULL;
+}
+
+static int
+Zstd_close(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ int result;
+
+ if (zstdcs->cstream)
+ {
+ size_t res,
+ cnt;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ /* Loop until the compression buffers are fully consumed */
+ for (;;)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_end);
+ if (ZSTD_isError(res))
+ {
+ zstdcs->zstderror = ZSTD_getErrorName(res);
+ return -1;
+ }
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ {
+ zstdcs->zstderror = strerror(errno);
+ return -1;
+ }
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ if (zstdcs->dstream)
+ {
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ }
+
+ result = fclose(zstdcs->fp);
+ pg_free(zstdcs);
+ return result;
+}
+
+static int
+Zstd_eof(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ return feof(zstdcs->fp);
+}
+
+static int
+Zstd_open(const char *path, int fd, const char *mode,
+ CompressFileHandle *CFH)
+{
+ FILE *fp;
+ ZstdCompressorState *zstdcs;
+
+ if (fd >= 0)
+ fp = fdopen(fd, mode);
+ else
+ fp = fopen(path, mode);
+
+ if (fp == NULL)
+ return 1;
+
+ zstdcs = (ZstdCompressorState *) pg_malloc0(sizeof(*zstdcs));
+ CFH->private_data = zstdcs;
+ zstdcs->fp = fp;
+
+ if (mode[0] == 'r')
+ {
+ zstdcs->input.src = pg_malloc0(ZSTD_DStreamInSize());
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+ }
+ else if (mode[0] == 'w' || mode[0] == 'a')
+ {
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc0(zstdcs->output.size);
+ zstdcs->cstream = ZstdCStreamParams(CFH->compression_spec);
+ if (zstdcs->cstream == NULL)
+ pg_fatal("could not initialize compression library");
+ }
+ else
+ pg_fatal("unhandled mode");
+
+ return 0;
+}
+
+static int
+Zstd_open_write(const char *path, const char *mode, CompressFileHandle *CFH)
+{
+ char fname[MAXPGPATH];
+
+ sprintf(fname, "%s.zst", path);
+ return CFH->open_func(fname, -1, mode, CFH);
+}
+
+static const char *
+Zstd_get_error(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ return zstdcs->zstderror;
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH,
+ const pg_compress_specification compression_spec)
+{
+ CFH->open_func = Zstd_open;
+ CFH->open_write_func = Zstd_open_write;
+ CFH->read_func = Zstd_read;
+ CFH->write_func = Zstd_write;
+ CFH->gets_func = Zstd_gets;
+ CFH->getc_func = Zstd_getc;
+ CFH->close_func = Zstd_close;
+ CFH->eof_func = Zstd_eof;
+ CFH->get_error_func = Zstd_get_error;
+
+ CFH->compression_spec = compression_spec;
+
+ CFH->private_data = NULL;
+}
+
+#endif /* USE_ZSTD */
diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h
new file mode 100644
index 00000000000..2aaa6b100b1
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_zstd.h
+ * Zstd interface to compress_io.c routines
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/compress_zstd.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef COMPRESS_ZSTD_H
+#define COMPRESS_ZSTD_H
+
+#include "compress_io.h"
+
+extern void InitCompressorZstd(CompressorState *cs,
+ const pg_compress_specification compression_spec);
+extern void InitCompressFileHandleZstd(CompressFileHandle *CFH,
+ const pg_compress_specification compression_spec);
+
+#endif /* COMPRESS_ZSTD_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index ab4c25c7811..2c5006c61f7 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -5,6 +5,7 @@ pg_dump_common_sources = files(
'compress_io.c',
'compress_lz4.c',
'compress_none.c',
+ 'compress_zstd.c',
'dumputils.c',
'parallel.c',
'pg_backup_archiver.c',
@@ -19,7 +20,7 @@ pg_dump_common_sources = files(
pg_dump_common = static_library('libpgdump_common',
pg_dump_common_sources,
c_pch: pch_postgres_fe_h,
- dependencies: [frontend_code, libpq, lz4, zlib],
+ dependencies: [frontend_code, libpq, lz4, zlib, zstd],
kwargs: internal_lib_args,
)
@@ -39,7 +40,7 @@ endif
pg_dump = executable('pg_dump',
pg_dump_sources,
link_with: [pg_dump_common],
- dependencies: [frontend_code, libpq, zlib],
+ dependencies: [frontend_code, libpq, zlib, zstd],
kwargs: default_bin_args,
)
bin_targets += pg_dump
@@ -58,7 +59,7 @@ endif
pg_dumpall = executable('pg_dumpall',
pg_dumpall_sources,
link_with: [pg_dump_common],
- dependencies: [frontend_code, libpq, zlib],
+ dependencies: [frontend_code, libpq, zlib, zstd],
kwargs: default_bin_args,
)
bin_targets += pg_dumpall
@@ -77,7 +78,7 @@ endif
pg_restore = executable('pg_restore',
pg_restore_sources,
link_with: [pg_dump_common],
- dependencies: [frontend_code, libpq, zlib],
+ dependencies: [frontend_code, libpq, zlib, zstd],
kwargs: default_bin_args,
)
bin_targets += pg_restore
@@ -90,6 +91,7 @@ tests += {
'env': {
'GZIP_PROGRAM': gzip.path(),
'LZ4': program_lz4.found() ? program_lz4.path() : '',
+ 'ZSTD': program_zstd.found() ? program_zstd.path() : '',
'with_icu': icu.found() ? 'yes' : 'no',
},
'tests': [
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 61ebb8fe85d..6fda84d8774 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -388,10 +388,12 @@ RestoreArchive(Archive *AHX)
{
if (te->hadDumper && (te->reqs & REQ_DATA) != 0)
{
- char *errmsg = supports_compression(AH->compression_spec);
- if (errmsg)
+ pg_compress_specification compress_spec;
+ parse_compress_specification(AH->compression_spec.algorithm,
+ NULL, &compress_spec);
+ if (compress_spec.parse_error != NULL)
pg_fatal("cannot restore from compressed archive (%s)",
- errmsg);
+ compress_spec.parse_error);
else
break;
}
@@ -2075,7 +2077,7 @@ _discoverArchiveFormat(ArchiveHandle *AH)
/*
* Check if the specified archive is a directory. If so, check if
- * there's a "toc.dat" (or "toc.dat.{gz,lz4}") file in it.
+ * there's a "toc.dat" (or "toc.dat.{gz,lz4,zst}") file in it.
*/
if (stat(AH->fSpec, &st) == 0 && S_ISDIR(st.st_mode))
{
@@ -2086,10 +2088,17 @@ _discoverArchiveFormat(ArchiveHandle *AH)
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.gz"))
return AH->format;
#endif
+
#ifdef USE_LZ4
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.lz4"))
return AH->format;
#endif
+
+#ifdef USE_ZSTD
+ if (_fileExistsInDirectory(AH->fSpec, "toc.dat.zst"))
+ return AH->format;
+#endif
+
pg_fatal("directory \"%s\" does not appear to be a valid archive (\"toc.dat\" does not exist)",
AH->fSpec);
fh = NULL; /* keep compiler quiet */
@@ -3674,7 +3683,7 @@ WriteHead(ArchiveHandle *AH)
void
ReadHead(ArchiveHandle *AH)
{
- char *errmsg;
+ pg_compress_specification compress_spec;
char vmaj,
vmin,
vrev;
@@ -3745,12 +3754,13 @@ ReadHead(ArchiveHandle *AH)
else
AH->compression_spec.algorithm = PG_COMPRESSION_GZIP;
- errmsg = supports_compression(AH->compression_spec);
- if (errmsg)
+ parse_compress_specification(AH->compression_spec.algorithm,
+ NULL, &compress_spec);
+ if (compress_spec.parse_error != NULL)
{
pg_log_warning("archive is compressed, but this installation does not support compression (%s) -- no data will be available",
- errmsg);
- pg_free(errmsg);
+ compress_spec.parse_error);
+ pg_free(compress_spec.parse_error);
}
if (AH->version >= K_VERS_1_4)
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 41c2b733e3e..29845340859 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -785,6 +785,8 @@ _PrepParallelRestore(ArchiveHandle *AH)
strlcat(fname, ".gz", sizeof(fname));
else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4)
strlcat(fname, ".lz4", sizeof(fname));
+ else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ strlcat(fname, ".zst", sizeof(fname));
if (stat(fname, &st) == 0)
te->dataLength = st.st_size;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2e068c6620e..398a7a7bb3d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -734,19 +734,6 @@ main(int argc, char **argv)
pg_fatal("invalid compression specification: %s",
error_detail);
- switch (compression_algorithm)
- {
- case PG_COMPRESSION_NONE:
- /* fallthrough */
- case PG_COMPRESSION_GZIP:
- /* fallthrough */
- case PG_COMPRESSION_LZ4:
- break;
- case PG_COMPRESSION_ZSTD:
- pg_fatal("compression with %s is not yet supported", "ZSTD");
- break;
- }
-
/*
* Custom and directory formats are compressed by default with gzip when
* available, not the others.
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 42215f82f7a..74f23ae7f74 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -54,8 +54,9 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir;
# those lines) to validate that part of the process.
my $supports_icu = ($ENV{with_icu} eq 'yes');
-my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1");
+my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
+my $supports_zstd = check_pg_config("#define USE_ZSTD 1");
my %pgdump_runs = (
binary_upgrade => {
@@ -213,6 +214,77 @@ my %pgdump_runs = (
},
},
+ compression_zstd_custom => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=custom',
+ '--compress=zstd', "--file=$tempdir/compression_zstd_custom.dump",
+ 'postgres',
+ ],
+ restore_cmd => [
+ 'pg_restore',
+ "--file=$tempdir/compression_zstd_custom.sql",
+ "$tempdir/compression_zstd_custom.dump",
+ ],
+ command_like => {
+ command => [
+ 'pg_restore',
+ '-l', "$tempdir/compression_zstd_custom.dump",
+ ],
+ expected => qr/Compression: zstd/,
+ name => 'data content is zstd compressed'
+ },
+ },
+
+ compression_zstd_dir => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--jobs=2',
+ '--format=directory', '--compress=zstd:1',
+ "--file=$tempdir/compression_zstd_dir", 'postgres',
+ ],
+ # Give coverage for manually compressed blob.toc files during
+ # restore.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-z', '-f', '--rm',
+ "$tempdir/compression_zstd_dir/blobs.toc",
+ "-o", "$tempdir/compression_zstd_dir/blobs.toc.zst",
+ ],
+ },
+ # Verify that data files were compressed
+ glob_patterns => [
+ "$tempdir/compression_zstd_dir/toc.dat",
+ "$tempdir/compression_zstd_dir/*.dat.zst",
+ ],
+ restore_cmd => [
+ 'pg_restore', '--jobs=2',
+ "--file=$tempdir/compression_zstd_dir.sql",
+ "$tempdir/compression_zstd_dir",
+ ],
+ },
+
+ compression_zstd_plain => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=plain', '--compress=zstd',
+ "--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
+ ],
+ # Decompress the generated file to run through the tests.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-d', '-f',
+ "$tempdir/compression_zstd_plain.sql.zst",
+ "-o", "$tempdir/compression_zstd_plain.sql",
+ ],
+ },
+ },
+
clean => {
dump_cmd => [
'pg_dump',
@@ -4648,10 +4720,11 @@ foreach my $run (sort keys %pgdump_runs)
my $test_key = $run;
my $run_db = 'postgres';
- # Skip command-level tests for gzip/lz4 if there is no support for it.
+ # Skip command-level tests for gzip/lz4/zstd if the tool is not supported
if ($pgdump_runs{$run}->{compile_option} &&
(($pgdump_runs{$run}->{compile_option} eq 'gzip' && !$supports_gzip) ||
- ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4)))
+ ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4) ||
+ ($pgdump_runs{$run}->{compile_option} eq 'zstd' && !$supports_zstd)))
{
note "$run: skipped due to no $pgdump_runs{$run}->{compile_option} support";
next;
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index 58039934756..10fb51585c9 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -154,6 +154,7 @@ do
test "$f" = src/bin/pg_dump/compress_io.h && continue
test "$f" = src/bin/pg_dump/compress_lz4.h && continue
test "$f" = src/bin/pg_dump/compress_none.h && continue
+ test "$f" = src/bin/pg_dump/compress_zstd.h && continue
test "$f" = src/bin/pg_dump/parallel.h && continue
test "$f" = src/bin/pg_dump/pg_backup_archiver.h && continue
test "$f" = src/bin/pg_dump/pg_dump.h && continue
--
2.34.1
[text/x-diff] 0002-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch (2.2K, ../../ZBKgBz+4blcKru6x@telsasoft.com/3-0002-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch)
download | inline diff:
From 12f839a8b1e601849304bff8b3fc12579d6d4ecc Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 4 Jan 2023 21:21:53 -0600
Subject: [PATCH 2/3] TMP: pg_dump: use Zstd by default, for CI only
//-os-only: warnings
---
src/bin/pg_dump/pg_dump.c | 4 ++--
src/bin/pg_dump/t/002_pg_dump.pl | 14 +++++++-------
2 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 398a7a7bb3d..cca52d2cd0d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -741,8 +741,8 @@ main(int argc, char **argv)
if ((archiveFormat == archCustom || archiveFormat == archDirectory) &&
!user_compression_defined)
{
-#ifdef HAVE_LIBZ
- parse_compress_specification(PG_COMPRESSION_GZIP, NULL,
+#ifdef USE_ZSTD
+ parse_compress_specification(PG_COMPRESSION_ZSTD, NULL,
&compression_spec);
#else
/* Nothing to do in the default case */
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 74f23ae7f74..87f1d2c692d 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -384,10 +384,10 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_custom_format.dump", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
- name => 'data content is gzip-compressed by default if available',
+ name => 'data content is zstd-compressed by default if available',
},
},
@@ -409,16 +409,16 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_dir_format", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
name => 'data content is gzip-compressed by default',
},
glob_patterns => [
"$tempdir/defaults_dir_format/toc.dat",
"$tempdir/defaults_dir_format/blobs.toc",
- $supports_gzip ?
- "$tempdir/defaults_dir_format/*.dat.gz" :
+ $supports_zstd ?
+ "$tempdir/defaults_dir_format/*.dat.zst" :
"$tempdir/defaults_dir_format/*.dat",
],
},
--
2.34.1
[text/x-diff] 0003-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch (13.1K, ../../ZBKgBz+4blcKru6x@telsasoft.com/4-0003-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch)
download | inline diff:
From e818c386dd511871a65350e2b2fe3951f78e1cd6 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 27 Mar 2022 11:55:01 -0500
Subject: [PATCH 3/3] zstd: support long distance mode in pg_dump/basebackup
First proposed here:
20220327205020.GM28503@telsasoft.com
//-os-only: freebsd
---
doc/src/sgml/protocol.sgml | 10 +++-
doc/src/sgml/ref/pg_basebackup.sgml | 4 +-
doc/src/sgml/ref/pg_dump.sgml | 7 ++-
src/backend/backup/basebackup_zstd.c | 12 ++++
src/bin/pg_basebackup/bbstreamer_zstd.c | 13 +++++
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 9 ++-
src/bin/pg_dump/compress_zstd.c | 5 ++
src/bin/pg_dump/t/002_pg_dump.pl | 3 +-
src/bin/pg_verifybackup/t/008_untar.pl | 8 +++
src/bin/pg_verifybackup/t/010_client_untar.pl | 8 +++
src/common/compression.c | 57 ++++++++++++++++++-
src/include/common/compression.h | 2 +
12 files changed, 130 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 73b7f4432f3..05a887cd092 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2747,7 +2747,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
level. Otherwise, it should be a comma-separated list of items,
each of the form <replaceable>keyword</replaceable> or
<replaceable>keyword=value</replaceable>. Currently, the supported
- keywords are <literal>level</literal> and <literal>workers</literal>.
+ keywords are <literal>level</literal>, <literal>long</literal> and
+ <literal>workers</literal>.
</para>
<para>
@@ -2764,6 +2765,13 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<literal>3</literal>).
</para>
+ <para>
+ The <literal>long</literal> keyword enables long-distance matching
+ mode, for improved compression ratio, at the expense of higher memory
+ use. Long-distance mode is supported only for
+ <literal>zstd</literal>.
+ </para>
+
<para>
The <literal>workers</literal> keyword sets the number of threads
that should be used for parallel compression. Parallel compression
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index db3ad9cd5eb..79d3e657c32 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -424,8 +424,8 @@ PostgreSQL documentation
level. Otherwise, it should be a comma-separated list of items,
each of the form <literal>keyword</literal> or
<literal>keyword=value</literal>.
- Currently, the supported keywords are <literal>level</literal>
- and <literal>workers</literal>.
+ Currently, the supported keywords are <literal>level</literal>,
+ <literal>long</literal>, and <literal>workers</literal>.
The detail string cannot be used when the compression method
is specified as a plain integer.
</para>
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 4652087144f..9379b98e7f9 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -678,8 +678,11 @@ PostgreSQL documentation
individual table-data segments, and the default is to compress using
<literal>gzip</literal> at a moderate level. For plain text output,
setting a nonzero compression level causes the entire output file to be compressed,
- as though it had been fed through <application>gzip</application> or
- <application>lz4</application>; but the default is not to compress.
+ as though it had been fed through <application>gzip</application>, or
+ <application>lz4</application>; or <application>zstd</application>,
+ but the default is not to compress.
+ With zstd compression, <literal>long</literal> mode may improve the
+ compression ratio, at the cost of increased memory use.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/backend/backup/basebackup_zstd.c b/src/backend/backup/basebackup_zstd.c
index ac6cac178a0..1bb5820c884 100644
--- a/src/backend/backup/basebackup_zstd.c
+++ b/src/backend/backup/basebackup_zstd.c
@@ -118,6 +118,18 @@ bbsink_zstd_begin_backup(bbsink *sink)
compress->workers, ZSTD_getErrorName(ret)));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(mysink->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret)));
+ }
+
/*
* We need our own buffer, because we're going to pass different data to
* the next sink than what gets passed to us.
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index fe17d6df4ef..fba391e2a0f 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -106,6 +106,19 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, pg_compress_specification *comp
compress->workers, ZSTD_getErrorName(ret));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(streamer->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ {
+ pg_log_error("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret));
+ exit(1);
+ }
+ }
+
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
streamer->zstd_outBuf.size = streamer->base.bbs_buffer.maxlen;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index b60cb78a0d5..4d130a7f944 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -139,7 +139,14 @@ SKIP:
'gzip:workers=3',
'invalid compression specification: compression algorithm "gzip" does not accept a worker count',
'failure on worker count for gzip'
- ],);
+ ],
+ [
+ 'gzip:long',
+ 'invalid compression specification: compression algorithm "gzip" does not support long-distance mode',
+ 'failure on long mode for gzip'
+ ],
+ );
+
for my $cft (@compression_failure_tests)
{
my $cfail = quotemeta($client_fails . $cft->[1]);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
index dc610a64de5..ee9513cd555 100644
--- a/src/bin/pg_dump/compress_zstd.c
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -84,6 +84,11 @@ ZstdCStreamParams(pg_compress_specification compress)
Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
compress.workers, "workers");
+ if (compress.options & PG_COMPRESSION_OPTION_LONG_DISTANCE)
+ Zstd_CCtx_setParam_or_die(cstream,
+ ZSTD_c_enableLongDistanceMatching,
+ compress.long_distance, "long");
+
return cstream;
}
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 87f1d2c692d..0a635ae9fc3 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -267,11 +267,12 @@ my %pgdump_runs = (
],
},
+ # Exercise long mode for test coverage
compression_zstd_plain => {
test_key => 'compression',
compile_option => 'zstd',
dump_cmd => [
- 'pg_dump', '--format=plain', '--compress=zstd',
+ 'pg_dump', '--format=plain', '--compress=zstd:long',
"--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
],
# Decompress the generated file to run through the tests.
diff --git a/src/bin/pg_verifybackup/t/008_untar.pl b/src/bin/pg_verifybackup/t/008_untar.pl
index 3007bbe8556..05754bc8ec7 100644
--- a/src/bin/pg_verifybackup/t/008_untar.pl
+++ b/src/bin/pg_verifybackup/t/008_untar.pl
@@ -49,6 +49,14 @@ my @test_configuration = (
'decompress_program' => $ENV{'ZSTD'},
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => [ '--compress', 'server-zstd:level=1,long' ],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => ['-d'],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
});
for my $tc (@test_configuration)
diff --git a/src/bin/pg_verifybackup/t/010_client_untar.pl b/src/bin/pg_verifybackup/t/010_client_untar.pl
index f3aa0f59e29..ac51a174d14 100644
--- a/src/bin/pg_verifybackup/t/010_client_untar.pl
+++ b/src/bin/pg_verifybackup/t/010_client_untar.pl
@@ -50,6 +50,14 @@ my @test_configuration = (
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
},
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => ['--compress', 'client-zstd:level=1,long'],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => [ '-d' ],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
{
'compression_method' => 'parallel zstd',
'backup_flags' => [ '--compress', 'client-zstd:workers=3' ],
diff --git a/src/common/compression.c b/src/common/compression.c
index 2d3e56b4d62..713a77c292d 100644
--- a/src/common/compression.c
+++ b/src/common/compression.c
@@ -12,7 +12,7 @@
* Otherwise, a compression specification is a comma-separated list of items,
* each having the form keyword or keyword=value.
*
- * Currently, the only supported keywords are "level" and "workers".
+ * Currently, the supported keywords are "level", "long", and "workers".
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
*
@@ -38,6 +38,8 @@
static int expect_integer_value(char *keyword, char *value,
pg_compress_specification *result);
+static bool expect_boolean_value(char *keyword, char *value,
+ pg_compress_specification *result);
/*
* Look up a compression algorithm by name. Returns true and sets *algorithm
@@ -232,6 +234,11 @@ parse_compress_specification(pg_compress_algorithm algorithm, char *specificatio
result->workers = expect_integer_value(keyword, value, result);
result->options |= PG_COMPRESSION_OPTION_WORKERS;
}
+ else if (strcmp(keyword, "long") == 0)
+ {
+ result->long_distance = expect_boolean_value(keyword, value, result);
+ result->options |= PG_COMPRESSION_OPTION_LONG_DISTANCE;
+ }
else
result->parse_error =
psprintf(_("unrecognized compression option: \"%s\""), keyword);
@@ -289,6 +296,43 @@ expect_integer_value(char *keyword, char *value, pg_compress_specification *resu
return ivalue;
}
+/*
+ * Parse 'value' as an boolean and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1. The caller must check result->parse_error to determine if
+ * the call was successful.
+ *
+ * Valid values are: yes, no, on, off, 1, 0.
+ *
+ * Inspired by ParseVariableBool().
+ */
+static bool
+expect_boolean_value(char *keyword, char *value, pg_compress_specification *result)
+{
+ if (value == NULL)
+ return true;
+
+ if (pg_strcasecmp(value, "yes") == 0)
+ return true;
+ if (pg_strcasecmp(value, "on") == 0)
+ return true;
+ if (pg_strcasecmp(value, "1") == 0)
+ return true;
+
+ if (pg_strcasecmp(value, "no") == 0)
+ return false;
+ if (pg_strcasecmp(value, "off") == 0)
+ return false;
+ if (pg_strcasecmp(value, "0") == 0)
+ return false;
+
+ result->parse_error =
+ psprintf(_("value for compression option \"%s\" must be a boolean"),
+ keyword);
+ return false;
+}
+
/*
* Returns NULL if the compression specification string was syntactically
* valid and semantically sensible. Otherwise, returns an error message.
@@ -354,6 +398,17 @@ validate_compress_specification(pg_compress_specification *spec)
get_compress_algorithm_name(spec->algorithm));
}
+ /*
+ * Of the compression algorithms that we currently support, only zstd
+ * supports long-distance mode.
+ */
+ if ((spec->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0 &&
+ (spec->algorithm != PG_COMPRESSION_ZSTD))
+ {
+ return psprintf(_("compression algorithm \"%s\" does not support long-distance mode"),
+ get_compress_algorithm_name(spec->algorithm));
+ }
+
return NULL;
}
diff --git a/src/include/common/compression.h b/src/include/common/compression.h
index b48c173022e..6cf8cf396a8 100644
--- a/src/include/common/compression.h
+++ b/src/include/common/compression.h
@@ -27,6 +27,7 @@ typedef enum pg_compress_algorithm
} pg_compress_algorithm;
#define PG_COMPRESSION_OPTION_WORKERS (1 << 0)
+#define PG_COMPRESSION_OPTION_LONG_DISTANCE (1 << 1)
typedef struct pg_compress_specification
{
@@ -34,6 +35,7 @@ typedef struct pg_compress_specification
unsigned options; /* OR of PG_COMPRESSION_OPTION constants */
int level;
int workers;
+ int long_distance;
char *parse_error; /* NULL if parsing was OK, else message */
} pg_compress_specification;
--
2.34.1
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-03-17 02:43 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 16:20 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
1 sibling, 2 replies; 46+ messages in thread
From: Tomas Vondra @ 2023-03-17 02:43 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; Jacob Champion <jchampion@timescale.com>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On 3/16/23 05:50, Justin Pryzby wrote:
> On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
>> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <jchampion@timescale.com> wrote:
>>> I did some smoke testing against zstd's GitHub release on Windows. To
>>> build against it, I had to construct an import library, and put that
>>> and the DLL into the `lib` folder expected by the MSVC scripts...
>>> which makes me wonder if I've chosen a harder way than necessary?
>>
>> It looks like pg_dump's meson.build is missing dependencies on zstd
>> (meson couldn't find the headers in the subproject without them).
>
> I saw that this was added for LZ4, but I hadn't added it for zstd since
> I didn't run into an issue without it. Could you check that what I've
> added works for your case ?
>
>>> Parallel zstd dumps seem to work as expected, in that the resulting
>>> pg_restore output is identical to uncompressed dumps and nothing
>>> explodes. I haven't inspected the threading implementation for safety
>>> yet, as you mentioned.
>>
>> Hm. Best I can tell, the CloneArchive() machinery is supposed to be
>> handling safety for this, by isolating each thread's state. I don't feel
>> comfortable pronouncing this new addition safe or not, because I'm not
>> sure I understand what the comments in the format-specific _Clone()
>> callbacks are saying yet.
>
> My line of reasoning for unix is that pg_dump forks before any calls to
> zstd. Nothing zstd does ought to affect the pg_dump layer. But that
> doesn't apply to pg_dump under windows. This is an opened question. If
> there's no solid answer, I could disable/ignore the option (maybe only
> under windows).
>
I may be missing something, but why would the patch affect this? Why
would it even affect safety of the parallel dump? And I don't see any
changes to the clone stuff ...
>> On to code (not a complete review):
>>
>>> if (hasSuffix(fname, ".gz"))
>>> compression_spec.algorithm = PG_COMPRESSION_GZIP;
>>> else
>>> {
>>> bool exists;
>>>
>>> exists = (stat(path, &st) == 0);
>>> /* avoid unused warning if it is not built with compression */
>>> if (exists)
>>> compression_spec.algorithm = PG_COMPRESSION_NONE;
>>> -#ifdef HAVE_LIBZ
>>> - if (!exists)
>>> - {
>>> - free_keep_errno(fname);
>>> - fname = psprintf("%s.gz", path);
>>> - exists = (stat(fname, &st) == 0);
>>> -
>>> - if (exists)
>>> - compression_spec.algorithm = PG_COMPRESSION_GZIP;
>>> - }
>>> -#endif
>>> -#ifdef USE_LZ4
>>> - if (!exists)
>>> - {
>>> - free_keep_errno(fname);
>>> - fname = psprintf("%s.lz4", path);
>>> - exists = (stat(fname, &st) == 0);
>>> -
>>> - if (exists)
>>> - compression_spec.algorithm = PG_COMPRESSION_LZ4;
>>> - }
>>> -#endif
>>> + else if (check_compressed_file(path, &fname, "gz"))
>>> + compression_spec.algorithm = PG_COMPRESSION_GZIP;
>>> + else if (check_compressed_file(path, &fname, "lz4"))
>>> + compression_spec.algorithm = PG_COMPRESSION_LZ4;
>>> + else if (check_compressed_file(path, &fname, "zst"))
>>> + compression_spec.algorithm = PG_COMPRESSION_ZSTD;
>>> }
>>
>> This function lost some coherence, I think. Should there be a hasSuffix
>> check at the top for ".zstd" (and, for that matter, ".lz4")?
>
This was discussed in the lz4 thread a couple days, and I think there
should be hasSuffix() cases for lz4/zstd too, not just for .gz.
> The function is first checking if it was passed a filename which already
> has a suffix. And if not, it searches through a list of suffixes,
> testing for an existing file with each suffix. The search with stat()
> doesn't happen if it has a suffix. I'm having trouble seeing how the
> hasSuffix() branch isn't dead code. Another opened question.
>
AFAICS it's done this way because of this comment in pg_backup_directory
* ...
* ".gz" suffix is added to the filenames. The TOC files are never
* compressed by pg_dump, however they are accepted with the .gz suffix
* too, in case the user has manually compressed them with 'gzip'.
I haven't tried, but I believe that if you manually compress the
directory, it may hit this branch. And IMO if we support that for gzip,
the other compression methods should do that too for consistency.
In any case, it's a tiny amount of code and I don't feel like ripping
that out when it might break some currently supported use case.
>> I'm a little suspicious of the replacement of supports_compression()
>> with parse_compress_specification(). For example:
>>
>>> - errmsg = supports_compression(AH->compression_spec);
>>> - if (errmsg)
>>> + parse_compress_specification(AH->compression_spec.algorithm,
>>> + NULL, &compress_spec);
>>> + if (compress_spec.parse_error != NULL)
>>> {
>>> pg_log_warning("archive is compressed, but this installation does not support compression (%s
>>> - errmsg);
>>> - pg_free(errmsg);
>>> + compress_spec.parse_error);
>>> + pg_free(compress_spec.parse_error);
>>> }
>>
>> The top-level error here is "does not support compression", but wouldn't
>> a bad specification option with a supported compression method trip this
>> path too?
>
> No - since the 2nd argument is passed as NULL, it just checks whether
> the compression is supported. Maybe there ought to be a more
> direct/clean way to do it. But up to now evidently nobody needed to do
> that.
>
I don't think the patch can use parse_compress_specification() instead
of replace supports_compression(). The parsing simply determines if the
build has the library, it doesn't say if a particular tool was modified
to support the algorithm. I might build --with-zstd and yet pg_dump does
not support that algorithm yet.
Even after we add zstd to pg_dump, it's quite likely other compression
algorithms may not be supported by pg_dump from day 1.
I haven't looked at / tested the patch yet, but I wonder if you have any
thoughts regarding the size_t / int tweaks. I don't know what types zstd
library uses, how it reports errors etc.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2023-03-27 16:20 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
1 sibling, 0 replies; 46+ messages in thread
From: Tomas Vondra @ 2023-03-27 16:20 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; Jacob Champion <jchampion@timescale.com>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
Hi,
On 3/17/23 03:43, Tomas Vondra wrote:
>
> ...
>
>>> I'm a little suspicious of the replacement of supports_compression()
>>> with parse_compress_specification(). For example:
>>>
>>>> - errmsg = supports_compression(AH->compression_spec);
>>>> - if (errmsg)
>>>> + parse_compress_specification(AH->compression_spec.algorithm,
>>>> + NULL, &compress_spec);
>>>> + if (compress_spec.parse_error != NULL)
>>>> {
>>>> pg_log_warning("archive is compressed, but this installation does not support compression (%s
>>>> - errmsg);
>>>> - pg_free(errmsg);
>>>> + compress_spec.parse_error);
>>>> + pg_free(compress_spec.parse_error);
>>>> }
>>>
>>> The top-level error here is "does not support compression", but wouldn't
>>> a bad specification option with a supported compression method trip this
>>> path too?
>>
>> No - since the 2nd argument is passed as NULL, it just checks whether
>> the compression is supported. Maybe there ought to be a more
>> direct/clean way to do it. But up to now evidently nobody needed to do
>> that.
>>
>
> I don't think the patch can use parse_compress_specification() instead
> of replace supports_compression(). The parsing simply determines if the
> build has the library, it doesn't say if a particular tool was modified
> to support the algorithm. I might build --with-zstd and yet pg_dump does
> not support that algorithm yet.
>
> Even after we add zstd to pg_dump, it's quite likely other compression
> algorithms may not be supported by pg_dump from day 1.
>
>
> I haven't looked at / tested the patch yet, but I wonder if you have any
> thoughts regarding the size_t / int tweaks. I don't know what types zstd
> library uses, how it reports errors etc.
>
Any thoughts regarding my comments on removing supports_compression()?
Also, this patch needs a rebase to adopt it to the API changes from last
week. The sooner the better, considering we're getting fairly close to
the end of the CF and code freeze.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2023-03-27 17:28 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
1 sibling, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2023-03-27 17:28 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Fri, Mar 17, 2023 at 03:43:31AM +0100, Tomas Vondra wrote:
> On 3/16/23 05:50, Justin Pryzby wrote:
> > On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
> >> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <jchampion@timescale.com> wrote:
> >>> I did some smoke testing against zstd's GitHub release on Windows. To
> >>> build against it, I had to construct an import library, and put that
> >>> and the DLL into the `lib` folder expected by the MSVC scripts...
> >>> which makes me wonder if I've chosen a harder way than necessary?
> >>
> >> It looks like pg_dump's meson.build is missing dependencies on zstd
> >> (meson couldn't find the headers in the subproject without them).
> >
> > I saw that this was added for LZ4, but I hadn't added it for zstd since
> > I didn't run into an issue without it. Could you check that what I've
> > added works for your case ?
> >
> >>> Parallel zstd dumps seem to work as expected, in that the resulting
> >>> pg_restore output is identical to uncompressed dumps and nothing
> >>> explodes. I haven't inspected the threading implementation for safety
> >>> yet, as you mentioned.
> >>
> >> Hm. Best I can tell, the CloneArchive() machinery is supposed to be
> >> handling safety for this, by isolating each thread's state. I don't feel
> >> comfortable pronouncing this new addition safe or not, because I'm not
> >> sure I understand what the comments in the format-specific _Clone()
> >> callbacks are saying yet.
> >
> > My line of reasoning for unix is that pg_dump forks before any calls to
> > zstd. Nothing zstd does ought to affect the pg_dump layer. But that
> > doesn't apply to pg_dump under windows. This is an opened question. If
> > there's no solid answer, I could disable/ignore the option (maybe only
> > under windows).
>
> I may be missing something, but why would the patch affect this? Why
> would it even affect safety of the parallel dump? And I don't see any
> changes to the clone stuff ...
zstd supports using threads during compression, with -Z zstd:workers=N.
When unix forks, the child processes can't do anything to mess up the
state of the parent processes.
But windows pg_dump uses threads instead of forking, so it seems
possible that the pg_dump -j threads that then spawn zstd threads could
"leak threads" and break the main thread. I suspect there's no issue,
but we still ought to verify that before declaring it safe.
> > The function is first checking if it was passed a filename which already
> > has a suffix. And if not, it searches through a list of suffixes,
> > testing for an existing file with each suffix. The search with stat()
> > doesn't happen if it has a suffix. I'm having trouble seeing how the
> > hasSuffix() branch isn't dead code. Another opened question.
>
> AFAICS it's done this way because of this comment in pg_backup_directory
>
> * ...
> * ".gz" suffix is added to the filenames. The TOC files are never
> * compressed by pg_dump, however they are accepted with the .gz suffix
> * too, in case the user has manually compressed them with 'gzip'.
>
> I haven't tried, but I believe that if you manually compress the
> directory, it may hit this branch.
That would make sense, but when I tried, it didn't work like that.
The filenames were all uncompressed names. Maybe it worked differently
in an older release. Or maybe it changed during development of the
parallel-directory-dump patch and it's actually dead code.
This is rebased over the updated compression API.
It seems like I misunderstood something you said before, so now I put
back "supports_compression()".
--
Justin
Attachments:
[text/x-diff] 0001-pg_dump-zstd-compression.patch (29.9K, ../../ZCHSSv3DgxUa0f3L@telsasoft.com/2-0001-pg_dump-zstd-compression.patch)
download | inline diff:
From b4cf9df2f8672b012301d58ff83f2b0344de9e96 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 7 Jan 2023 15:45:06 -0600
Subject: [PATCH 1/3] pg_dump: zstd compression
Previously proposed at: 20201221194924.GI30237@telsasoft.com
---
doc/src/sgml/ref/pg_dump.sgml | 13 +-
src/bin/pg_dump/Makefile | 2 +
src/bin/pg_dump/compress_io.c | 62 ++-
src/bin/pg_dump/compress_zstd.c | 535 ++++++++++++++++++++++++++
src/bin/pg_dump/compress_zstd.h | 25 ++
src/bin/pg_dump/meson.build | 4 +-
src/bin/pg_dump/pg_backup_archiver.c | 9 +-
src/bin/pg_dump/pg_backup_directory.c | 2 +
src/bin/pg_dump/pg_dump.c | 16 +-
src/bin/pg_dump/t/002_pg_dump.pl | 79 +++-
src/tools/pginclude/cpluspluscheck | 1 +
11 files changed, 694 insertions(+), 54 deletions(-)
create mode 100644 src/bin/pg_dump/compress_zstd.c
create mode 100644 src/bin/pg_dump/compress_zstd.h
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index d6b1faa8042..62b3ed2dad6 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -330,8 +330,9 @@ PostgreSQL documentation
machine-readable format that <application>pg_restore</application>
can read. A directory format archive can be manipulated with
standard Unix tools; for example, files in an uncompressed archive
- can be compressed with the <application>gzip</application> or
- <application>lz4</application> tools.
+ can be compressed with the <application>gzip</application>,
+ <application>lz4</application>, or
+ <application>zstd</application> tools.
This format is compressed by default using <literal>gzip</literal>
and also supports parallel dumps.
</para>
@@ -655,7 +656,8 @@ PostgreSQL documentation
<para>
Specify the compression method and/or the compression level to use.
The compression method can be set to <literal>gzip</literal>,
- <literal>lz4</literal>, or <literal>none</literal> for no compression.
+ <literal>lz4</literal>, <literal>zstd</literal>,
+ or <literal>none</literal> for no compression.
A compression detail string can optionally be specified. If the
detail string is an integer, it specifies the compression level.
Otherwise, it should be a comma-separated list of items, each of the
@@ -676,8 +678,9 @@ PostgreSQL documentation
individual table-data segments, and the default is to compress using
<literal>gzip</literal> at a moderate level. For plain text output,
setting a nonzero compression level causes the entire output file to be compressed,
- as though it had been fed through <application>gzip</application> or
- <application>lz4</application>; but the default is not to compress.
+ as though it had been fed through <application>gzip</application>,
+ <application>lz4</application>, or <application>zstd</application>;
+ but the default is not to compress.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index eb8f59459a1..24de7593a6a 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -18,6 +18,7 @@ include $(top_builddir)/src/Makefile.global
export GZIP_PROGRAM=$(GZIP)
export LZ4
+export ZSTD
export with_icu
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
@@ -29,6 +30,7 @@ OBJS = \
compress_io.o \
compress_lz4.o \
compress_none.o \
+ compress_zstd.o \
dumputils.o \
parallel.o \
pg_backup_archiver.o \
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 0972a4f934a..b0f59c069c7 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -52,8 +52,8 @@
*
* InitDiscoverCompressFileHandle tries to infer the compression by the
* filename suffix. If the suffix is not yet known then it tries to simply
- * open the file and if it fails, it tries to open the same file with the .gz
- * suffix, and then again with the .lz4 suffix.
+ * open the file and if it fails, it tries to open the same file with
+ * compressed suffixes.
*
* IDENTIFICATION
* src/bin/pg_dump/compress_io.c
@@ -69,6 +69,7 @@
#include "compress_io.h"
#include "compress_lz4.h"
#include "compress_none.h"
+#include "compress_zstd.h"
#include "pg_backup_utils.h"
/*----------------------
@@ -77,7 +78,8 @@
*/
/*
- * Checks whether a compression algorithm is supported.
+ * Checks whether support for a compression algorithm is implemented in
+ * pg_dump/restore.
*
* On success returns NULL, otherwise returns a malloc'ed string which can be
* used by the caller in an error message.
@@ -98,6 +100,10 @@ supports_compression(const pg_compress_specification compression_spec)
if (algorithm == PG_COMPRESSION_LZ4)
supported = true;
#endif
+#ifdef USE_ZSTD
+ if (algorithm == PG_COMPRESSION_ZSTD)
+ supported = true;
+#endif
if (!supported)
return psprintf("this build does not support compression with %s",
@@ -130,6 +136,8 @@ AllocateCompressor(const pg_compress_specification compression_spec,
InitCompressorGzip(cs, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressorLZ4(cs, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressorZstd(cs, compression_spec);
return cs;
}
@@ -196,20 +204,30 @@ InitCompressFileHandle(const pg_compress_specification compression_spec)
InitCompressFileHandleGzip(CFH, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressFileHandleLZ4(CFH, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressFileHandleZstd(CFH, compression_spec);
return CFH;
}
+static bool
+check_compressed_file(const char *path, char **fname, char *ext)
+{
+ free_keep_errno(*fname);
+ *fname = psprintf("%s.%s", path, ext);
+ return (access(*fname, F_OK) == 0);
+}
+
/*
* Open a file for reading. 'path' is the file to open, and 'mode' should
* be either "r" or "rb".
*
* If the file at 'path' contains the suffix of a supported compression method,
- * currently this includes ".gz" and ".lz4", then this compression will be used
+ * currently this includes ".gz", ".lz4" and ".zst", then this compression will be used
* throughout. Otherwise the compression will be inferred by iteratively trying
* to open the file at 'path', first as is, then by appending known compression
* suffixes. So if you pass "foo" as 'path', this will open either "foo" or
- * "foo.gz" or "foo.lz4", trying in that order.
+ * "foo.{gz,lz4,zst}", trying in that order.
*
* On failure, return NULL with an error code in errno.
*/
@@ -231,34 +249,14 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode)
compression_spec.algorithm = PG_COMPRESSION_GZIP;
else
{
- bool exists;
-
- exists = (stat(path, &st) == 0);
- /* avoid unused warning if it is not built with compression */
- if (exists)
+ if (stat(path, &st) == 0)
compression_spec.algorithm = PG_COMPRESSION_NONE;
-#ifdef HAVE_LIBZ
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.gz", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_GZIP;
- }
-#endif
-#ifdef USE_LZ4
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.lz4", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_LZ4;
- }
-#endif
+ else if (check_compressed_file(path, &fname, "gz"))
+ compression_spec.algorithm = PG_COMPRESSION_GZIP;
+ else if (check_compressed_file(path, &fname, "lz4"))
+ compression_spec.algorithm = PG_COMPRESSION_LZ4;
+ else if (check_compressed_file(path, &fname, "zst"))
+ compression_spec.algorithm = PG_COMPRESSION_ZSTD;
}
CFH = InitCompressFileHandle(compression_spec);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
new file mode 100644
index 00000000000..944e786320a
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -0,0 +1,535 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_zstd.c
+ * Routines for archivers to write a Zstd compressed data stream.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/compress_zstd.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "pg_backup_utils.h"
+#include "compress_zstd.h"
+
+#ifndef USE_ZSTD
+
+void
+InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+#else
+
+#include <zstd.h>
+
+typedef struct ZstdCompressorState
+{
+ /* This is a normal file to which we read/write compressed data */
+ FILE *fp;
+
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+
+ /* pointer to a static string like from strerror(), for Zstd_write() */
+ const char *zstderror;
+} ZstdCompressorState;
+
+static ZSTD_CStream *_ZstdCStreamParams(pg_compress_specification compress);
+static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen);
+static void ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs);
+
+static void
+_Zstd_CCtx_setParam_or_die(ZSTD_CStream *cstream,
+ ZSTD_cParameter param, int value, char *paramname)
+{
+ size_t res;
+
+ res = ZSTD_CCtx_setParameter(cstream, param, value);
+ if (ZSTD_isError(res))
+ pg_fatal("could not set compression parameter: \"%s\": %s",
+ paramname, ZSTD_getErrorName(res));
+}
+
+/* Return a compression stream with parameters set per argument */
+static ZSTD_CStream *
+_ZstdCStreamParams(pg_compress_specification compress)
+{
+ ZSTD_CStream *cstream;
+
+ cstream = ZSTD_createCStream();
+ if (cstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ _Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_compressionLevel,
+ compress.level, "level");
+
+ if (compress.options & PG_COMPRESSION_OPTION_WORKERS)
+ _Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
+ compress.workers, "workers");
+
+ return cstream;
+}
+
+/* Helper function for WriteDataToArchiveZstd and EndCompressorZstd */
+static void
+_ZstdWriteCommon(ArchiveHandle *AH, CompressorState *cs, bool flush)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ /* Loop while there's any input or until flushed */
+ while (input->pos != input->size || flush)
+ {
+ size_t res;
+
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output,
+ input, flush ? ZSTD_e_end : ZSTD_e_continue);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ /*
+ * Extra paranoia: avoid zero-length chunks, since a zero length chunk
+ * is the EOF marker in the custom format. This should never happen
+ * but...
+ */
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ if (res == 0)
+ break;
+ }
+}
+
+static void
+EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ /* We expect that exactly one of readF/writeF is specified */
+ Assert((cs->readF == NULL) != (cs->writeF == NULL));
+
+ if (cs->readF != NULL)
+ {
+ Assert(zstdcs->cstream == NULL);
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ }
+ else if (cs->writeF != NULL)
+ {
+ Assert(zstdcs->dstream == NULL);
+ _ZstdWriteCommon(AH, cs, true);
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ pg_free(zstdcs);
+}
+
+static void
+WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ zstdcs->input.src = data;
+ zstdcs->input.size = dLen;
+ zstdcs->input.pos = 0;
+
+ _ZstdWriteCommon(AH, cs, false);
+}
+
+static void
+ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ size_t input_allocated_size = ZSTD_DStreamInSize();
+ size_t res;
+
+ for (;;)
+ {
+ size_t cnt;
+
+ /*
+ * Read compressed data. Note that readF can resize the buffer; the
+ * new size is tracked and used for future loops.
+ */
+ input->size = input_allocated_size;
+ cnt = cs->readF(AH, (char **) unconstify(void **, &input->src), &input->size);
+
+ /* ensure that readF didn't *shrink* the buffer */
+ Assert(input->size >= input_allocated_size);
+ input_allocated_size = input->size;
+ input->size = cnt;
+ input->pos = 0;
+
+ if (cnt == 0)
+ break;
+
+ /* Now decompress */
+ while (input->pos < input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ /* then write the decompressed data to the output handle */
+ ((char *) output->dst)[output->pos] = '\0';
+ ahwrite(output->dst, 1, output->pos, AH);
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+ }
+}
+
+/* Public routine that supports Zstd compressed data I/O */
+void
+InitCompressorZstd(CompressorState *cs,
+ const pg_compress_specification compression_spec)
+{
+ ZstdCompressorState *zstdcs;
+
+ cs->readData = ReadDataFromArchiveZstd;
+ cs->writeData = WriteDataToArchiveZstd;
+ cs->end = EndCompressorZstd;
+
+ cs->compression_spec = compression_spec;
+
+ zstdcs = (ZstdCompressorState *) pg_malloc0(sizeof(*zstdcs));
+ cs->private_data = zstdcs;
+
+ /* We expect that exactly one of readF/writeF is specified */
+ Assert((cs->readF == NULL) != (cs->writeF == NULL));
+
+ if (cs->readF != NULL)
+ {
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ zstdcs->input.size = ZSTD_DStreamInSize();
+ zstdcs->input.src = pg_malloc(zstdcs->input.size);
+
+ zstdcs->output.size = ZSTD_DStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size + 1);
+ }
+ else if (cs->writeF != NULL)
+ {
+ zstdcs->cstream = _ZstdCStreamParams(cs->compression_spec);
+
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size);
+ zstdcs->output.pos = 0;
+ }
+}
+
+/*
+ * Compressed stream API
+ */
+
+static bool
+Zstd_read(void *ptr, size_t size, size_t *rdsize, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t input_allocated_size = ZSTD_DStreamInSize();
+ size_t res,
+ cnt;
+
+ output->size = size;
+ output->dst = ptr;
+ output->pos = 0;
+
+ for (;;)
+ {
+ Assert(input->pos <= input->size);
+ Assert(input->size <= input_allocated_size);
+
+ /* If the input is completely consumed, start back at the beginning */
+ if (input->pos == input->size)
+ {
+ /* input->size is size produced by "fread" */
+ input->size = 0;
+ /* input->pos is position consumed by decompress */
+ input->pos = 0;
+ }
+
+ /* read compressed data if we must produce more input */
+ if (input->pos == input->size)
+ {
+ cnt = fread(unconstify(void *, input->src), 1, input_allocated_size, zstdcs->fp);
+ input->size = cnt;
+
+ Assert(cnt >= 0);
+ Assert(cnt <= input_allocated_size);
+
+ /* If we have no more input to consume, we're done */
+ if (cnt == 0)
+ break;
+ }
+
+ while (input->pos < input->size)
+ {
+ /* now decompress */
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ if (output->pos == output->size)
+ break; /* No more room for output */
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+
+ if (output->pos == output->size)
+ break; /* We read all the data that fits */
+ }
+
+ if (rdsize != NULL)
+ *rdsize = output->pos;
+
+ return true;
+}
+
+static bool
+Zstd_write(const void *ptr, size_t size, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t res,
+ cnt;
+
+ input->src = ptr;
+ input->size = size;
+ input->pos = 0;
+
+ /* Consume all input, to be flushed later */
+ while (input->pos != input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_continue);
+ if (ZSTD_isError(res))
+ {
+ zstdcs->zstderror = ZSTD_getErrorName(res);
+ return false;
+ }
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ {
+ zstdcs->zstderror = strerror(errno);
+ return false;
+ }
+ }
+
+ return size;
+}
+
+static int
+Zstd_getc(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ int ret;
+
+ if (CFH->read_func(&ret, 1, NULL, CFH) != 1)
+ {
+ if (feof(zstdcs->fp))
+ pg_fatal("could not read from input file: end of file");
+ else
+ pg_fatal("could not read from input file: %m");
+ }
+ return ret;
+}
+
+static char *
+Zstd_gets(char *buf, int len, CompressFileHandle *CFH)
+{
+ int i;
+
+ Assert(len > 0);
+
+ /*
+ * Read one byte at a time until newline or EOF. This is only used to read
+ * the list of LOs, and the I/O is buffered anyway.
+ */
+ for (i = 0; i < len - 1; ++i)
+ {
+ size_t readsz;
+
+ if (!CFH->read_func(&buf[i], 1, &readsz, CFH))
+ break;
+ if (readsz != 1)
+ break;
+ if (buf[i] == '\n')
+ {
+ ++i;
+ break;
+ }
+ }
+ buf[i] = '\0';
+ return i > 0 ? buf : NULL;
+}
+
+static bool
+Zstd_close(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ if (zstdcs->cstream)
+ {
+ size_t res,
+ cnt;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ /* Loop until the compression buffers are fully consumed */
+ for (;;)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_end);
+ if (ZSTD_isError(res))
+ {
+ zstdcs->zstderror = ZSTD_getErrorName(res);
+ return false;
+ }
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ {
+ zstdcs->zstderror = strerror(errno);
+ return false;
+ }
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ if (zstdcs->dstream)
+ {
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ }
+
+ if (fclose(zstdcs->fp) != 0)
+ return false;
+
+ pg_free(zstdcs);
+ return true;
+}
+
+static bool
+Zstd_eof(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ return feof(zstdcs->fp);
+}
+
+static bool
+Zstd_open(const char *path, int fd, const char *mode,
+ CompressFileHandle *CFH)
+{
+ FILE *fp;
+ ZstdCompressorState *zstdcs;
+
+ if (fd >= 0)
+ fp = fdopen(fd, mode);
+ else
+ fp = fopen(path, mode);
+
+ if (fp == NULL)
+ return false;
+
+ zstdcs = (ZstdCompressorState *) pg_malloc0(sizeof(*zstdcs));
+ CFH->private_data = zstdcs;
+ zstdcs->fp = fp;
+
+ if (mode[0] == 'r')
+ {
+ zstdcs->input.src = pg_malloc0(ZSTD_DStreamInSize());
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+ }
+ else if (mode[0] == 'w' || mode[0] == 'a')
+ {
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc0(zstdcs->output.size);
+ zstdcs->cstream = _ZstdCStreamParams(CFH->compression_spec);
+ if (zstdcs->cstream == NULL)
+ pg_fatal("could not initialize compression library");
+ }
+ else
+ pg_fatal("unhandled mode");
+
+ return true;
+}
+
+static bool
+Zstd_open_write(const char *path, const char *mode, CompressFileHandle *CFH)
+{
+ char fname[MAXPGPATH];
+
+ sprintf(fname, "%s.zst", path);
+ return CFH->open_func(fname, -1, mode, CFH);
+}
+
+static const char *
+Zstd_get_error(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ return zstdcs->zstderror;
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH,
+ const pg_compress_specification compression_spec)
+{
+ CFH->open_func = Zstd_open;
+ CFH->open_write_func = Zstd_open_write;
+ CFH->read_func = Zstd_read;
+ CFH->write_func = Zstd_write;
+ CFH->gets_func = Zstd_gets;
+ CFH->getc_func = Zstd_getc;
+ CFH->close_func = Zstd_close;
+ CFH->eof_func = Zstd_eof;
+ CFH->get_error_func = Zstd_get_error;
+
+ CFH->compression_spec = compression_spec;
+
+ CFH->private_data = NULL;
+}
+
+#endif /* USE_ZSTD */
diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h
new file mode 100644
index 00000000000..2aaa6b100b1
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_zstd.h
+ * Zstd interface to compress_io.c routines
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/compress_zstd.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef COMPRESS_ZSTD_H
+#define COMPRESS_ZSTD_H
+
+#include "compress_io.h"
+
+extern void InitCompressorZstd(CompressorState *cs,
+ const pg_compress_specification compression_spec);
+extern void InitCompressFileHandleZstd(CompressFileHandle *CFH,
+ const pg_compress_specification compression_spec);
+
+#endif /* COMPRESS_ZSTD_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index b2fb7ac77fd..9d59a106f36 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -5,6 +5,7 @@ pg_dump_common_sources = files(
'compress_io.c',
'compress_lz4.c',
'compress_none.c',
+ 'compress_zstd.c',
'dumputils.c',
'parallel.c',
'pg_backup_archiver.c',
@@ -19,7 +20,7 @@ pg_dump_common_sources = files(
pg_dump_common = static_library('libpgdump_common',
pg_dump_common_sources,
c_pch: pch_postgres_fe_h,
- dependencies: [frontend_code, libpq, lz4, zlib],
+ dependencies: [frontend_code, libpq, lz4, zlib, zstd],
kwargs: internal_lib_args,
)
@@ -90,6 +91,7 @@ tests += {
'env': {
'GZIP_PROGRAM': gzip.path(),
'LZ4': program_lz4.found() ? program_lz4.path() : '',
+ 'ZSTD': program_zstd.found() ? program_zstd.path() : '',
'with_icu': icu.found() ? 'yes' : 'no',
},
'tests': [
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index ab77e373e91..e8ee6b1ad86 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -2120,7 +2120,7 @@ _discoverArchiveFormat(ArchiveHandle *AH)
/*
* Check if the specified archive is a directory. If so, check if
- * there's a "toc.dat" (or "toc.dat.{gz,lz4}") file in it.
+ * there's a "toc.dat" (or "toc.dat.{gz,lz4,zst}") file in it.
*/
if (stat(AH->fSpec, &st) == 0 && S_ISDIR(st.st_mode))
{
@@ -2131,10 +2131,17 @@ _discoverArchiveFormat(ArchiveHandle *AH)
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.gz"))
return AH->format;
#endif
+
#ifdef USE_LZ4
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.lz4"))
return AH->format;
#endif
+
+#ifdef USE_ZSTD
+ if (_fileExistsInDirectory(AH->fSpec, "toc.dat.zst"))
+ return AH->format;
+#endif
+
pg_fatal("directory \"%s\" does not appear to be a valid archive (\"toc.dat\" does not exist)",
AH->fSpec);
fh = NULL; /* keep compiler quiet */
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index abaaa3b10e3..2177d5ff425 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -785,6 +785,8 @@ _PrepParallelRestore(ArchiveHandle *AH)
strlcat(fname, ".gz", sizeof(fname));
else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4)
strlcat(fname, ".lz4", sizeof(fname));
+ else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ strlcat(fname, ".zst", sizeof(fname));
if (stat(fname, &st) == 0)
te->dataLength = st.st_size;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index d62780a0880..1844100be8a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -61,6 +61,7 @@
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
+#include "compress_io.h"
#include "parallel.h"
#include "pg_backup_db.h"
#include "pg_backup_utils.h"
@@ -735,18 +736,9 @@ main(int argc, char **argv)
pg_fatal("invalid compression specification: %s",
error_detail);
- switch (compression_algorithm)
- {
- case PG_COMPRESSION_NONE:
- /* fallthrough */
- case PG_COMPRESSION_GZIP:
- /* fallthrough */
- case PG_COMPRESSION_LZ4:
- break;
- case PG_COMPRESSION_ZSTD:
- pg_fatal("compression with %s is not yet supported", "ZSTD");
- break;
- }
+ error_detail = supports_compression(compression_spec);
+ if (error_detail != NULL)
+ pg_fatal("%s", error_detail);
/*
* Custom and directory formats are compressed by default with gzip when
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index a22f27f300f..33435a0a421 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -54,8 +54,9 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir;
# those lines) to validate that part of the process.
my $supports_icu = ($ENV{with_icu} eq 'yes');
-my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1");
+my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
+my $supports_zstd = check_pg_config("#define USE_ZSTD 1");
my %pgdump_runs = (
binary_upgrade => {
@@ -213,6 +214,77 @@ my %pgdump_runs = (
},
},
+ compression_zstd_custom => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=custom',
+ '--compress=zstd', "--file=$tempdir/compression_zstd_custom.dump",
+ 'postgres',
+ ],
+ restore_cmd => [
+ 'pg_restore',
+ "--file=$tempdir/compression_zstd_custom.sql",
+ "$tempdir/compression_zstd_custom.dump",
+ ],
+ command_like => {
+ command => [
+ 'pg_restore',
+ '-l', "$tempdir/compression_zstd_custom.dump",
+ ],
+ expected => qr/Compression: zstd/,
+ name => 'data content is zstd compressed'
+ },
+ },
+
+ compression_zstd_dir => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--jobs=2',
+ '--format=directory', '--compress=zstd:1',
+ "--file=$tempdir/compression_zstd_dir", 'postgres',
+ ],
+ # Give coverage for manually compressed blob.toc files during
+ # restore.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-z', '-f', '--rm',
+ "$tempdir/compression_zstd_dir/blobs.toc",
+ "-o", "$tempdir/compression_zstd_dir/blobs.toc.zst",
+ ],
+ },
+ # Verify that data files were compressed
+ glob_patterns => [
+ "$tempdir/compression_zstd_dir/toc.dat",
+ "$tempdir/compression_zstd_dir/*.dat.zst",
+ ],
+ restore_cmd => [
+ 'pg_restore', '--jobs=2',
+ "--file=$tempdir/compression_zstd_dir.sql",
+ "$tempdir/compression_zstd_dir",
+ ],
+ },
+
+ compression_zstd_plain => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=plain', '--compress=zstd',
+ "--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
+ ],
+ # Decompress the generated file to run through the tests.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-d', '-f',
+ "$tempdir/compression_zstd_plain.sql.zst",
+ "-o", "$tempdir/compression_zstd_plain.sql",
+ ],
+ },
+ },
+
clean => {
dump_cmd => [
'pg_dump',
@@ -4625,10 +4697,11 @@ foreach my $run (sort keys %pgdump_runs)
my $test_key = $run;
my $run_db = 'postgres';
- # Skip command-level tests for gzip/lz4 if there is no support for it.
+ # Skip command-level tests for gzip/lz4/zstd if the tool is not supported
if ($pgdump_runs{$run}->{compile_option} &&
(($pgdump_runs{$run}->{compile_option} eq 'gzip' && !$supports_gzip) ||
- ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4)))
+ ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4) ||
+ ($pgdump_runs{$run}->{compile_option} eq 'zstd' && !$supports_zstd)))
{
note "$run: skipped due to no $pgdump_runs{$run}->{compile_option} support";
next;
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index 58039934756..10fb51585c9 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -154,6 +154,7 @@ do
test "$f" = src/bin/pg_dump/compress_io.h && continue
test "$f" = src/bin/pg_dump/compress_lz4.h && continue
test "$f" = src/bin/pg_dump/compress_none.h && continue
+ test "$f" = src/bin/pg_dump/compress_zstd.h && continue
test "$f" = src/bin/pg_dump/parallel.h && continue
test "$f" = src/bin/pg_dump/pg_backup_archiver.h && continue
test "$f" = src/bin/pg_dump/pg_dump.h && continue
--
2.34.1
[text/x-diff] 0002-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch (2.2K, ../../ZCHSSv3DgxUa0f3L@telsasoft.com/3-0002-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch)
download | inline diff:
From c59f35a4557e3f404732d9c1c04621cae9c92217 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 4 Jan 2023 21:21:53 -0600
Subject: [PATCH 2/3] TMP: pg_dump: use Zstd by default, for CI only
//-os-only: warnings
---
src/bin/pg_dump/pg_dump.c | 4 ++--
src/bin/pg_dump/t/002_pg_dump.pl | 14 +++++++-------
2 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 1844100be8a..fe5c7ec461b 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -747,8 +747,8 @@ main(int argc, char **argv)
if ((archiveFormat == archCustom || archiveFormat == archDirectory) &&
!user_compression_defined)
{
-#ifdef HAVE_LIBZ
- parse_compress_specification(PG_COMPRESSION_GZIP, NULL,
+#ifdef USE_ZSTD
+ parse_compress_specification(PG_COMPRESSION_ZSTD, NULL,
&compression_spec);
#else
/* Nothing to do in the default case */
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 33435a0a421..7a566db6614 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -384,10 +384,10 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_custom_format.dump", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
- name => 'data content is gzip-compressed by default if available',
+ name => 'data content is zstd-compressed by default if available',
},
},
@@ -409,16 +409,16 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_dir_format", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
name => 'data content is gzip-compressed by default',
},
glob_patterns => [
"$tempdir/defaults_dir_format/toc.dat",
"$tempdir/defaults_dir_format/blobs.toc",
- $supports_gzip ?
- "$tempdir/defaults_dir_format/*.dat.gz" :
+ $supports_zstd ?
+ "$tempdir/defaults_dir_format/*.dat.zst" :
"$tempdir/defaults_dir_format/*.dat",
],
},
--
2.34.1
[text/x-diff] 0003-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch (12.7K, ../../ZCHSSv3DgxUa0f3L@telsasoft.com/4-0003-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch)
download | inline diff:
From 7dd22d93ee24584060c061f7a682f371f1a1627a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 27 Mar 2022 11:55:01 -0500
Subject: [PATCH 3/3] zstd: support long distance mode in pg_dump/basebackup
First proposed here:
20220327205020.GM28503@telsasoft.com
---
doc/src/sgml/protocol.sgml | 10 +++-
doc/src/sgml/ref/pg_basebackup.sgml | 4 +-
doc/src/sgml/ref/pg_dump.sgml | 2 +
src/backend/backup/basebackup_zstd.c | 12 ++++
src/bin/pg_basebackup/bbstreamer_zstd.c | 13 +++++
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 9 ++-
src/bin/pg_dump/compress_zstd.c | 5 ++
src/bin/pg_dump/t/002_pg_dump.pl | 3 +-
src/bin/pg_verifybackup/t/008_untar.pl | 8 +++
src/bin/pg_verifybackup/t/010_client_untar.pl | 8 +++
src/common/compression.c | 57 ++++++++++++++++++-
src/include/common/compression.h | 2 +
12 files changed, 127 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 8b5e7b1ad7f..b11d9a6ba35 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2729,7 +2729,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
level. Otherwise, it should be a comma-separated list of items,
each of the form <replaceable>keyword</replaceable> or
<replaceable>keyword=value</replaceable>. Currently, the supported
- keywords are <literal>level</literal> and <literal>workers</literal>.
+ keywords are <literal>level</literal>, <literal>long</literal> and
+ <literal>workers</literal>.
</para>
<para>
@@ -2746,6 +2747,13 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<literal>3</literal>).
</para>
+ <para>
+ The <literal>long</literal> keyword enables long-distance matching
+ mode, for improved compression ratio, at the expense of higher memory
+ use. Long-distance mode is supported only for
+ <literal>zstd</literal>.
+ </para>
+
<para>
The <literal>workers</literal> keyword sets the number of threads
that should be used for parallel compression. Parallel compression
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index db3ad9cd5eb..79d3e657c32 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -424,8 +424,8 @@ PostgreSQL documentation
level. Otherwise, it should be a comma-separated list of items,
each of the form <literal>keyword</literal> or
<literal>keyword=value</literal>.
- Currently, the supported keywords are <literal>level</literal>
- and <literal>workers</literal>.
+ Currently, the supported keywords are <literal>level</literal>,
+ <literal>long</literal>, and <literal>workers</literal>.
The detail string cannot be used when the compression method
is specified as a plain integer.
</para>
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 62b3ed2dad6..35147acfca6 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -681,6 +681,8 @@ PostgreSQL documentation
as though it had been fed through <application>gzip</application>,
<application>lz4</application>, or <application>zstd</application>;
but the default is not to compress.
+ With zstd compression, <literal>long</literal> mode may improve the
+ compression ratio, at the cost of increased memory use.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/backend/backup/basebackup_zstd.c b/src/backend/backup/basebackup_zstd.c
index ac6cac178a0..1bb5820c884 100644
--- a/src/backend/backup/basebackup_zstd.c
+++ b/src/backend/backup/basebackup_zstd.c
@@ -118,6 +118,18 @@ bbsink_zstd_begin_backup(bbsink *sink)
compress->workers, ZSTD_getErrorName(ret)));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(mysink->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret)));
+ }
+
/*
* We need our own buffer, because we're going to pass different data to
* the next sink than what gets passed to us.
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index fe17d6df4ef..fba391e2a0f 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -106,6 +106,19 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, pg_compress_specification *comp
compress->workers, ZSTD_getErrorName(ret));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(streamer->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ {
+ pg_log_error("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret));
+ exit(1);
+ }
+ }
+
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
streamer->zstd_outBuf.size = streamer->base.bbs_buffer.maxlen;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index b60cb78a0d5..4d130a7f944 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -139,7 +139,14 @@ SKIP:
'gzip:workers=3',
'invalid compression specification: compression algorithm "gzip" does not accept a worker count',
'failure on worker count for gzip'
- ],);
+ ],
+ [
+ 'gzip:long',
+ 'invalid compression specification: compression algorithm "gzip" does not support long-distance mode',
+ 'failure on long mode for gzip'
+ ],
+ );
+
for my $cft (@compression_failure_tests)
{
my $cfail = quotemeta($client_fails . $cft->[1]);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
index 944e786320a..c2cb45a43a1 100644
--- a/src/bin/pg_dump/compress_zstd.c
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -84,6 +84,11 @@ _ZstdCStreamParams(pg_compress_specification compress)
_Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
compress.workers, "workers");
+ if (compress.options & PG_COMPRESSION_OPTION_LONG_DISTANCE)
+ _Zstd_CCtx_setParam_or_die(cstream,
+ ZSTD_c_enableLongDistanceMatching,
+ compress.long_distance, "long");
+
return cstream;
}
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 7a566db6614..77c655c6b44 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -267,11 +267,12 @@ my %pgdump_runs = (
],
},
+ # Exercise long mode for test coverage
compression_zstd_plain => {
test_key => 'compression',
compile_option => 'zstd',
dump_cmd => [
- 'pg_dump', '--format=plain', '--compress=zstd',
+ 'pg_dump', '--format=plain', '--compress=zstd:long',
"--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
],
# Decompress the generated file to run through the tests.
diff --git a/src/bin/pg_verifybackup/t/008_untar.pl b/src/bin/pg_verifybackup/t/008_untar.pl
index 3007bbe8556..05754bc8ec7 100644
--- a/src/bin/pg_verifybackup/t/008_untar.pl
+++ b/src/bin/pg_verifybackup/t/008_untar.pl
@@ -49,6 +49,14 @@ my @test_configuration = (
'decompress_program' => $ENV{'ZSTD'},
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => [ '--compress', 'server-zstd:level=1,long' ],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => ['-d'],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
});
for my $tc (@test_configuration)
diff --git a/src/bin/pg_verifybackup/t/010_client_untar.pl b/src/bin/pg_verifybackup/t/010_client_untar.pl
index f3aa0f59e29..ac51a174d14 100644
--- a/src/bin/pg_verifybackup/t/010_client_untar.pl
+++ b/src/bin/pg_verifybackup/t/010_client_untar.pl
@@ -50,6 +50,14 @@ my @test_configuration = (
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
},
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => ['--compress', 'client-zstd:level=1,long'],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => [ '-d' ],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
{
'compression_method' => 'parallel zstd',
'backup_flags' => [ '--compress', 'client-zstd:workers=3' ],
diff --git a/src/common/compression.c b/src/common/compression.c
index 2d3e56b4d62..713a77c292d 100644
--- a/src/common/compression.c
+++ b/src/common/compression.c
@@ -12,7 +12,7 @@
* Otherwise, a compression specification is a comma-separated list of items,
* each having the form keyword or keyword=value.
*
- * Currently, the only supported keywords are "level" and "workers".
+ * Currently, the supported keywords are "level", "long", and "workers".
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
*
@@ -38,6 +38,8 @@
static int expect_integer_value(char *keyword, char *value,
pg_compress_specification *result);
+static bool expect_boolean_value(char *keyword, char *value,
+ pg_compress_specification *result);
/*
* Look up a compression algorithm by name. Returns true and sets *algorithm
@@ -232,6 +234,11 @@ parse_compress_specification(pg_compress_algorithm algorithm, char *specificatio
result->workers = expect_integer_value(keyword, value, result);
result->options |= PG_COMPRESSION_OPTION_WORKERS;
}
+ else if (strcmp(keyword, "long") == 0)
+ {
+ result->long_distance = expect_boolean_value(keyword, value, result);
+ result->options |= PG_COMPRESSION_OPTION_LONG_DISTANCE;
+ }
else
result->parse_error =
psprintf(_("unrecognized compression option: \"%s\""), keyword);
@@ -289,6 +296,43 @@ expect_integer_value(char *keyword, char *value, pg_compress_specification *resu
return ivalue;
}
+/*
+ * Parse 'value' as an boolean and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1. The caller must check result->parse_error to determine if
+ * the call was successful.
+ *
+ * Valid values are: yes, no, on, off, 1, 0.
+ *
+ * Inspired by ParseVariableBool().
+ */
+static bool
+expect_boolean_value(char *keyword, char *value, pg_compress_specification *result)
+{
+ if (value == NULL)
+ return true;
+
+ if (pg_strcasecmp(value, "yes") == 0)
+ return true;
+ if (pg_strcasecmp(value, "on") == 0)
+ return true;
+ if (pg_strcasecmp(value, "1") == 0)
+ return true;
+
+ if (pg_strcasecmp(value, "no") == 0)
+ return false;
+ if (pg_strcasecmp(value, "off") == 0)
+ return false;
+ if (pg_strcasecmp(value, "0") == 0)
+ return false;
+
+ result->parse_error =
+ psprintf(_("value for compression option \"%s\" must be a boolean"),
+ keyword);
+ return false;
+}
+
/*
* Returns NULL if the compression specification string was syntactically
* valid and semantically sensible. Otherwise, returns an error message.
@@ -354,6 +398,17 @@ validate_compress_specification(pg_compress_specification *spec)
get_compress_algorithm_name(spec->algorithm));
}
+ /*
+ * Of the compression algorithms that we currently support, only zstd
+ * supports long-distance mode.
+ */
+ if ((spec->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0 &&
+ (spec->algorithm != PG_COMPRESSION_ZSTD))
+ {
+ return psprintf(_("compression algorithm \"%s\" does not support long-distance mode"),
+ get_compress_algorithm_name(spec->algorithm));
+ }
+
return NULL;
}
diff --git a/src/include/common/compression.h b/src/include/common/compression.h
index b48c173022e..6cf8cf396a8 100644
--- a/src/include/common/compression.h
+++ b/src/include/common/compression.h
@@ -27,6 +27,7 @@ typedef enum pg_compress_algorithm
} pg_compress_algorithm;
#define PG_COMPRESSION_OPTION_WORKERS (1 << 0)
+#define PG_COMPRESSION_OPTION_LONG_DISTANCE (1 << 1)
typedef struct pg_compress_specification
{
@@ -34,6 +35,7 @@ typedef struct pg_compress_specification
unsigned options; /* OR of PG_COMPRESSION_OPTION constants */
int level;
int workers;
+ int long_distance;
char *parse_error; /* NULL if parsing was OK, else message */
} pg_compress_specification;
--
2.34.1
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-03-28 16:23 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-28 18:03 ` Re: zstd compression for pg_dump Kirk Wolak <wolakk@gmail.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 2 replies; 46+ messages in thread
From: Tomas Vondra @ 2023-03-28 16:23 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On 3/27/23 19:28, Justin Pryzby wrote:
> On Fri, Mar 17, 2023 at 03:43:31AM +0100, Tomas Vondra wrote:
>> On 3/16/23 05:50, Justin Pryzby wrote:
>>> On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
>>>> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <jchampion@timescale.com> wrote:
>>>>> I did some smoke testing against zstd's GitHub release on Windows. To
>>>>> build against it, I had to construct an import library, and put that
>>>>> and the DLL into the `lib` folder expected by the MSVC scripts...
>>>>> which makes me wonder if I've chosen a harder way than necessary?
>>>>
>>>> It looks like pg_dump's meson.build is missing dependencies on zstd
>>>> (meson couldn't find the headers in the subproject without them).
>>>
>>> I saw that this was added for LZ4, but I hadn't added it for zstd since
>>> I didn't run into an issue without it. Could you check that what I've
>>> added works for your case ?
>>>
>>>>> Parallel zstd dumps seem to work as expected, in that the resulting
>>>>> pg_restore output is identical to uncompressed dumps and nothing
>>>>> explodes. I haven't inspected the threading implementation for safety
>>>>> yet, as you mentioned.
>>>>
>>>> Hm. Best I can tell, the CloneArchive() machinery is supposed to be
>>>> handling safety for this, by isolating each thread's state. I don't feel
>>>> comfortable pronouncing this new addition safe or not, because I'm not
>>>> sure I understand what the comments in the format-specific _Clone()
>>>> callbacks are saying yet.
>>>
>>> My line of reasoning for unix is that pg_dump forks before any calls to
>>> zstd. Nothing zstd does ought to affect the pg_dump layer. But that
>>> doesn't apply to pg_dump under windows. This is an opened question. If
>>> there's no solid answer, I could disable/ignore the option (maybe only
>>> under windows).
>>
>> I may be missing something, but why would the patch affect this? Why
>> would it even affect safety of the parallel dump? And I don't see any
>> changes to the clone stuff ...
>
> zstd supports using threads during compression, with -Z zstd:workers=N.
> When unix forks, the child processes can't do anything to mess up the
> state of the parent processes.
>
> But windows pg_dump uses threads instead of forking, so it seems
> possible that the pg_dump -j threads that then spawn zstd threads could
> "leak threads" and break the main thread. I suspect there's no issue,
> but we still ought to verify that before declaring it safe.
>
OK. I don't have access to a Windows machine so I can't test that. Is it
possible to disable the zstd threading, until we figure this out?
>>> The function is first checking if it was passed a filename which already
>>> has a suffix. And if not, it searches through a list of suffixes,
>>> testing for an existing file with each suffix. The search with stat()
>>> doesn't happen if it has a suffix. I'm having trouble seeing how the
>>> hasSuffix() branch isn't dead code. Another opened question.
>>
>> AFAICS it's done this way because of this comment in pg_backup_directory
>>
>> * ...
>> * ".gz" suffix is added to the filenames. The TOC files are never
>> * compressed by pg_dump, however they are accepted with the .gz suffix
>> * too, in case the user has manually compressed them with 'gzip'.
>>
>> I haven't tried, but I believe that if you manually compress the
>> directory, it may hit this branch.
>
> That would make sense, but when I tried, it didn't work like that.
> The filenames were all uncompressed names. Maybe it worked differently
> in an older release. Or maybe it changed during development of the
> parallel-directory-dump patch and it's actually dead code.
>
Interesting. Would be good to find out. I wonder if a little bit of
git-log digging could tell us more. Anyway, until we confirm it's dead
code, we should probably do what .gz does and have the same check for
.lz4 and .zst files.
> This is rebased over the updated compression API.
>
> It seems like I misunderstood something you said before, so now I put
> back "supports_compression()".
>
Thanks! I need to do a bit more testing and review, but it seems pretty
much RFC to me, assuming we can figure out what to do about threading.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2023-03-28 18:03 ` Kirk Wolak <wolakk@gmail.com>
2023-03-28 18:45 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-29 13:35 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
1 sibling, 2 replies; 46+ messages in thread
From: Kirk Wolak @ 2023-03-28 18:03 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Tue, Mar 28, 2023 at 12:23 PM Tomas Vondra <tomas.vondra@enterprisedb.com>
wrote:
> On 3/27/23 19:28, Justin Pryzby wrote:
> > On Fri, Mar 17, 2023 at 03:43:31AM +0100, Tomas Vondra wrote:
> >> On 3/16/23 05:50, Justin Pryzby wrote:
> >>> On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
> >>>> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <
> jchampion@timescale.com> wrote:
> >>>>> I did some smoke testing against zstd's GitHub release on Windows. To
> ...
> OK. I don't have access to a Windows machine so I can't test that. Is it
> possible to disable the zstd threading, until we figure this out?
>
> Thomas since I appear to be one of the few windows users (I use both), can
I help?
I can test pg_dump... for you, easy to do. I do about 5-10 pg_dumps a day
on windows while developing.
Also, I have an AWS instance I created to build PG/Win with readline back
in November.
I could give you access to that... (you are not the only person who has
made this statement here).
I've made such instances available for other Open Source developers, to
support them.
Obvi I would share connection credentials privately.
Regards, Kirk
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-28 18:03 ` Re: zstd compression for pg_dump Kirk Wolak <wolakk@gmail.com>
@ 2023-03-28 18:45 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
1 sibling, 0 replies; 46+ messages in thread
From: Tomas Vondra @ 2023-03-28 18:45 UTC (permalink / raw)
To: Kirk Wolak <wolakk@gmail.com>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On 3/28/23 20:03, Kirk Wolak wrote:
> On Tue, Mar 28, 2023 at 12:23 PM Tomas Vondra
> <tomas.vondra@enterprisedb.com <mailto:tomas.vondra@enterprisedb.com>>
> wrote:
>
> On 3/27/23 19:28, Justin Pryzby wrote:
> > On Fri, Mar 17, 2023 at 03:43:31AM +0100, Tomas Vondra wrote:
> >> On 3/16/23 05:50, Justin Pryzby wrote:
> >>> On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
> >>>> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion
> <jchampion@timescale.com <mailto:jchampion@timescale.com>> wrote:
> >>>>> I did some smoke testing against zstd's GitHub release on
> Windows. To
> ...
> OK. I don't have access to a Windows machine so I can't test that. Is it
> possible to disable the zstd threading, until we figure this out?
>
> Thomas since I appear to be one of the few windows users (I use both),
> can I help?
> I can test pg_dump... for you, easy to do. I do about 5-10 pg_dumps a
> day on windows while developing.
>
Perhaps. But I'll leave the details up to Justin - it's his patch, and
I'm not sure how to verify the threading is OK.
I'd try applying this patch, build with --with-zstd and then run the
pg_dump TAP tests, and perhaps do some manual tests.
And perhaps do the same for --with-lz4 - there's a thread [1] suggesting
we don't detect lz4 stuff on Windows, so the TAP tests do nothing.
https://www.postgresql.org/message-id/ZAjL96N9ZW84U59p@msg.df7cb.de
> Also, I have an AWS instance I created to build PG/Win with readline
> back in November.
> I could give you access to that... (you are not the only person who has
> made this statement here).
> I've made such instances available for other Open Source developers, to
> support them.
>
> Obvi I would share connection credentials privately.
>
I'd rather leave the Windows stuff up to someone with more experience
with that platform. I have plenty of other stuff on my plate atm.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-28 18:03 ` Re: zstd compression for pg_dump Kirk Wolak <wolakk@gmail.com>
@ 2023-03-29 13:35 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-29 15:10 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
1 sibling, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2023-03-29 13:35 UTC (permalink / raw)
To: Kirk Wolak <wolakk@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>; Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Tue, Mar 28, 2023 at 02:03:49PM -0400, Kirk Wolak wrote:
> On Tue, Mar 28, 2023 at 12:23 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
> > On 3/27/23 19:28, Justin Pryzby wrote:
> > > On Fri, Mar 17, 2023 at 03:43:31AM +0100, Tomas Vondra wrote:
> > >> On 3/16/23 05:50, Justin Pryzby wrote:
> > >>> On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
> > >>>> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <jchampion@timescale.com> wrote:
> > >>>>> I did some smoke testing against zstd's GitHub release on Windows. To
> > ...
> > OK. I don't have access to a Windows machine so I can't test that. Is it
> > possible to disable the zstd threading, until we figure this out?
>
> Thomas since I appear to be one of the few windows users (I use both), can I help?
> I can test pg_dump... for you, easy to do. I do about 5-10 pg_dumps a day
> on windows while developing.
It'd be great if you'd exercise this and other changes to
pg_dump/restore. Tomas just pushed a bugfix, so be sure to "git pull"
before testing, or else you might rediscover the bug.
If you have a zstd library with thread support, you could test with
-Z zstd:workers=3. But I think threads aren't enabled in the common
libzstd packages. Jacob figured out how to compile libzstd easily using
"meson wraps" - but I don't know the details.
--
Justin
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-28 18:03 ` Re: zstd compression for pg_dump Kirk Wolak <wolakk@gmail.com>
2023-03-29 13:35 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-03-29 15:10 ` Jacob Champion <jchampion@timescale.com>
0 siblings, 0 replies; 46+ messages in thread
From: Jacob Champion @ 2023-03-29 15:10 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Kirk Wolak <wolakk@gmail.com>; Tomas Vondra <tomas.vondra@enterprisedb.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Wed, Mar 29, 2023 at 6:35 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> If you have a zstd library with thread support, you could test with
> -Z zstd:workers=3. But I think threads aren't enabled in the common
> libzstd packages. Jacob figured out how to compile libzstd easily using
> "meson wraps" - but I don't know the details.
From the source root,
$ mkdir subprojects
$ meson wrap install zstd
From then on, Meson was pretty automagical about it during the ninja
build. The subproject's settings are themselves inspectable and
settable via `meson configure`:
$ meson configure -Dzstd:<option>=<value>
--Jacob
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2023-03-31 23:16 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
1 sibling, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2023-03-31 23:16 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Tue, Mar 28, 2023 at 06:23:26PM +0200, Tomas Vondra wrote:
> On 3/27/23 19:28, Justin Pryzby wrote:
> > On Fri, Mar 17, 2023 at 03:43:31AM +0100, Tomas Vondra wrote:
> >> On 3/16/23 05:50, Justin Pryzby wrote:
> >>> On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
> >>>> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <jchampion@timescale.com> wrote:
> >>>>> I did some smoke testing against zstd's GitHub release on Windows. To
> >>>>> build against it, I had to construct an import library, and put that
> >>>>> and the DLL into the `lib` folder expected by the MSVC scripts...
> >>>>> which makes me wonder if I've chosen a harder way than necessary?
> >>>>
> >>>> It looks like pg_dump's meson.build is missing dependencies on zstd
> >>>> (meson couldn't find the headers in the subproject without them).
> >>>
> >>> I saw that this was added for LZ4, but I hadn't added it for zstd since
> >>> I didn't run into an issue without it. Could you check that what I've
> >>> added works for your case ?
> >>>
> >>>>> Parallel zstd dumps seem to work as expected, in that the resulting
> >>>>> pg_restore output is identical to uncompressed dumps and nothing
> >>>>> explodes. I haven't inspected the threading implementation for safety
> >>>>> yet, as you mentioned.
> >>>>
> >>>> Hm. Best I can tell, the CloneArchive() machinery is supposed to be
> >>>> handling safety for this, by isolating each thread's state. I don't feel
> >>>> comfortable pronouncing this new addition safe or not, because I'm not
> >>>> sure I understand what the comments in the format-specific _Clone()
> >>>> callbacks are saying yet.
> >>>
> >>> My line of reasoning for unix is that pg_dump forks before any calls to
> >>> zstd. Nothing zstd does ought to affect the pg_dump layer. But that
> >>> doesn't apply to pg_dump under windows. This is an opened question. If
> >>> there's no solid answer, I could disable/ignore the option (maybe only
> >>> under windows).
> >>
> >> I may be missing something, but why would the patch affect this? Why
> >> would it even affect safety of the parallel dump? And I don't see any
> >> changes to the clone stuff ...
> >
> > zstd supports using threads during compression, with -Z zstd:workers=N.
> > When unix forks, the child processes can't do anything to mess up the
> > state of the parent processes.
> >
> > But windows pg_dump uses threads instead of forking, so it seems
> > possible that the pg_dump -j threads that then spawn zstd threads could
> > "leak threads" and break the main thread. I suspect there's no issue,
> > but we still ought to verify that before declaring it safe.
>
> OK. I don't have access to a Windows machine so I can't test that. Is it
> possible to disable the zstd threading, until we figure this out?
I think that's what's best. I made it issue a warning if "workers" was
specified. It could also be an error, or just ignored.
I considered disabling workers only for windows, but realized that I
haven't tested with threads myself - my local zstd package is compiled
without threading, and I remember having some issue recompiling it with
threading. Jacob's recipe for using meson wraps works well, but it
still seems better to leave it as a future feature. I used that recipe
to enabled zstd with threading on CI (except for linux/autoconf).
> >>> The function is first checking if it was passed a filename which already
> >>> has a suffix. And if not, it searches through a list of suffixes,
> >>> testing for an existing file with each suffix. The search with stat()
> >>> doesn't happen if it has a suffix. I'm having trouble seeing how the
> >>> hasSuffix() branch isn't dead code. Another opened question.
> >>
> >> AFAICS it's done this way because of this comment in pg_backup_directory
> >>
> >> * ...
> >> * ".gz" suffix is added to the filenames. The TOC files are never
> >> * compressed by pg_dump, however they are accepted with the .gz suffix
> >> * too, in case the user has manually compressed them with 'gzip'.
> >>
> >> I haven't tried, but I believe that if you manually compress the
> >> directory, it may hit this branch.
> >
> > That would make sense, but when I tried, it didn't work like that.
> > The filenames were all uncompressed names. Maybe it worked differently
> > in an older release. Or maybe it changed during development of the
> > parallel-directory-dump patch and it's actually dead code.
>
> Interesting. Would be good to find out. I wonder if a little bit of
> git-log digging could tell us more. Anyway, until we confirm it's dead
> code, we should probably do what .gz does and have the same check for
> .lz4 and .zst files.
I found that hasSuffix() and cfopen() originated in the refactored patch
Heikki's sent here; there's no history beyond that.
https://www.postgresql.org/message-id/4D3954C7.9060503%40enterprisedb.com
The patch published there appends the .gz within cfopen(), and the
caller writes into the TOC the filename without ".gz". It seems like
maybe a few hours prior, Heikki may have been appending the ".gz" suffix
in the caller, and then writing the TOC with filename.gz.
The only way I've been able to get a "filename.gz" passed to hasSuffix
is to write a directory-format dump, with LOs, and without compression,
and then compress the blobs with "gzip", and *also* edit the blobs.toc
file to say ".gz" (which isn't necessary since, if the original file
isn't found, the restore would search for files with compressed
suffixes).
So .. it's not *technically* unreachable, but I can't see why it'd be
useful to support editing the *content* of the blob TOC (other than
compressing it). I might give some weight to the idea if it were also
possible to edit the non-blob TOC; but, it's a binary file, so no.
For now, I made the change to make zstd and lz4 to behave the same here
as .gz, unless Heikki has a memory or a git reflog going back far enough
to further support the idea that the code path isn't useful.
I'm going to set the patch as RFC as a hint to anyone who would want to
make a final review.
--
Justin
Attachments:
[text/x-diff] 0001-pg_dump-zstd-compression.patch (30.1K, ../../ZCdpzywVGl0BLan2@telsasoft.com/2-0001-pg_dump-zstd-compression.patch)
download | inline diff:
From a4d2f22d98c16e16c718733f30d71dfb0e3adfe2 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 7 Jan 2023 15:45:06 -0600
Subject: [PATCH 1/4] pg_dump: zstd compression
Previously proposed at: 20201221194924.GI30237@telsasoft.com
---
doc/src/sgml/ref/pg_dump.sgml | 13 +-
src/bin/pg_dump/Makefile | 2 +
src/bin/pg_dump/compress_io.c | 66 ++--
src/bin/pg_dump/compress_zstd.c | 530 ++++++++++++++++++++++++++
src/bin/pg_dump/compress_zstd.h | 25 ++
src/bin/pg_dump/meson.build | 4 +-
src/bin/pg_dump/pg_backup_archiver.c | 9 +-
src/bin/pg_dump/pg_backup_directory.c | 2 +
src/bin/pg_dump/pg_dump.c | 20 +-
src/bin/pg_dump/t/002_pg_dump.pl | 79 +++-
src/tools/pginclude/cpluspluscheck | 1 +
11 files changed, 697 insertions(+), 54 deletions(-)
create mode 100644 src/bin/pg_dump/compress_zstd.c
create mode 100644 src/bin/pg_dump/compress_zstd.h
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 77299878e02..8de38e0fd0d 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -330,8 +330,9 @@ PostgreSQL documentation
machine-readable format that <application>pg_restore</application>
can read. A directory format archive can be manipulated with
standard Unix tools; for example, files in an uncompressed archive
- can be compressed with the <application>gzip</application> or
- <application>lz4</application> tools.
+ can be compressed with the <application>gzip</application>,
+ <application>lz4</application>, or
+ <application>zstd</application> tools.
This format is compressed by default using <literal>gzip</literal>
and also supports parallel dumps.
</para>
@@ -655,7 +656,8 @@ PostgreSQL documentation
<para>
Specify the compression method and/or the compression level to use.
The compression method can be set to <literal>gzip</literal>,
- <literal>lz4</literal>, or <literal>none</literal> for no compression.
+ <literal>lz4</literal>, <literal>zstd</literal>,
+ or <literal>none</literal> for no compression.
A compression detail string can optionally be specified. If the
detail string is an integer, it specifies the compression level.
Otherwise, it should be a comma-separated list of items, each of the
@@ -676,8 +678,9 @@ PostgreSQL documentation
individual table-data segments, and the default is to compress using
<literal>gzip</literal> at a moderate level. For plain text output,
setting a nonzero compression level causes the entire output file to be compressed,
- as though it had been fed through <application>gzip</application> or
- <application>lz4</application>; but the default is not to compress.
+ as though it had been fed through <application>gzip</application>,
+ <application>lz4</application>, or <application>zstd</application>;
+ but the default is not to compress.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index eb8f59459a1..24de7593a6a 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -18,6 +18,7 @@ include $(top_builddir)/src/Makefile.global
export GZIP_PROGRAM=$(GZIP)
export LZ4
+export ZSTD
export with_icu
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
@@ -29,6 +30,7 @@ OBJS = \
compress_io.o \
compress_lz4.o \
compress_none.o \
+ compress_zstd.o \
dumputils.o \
parallel.o \
pg_backup_archiver.o \
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 0972a4f934a..4f06bb024f9 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -52,8 +52,8 @@
*
* InitDiscoverCompressFileHandle tries to infer the compression by the
* filename suffix. If the suffix is not yet known then it tries to simply
- * open the file and if it fails, it tries to open the same file with the .gz
- * suffix, and then again with the .lz4 suffix.
+ * open the file and if it fails, it tries to open the same file with
+ * compressed suffixes.
*
* IDENTIFICATION
* src/bin/pg_dump/compress_io.c
@@ -69,6 +69,7 @@
#include "compress_io.h"
#include "compress_lz4.h"
#include "compress_none.h"
+#include "compress_zstd.h"
#include "pg_backup_utils.h"
/*----------------------
@@ -77,7 +78,8 @@
*/
/*
- * Checks whether a compression algorithm is supported.
+ * Checks whether support for a compression algorithm is implemented in
+ * pg_dump/restore.
*
* On success returns NULL, otherwise returns a malloc'ed string which can be
* used by the caller in an error message.
@@ -98,6 +100,10 @@ supports_compression(const pg_compress_specification compression_spec)
if (algorithm == PG_COMPRESSION_LZ4)
supported = true;
#endif
+#ifdef USE_ZSTD
+ if (algorithm == PG_COMPRESSION_ZSTD)
+ supported = true;
+#endif
if (!supported)
return psprintf("this build does not support compression with %s",
@@ -130,6 +136,8 @@ AllocateCompressor(const pg_compress_specification compression_spec,
InitCompressorGzip(cs, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressorLZ4(cs, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressorZstd(cs, compression_spec);
return cs;
}
@@ -196,20 +204,30 @@ InitCompressFileHandle(const pg_compress_specification compression_spec)
InitCompressFileHandleGzip(CFH, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressFileHandleLZ4(CFH, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressFileHandleZstd(CFH, compression_spec);
return CFH;
}
+static bool
+check_compressed_file(const char *path, char **fname, char *ext)
+{
+ free_keep_errno(*fname);
+ *fname = psprintf("%s.%s", path, ext);
+ return (access(*fname, F_OK) == 0);
+}
+
/*
* Open a file for reading. 'path' is the file to open, and 'mode' should
* be either "r" or "rb".
*
* If the file at 'path' contains the suffix of a supported compression method,
- * currently this includes ".gz" and ".lz4", then this compression will be used
+ * currently this includes ".gz", ".lz4" and ".zst", then this compression will be used
* throughout. Otherwise the compression will be inferred by iteratively trying
* to open the file at 'path', first as is, then by appending known compression
* suffixes. So if you pass "foo" as 'path', this will open either "foo" or
- * "foo.gz" or "foo.lz4", trying in that order.
+ * "foo.{gz,lz4,zst}", trying in that order.
*
* On failure, return NULL with an error code in errno.
*/
@@ -229,36 +247,20 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode)
if (hasSuffix(fname, ".gz"))
compression_spec.algorithm = PG_COMPRESSION_GZIP;
+ else if (hasSuffix(fname, ".lz4"))
+ compression_spec.algorithm = PG_COMPRESSION_LZ4;
+ else if (hasSuffix(fname, ".zst"))
+ compression_spec.algorithm = PG_COMPRESSION_ZSTD;
else
{
- bool exists;
-
- exists = (stat(path, &st) == 0);
- /* avoid unused warning if it is not built with compression */
- if (exists)
+ if (stat(path, &st) == 0)
compression_spec.algorithm = PG_COMPRESSION_NONE;
-#ifdef HAVE_LIBZ
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.gz", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_GZIP;
- }
-#endif
-#ifdef USE_LZ4
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.lz4", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_LZ4;
- }
-#endif
+ else if (check_compressed_file(path, &fname, "gz"))
+ compression_spec.algorithm = PG_COMPRESSION_GZIP;
+ else if (check_compressed_file(path, &fname, "lz4"))
+ compression_spec.algorithm = PG_COMPRESSION_LZ4;
+ else if (check_compressed_file(path, &fname, "zst"))
+ compression_spec.algorithm = PG_COMPRESSION_ZSTD;
}
CFH = InitCompressFileHandle(compression_spec);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
new file mode 100644
index 00000000000..c19d5262943
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -0,0 +1,530 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_zstd.c
+ * Routines for archivers to write a Zstd compressed data stream.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/compress_zstd.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "pg_backup_utils.h"
+#include "compress_zstd.h"
+
+#ifndef USE_ZSTD
+
+void
+InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+#else
+
+#include <zstd.h>
+
+typedef struct ZstdCompressorState
+{
+ /* This is a normal file to which we read/write compressed data */
+ FILE *fp;
+
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+
+ /* pointer to a static string like from strerror(), for Zstd_write() */
+ const char *zstderror;
+} ZstdCompressorState;
+
+static ZSTD_CStream *_ZstdCStreamParams(pg_compress_specification compress);
+static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen);
+static void ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs);
+
+static void
+_Zstd_CCtx_setParam_or_die(ZSTD_CStream *cstream,
+ ZSTD_cParameter param, int value, char *paramname)
+{
+ size_t res;
+
+ res = ZSTD_CCtx_setParameter(cstream, param, value);
+ if (ZSTD_isError(res))
+ pg_fatal("could not set compression parameter: \"%s\": %s",
+ paramname, ZSTD_getErrorName(res));
+}
+
+/* Return a compression stream with parameters set per argument */
+static ZSTD_CStream *
+_ZstdCStreamParams(pg_compress_specification compress)
+{
+ ZSTD_CStream *cstream;
+
+ cstream = ZSTD_createCStream();
+ if (cstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ _Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_compressionLevel,
+ compress.level, "level");
+
+ return cstream;
+}
+
+/* Helper function for WriteDataToArchiveZstd and EndCompressorZstd */
+static void
+_ZstdWriteCommon(ArchiveHandle *AH, CompressorState *cs, bool flush)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ /* Loop while there's any input or until flushed */
+ while (input->pos != input->size || flush)
+ {
+ size_t res;
+
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output,
+ input, flush ? ZSTD_e_end : ZSTD_e_continue);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ /*
+ * Extra paranoia: avoid zero-length chunks, since a zero length chunk
+ * is the EOF marker in the custom format. This should never happen
+ * but...
+ */
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ if (res == 0)
+ break;
+ }
+}
+
+static void
+EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ /* We expect that exactly one of readF/writeF is specified */
+ Assert((cs->readF == NULL) != (cs->writeF == NULL));
+
+ if (cs->readF != NULL)
+ {
+ Assert(zstdcs->cstream == NULL);
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ }
+ else if (cs->writeF != NULL)
+ {
+ Assert(zstdcs->dstream == NULL);
+ _ZstdWriteCommon(AH, cs, true);
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ pg_free(zstdcs);
+}
+
+static void
+WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ zstdcs->input.src = data;
+ zstdcs->input.size = dLen;
+ zstdcs->input.pos = 0;
+
+ _ZstdWriteCommon(AH, cs, false);
+}
+
+static void
+ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ size_t input_allocated_size = ZSTD_DStreamInSize();
+ size_t res;
+
+ for (;;)
+ {
+ size_t cnt;
+
+ /*
+ * Read compressed data. Note that readF can resize the buffer; the
+ * new size is tracked and used for future loops.
+ */
+ input->size = input_allocated_size;
+ cnt = cs->readF(AH, (char **) unconstify(void **, &input->src), &input->size);
+
+ /* ensure that readF didn't *shrink* the buffer */
+ Assert(input->size >= input_allocated_size);
+ input_allocated_size = input->size;
+ input->size = cnt;
+ input->pos = 0;
+
+ if (cnt == 0)
+ break;
+
+ /* Now decompress */
+ while (input->pos < input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ /* then write the decompressed data to the output handle */
+ ((char *) output->dst)[output->pos] = '\0';
+ ahwrite(output->dst, 1, output->pos, AH);
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+ }
+}
+
+/* Public routine that supports Zstd compressed data I/O */
+void
+InitCompressorZstd(CompressorState *cs,
+ const pg_compress_specification compression_spec)
+{
+ ZstdCompressorState *zstdcs;
+
+ cs->readData = ReadDataFromArchiveZstd;
+ cs->writeData = WriteDataToArchiveZstd;
+ cs->end = EndCompressorZstd;
+
+ cs->compression_spec = compression_spec;
+
+ zstdcs = (ZstdCompressorState *) pg_malloc0(sizeof(*zstdcs));
+ cs->private_data = zstdcs;
+
+ /* We expect that exactly one of readF/writeF is specified */
+ Assert((cs->readF == NULL) != (cs->writeF == NULL));
+
+ if (cs->readF != NULL)
+ {
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ zstdcs->input.size = ZSTD_DStreamInSize();
+ zstdcs->input.src = pg_malloc(zstdcs->input.size);
+
+ zstdcs->output.size = ZSTD_DStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size + 1);
+ }
+ else if (cs->writeF != NULL)
+ {
+ zstdcs->cstream = _ZstdCStreamParams(cs->compression_spec);
+
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size);
+ zstdcs->output.pos = 0;
+ }
+}
+
+/*
+ * Compressed stream API
+ */
+
+static bool
+Zstd_read(void *ptr, size_t size, size_t *rdsize, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t input_allocated_size = ZSTD_DStreamInSize();
+ size_t res,
+ cnt;
+
+ output->size = size;
+ output->dst = ptr;
+ output->pos = 0;
+
+ for (;;)
+ {
+ Assert(input->pos <= input->size);
+ Assert(input->size <= input_allocated_size);
+
+ /* If the input is completely consumed, start back at the beginning */
+ if (input->pos == input->size)
+ {
+ /* input->size is size produced by "fread" */
+ input->size = 0;
+ /* input->pos is position consumed by decompress */
+ input->pos = 0;
+ }
+
+ /* read compressed data if we must produce more input */
+ if (input->pos == input->size)
+ {
+ cnt = fread(unconstify(void *, input->src), 1, input_allocated_size, zstdcs->fp);
+ input->size = cnt;
+
+ Assert(cnt <= input_allocated_size);
+
+ /* If we have no more input to consume, we're done */
+ if (cnt == 0)
+ break;
+ }
+
+ while (input->pos < input->size)
+ {
+ /* now decompress */
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ if (output->pos == output->size)
+ break; /* No more room for output */
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+
+ if (output->pos == output->size)
+ break; /* We read all the data that fits */
+ }
+
+ if (rdsize != NULL)
+ *rdsize = output->pos;
+
+ return true;
+}
+
+static bool
+Zstd_write(const void *ptr, size_t size, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t res,
+ cnt;
+
+ input->src = ptr;
+ input->size = size;
+ input->pos = 0;
+
+ /* Consume all input, to be flushed later */
+ while (input->pos != input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_continue);
+ if (ZSTD_isError(res))
+ {
+ zstdcs->zstderror = ZSTD_getErrorName(res);
+ return false;
+ }
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ {
+ zstdcs->zstderror = strerror(errno);
+ return false;
+ }
+ }
+
+ return size;
+}
+
+static int
+Zstd_getc(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ int ret;
+
+ if (CFH->read_func(&ret, 1, NULL, CFH) != 1)
+ {
+ if (feof(zstdcs->fp))
+ pg_fatal("could not read from input file: end of file");
+ else
+ pg_fatal("could not read from input file: %m");
+ }
+ return ret;
+}
+
+static char *
+Zstd_gets(char *buf, int len, CompressFileHandle *CFH)
+{
+ int i;
+
+ Assert(len > 0);
+
+ /*
+ * Read one byte at a time until newline or EOF. This is only used to read
+ * the list of LOs, and the I/O is buffered anyway.
+ */
+ for (i = 0; i < len - 1; ++i)
+ {
+ size_t readsz;
+
+ if (!CFH->read_func(&buf[i], 1, &readsz, CFH))
+ break;
+ if (readsz != 1)
+ break;
+ if (buf[i] == '\n')
+ {
+ ++i;
+ break;
+ }
+ }
+ buf[i] = '\0';
+ return i > 0 ? buf : NULL;
+}
+
+static bool
+Zstd_close(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ if (zstdcs->cstream)
+ {
+ size_t res,
+ cnt;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ /* Loop until the compression buffers are fully consumed */
+ for (;;)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_end);
+ if (ZSTD_isError(res))
+ {
+ zstdcs->zstderror = ZSTD_getErrorName(res);
+ return false;
+ }
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ {
+ zstdcs->zstderror = strerror(errno);
+ return false;
+ }
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ if (zstdcs->dstream)
+ {
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ }
+
+ if (fclose(zstdcs->fp) != 0)
+ return false;
+
+ pg_free(zstdcs);
+ return true;
+}
+
+static bool
+Zstd_eof(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ return feof(zstdcs->fp);
+}
+
+static bool
+Zstd_open(const char *path, int fd, const char *mode,
+ CompressFileHandle *CFH)
+{
+ FILE *fp;
+ ZstdCompressorState *zstdcs;
+
+ if (fd >= 0)
+ fp = fdopen(fd, mode);
+ else
+ fp = fopen(path, mode);
+
+ if (fp == NULL)
+ return false;
+
+ zstdcs = (ZstdCompressorState *) pg_malloc0(sizeof(*zstdcs));
+ CFH->private_data = zstdcs;
+ zstdcs->fp = fp;
+
+ if (mode[0] == 'r')
+ {
+ zstdcs->input.src = pg_malloc0(ZSTD_DStreamInSize());
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+ }
+ else if (mode[0] == 'w' || mode[0] == 'a')
+ {
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc0(zstdcs->output.size);
+ zstdcs->cstream = _ZstdCStreamParams(CFH->compression_spec);
+ if (zstdcs->cstream == NULL)
+ pg_fatal("could not initialize compression library");
+ }
+ else
+ pg_fatal("unhandled mode");
+
+ return true;
+}
+
+static bool
+Zstd_open_write(const char *path, const char *mode, CompressFileHandle *CFH)
+{
+ char fname[MAXPGPATH];
+
+ sprintf(fname, "%s.zst", path);
+ return CFH->open_func(fname, -1, mode, CFH);
+}
+
+static const char *
+Zstd_get_error(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ return zstdcs->zstderror;
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH,
+ const pg_compress_specification compression_spec)
+{
+ CFH->open_func = Zstd_open;
+ CFH->open_write_func = Zstd_open_write;
+ CFH->read_func = Zstd_read;
+ CFH->write_func = Zstd_write;
+ CFH->gets_func = Zstd_gets;
+ CFH->getc_func = Zstd_getc;
+ CFH->close_func = Zstd_close;
+ CFH->eof_func = Zstd_eof;
+ CFH->get_error_func = Zstd_get_error;
+
+ CFH->compression_spec = compression_spec;
+
+ CFH->private_data = NULL;
+}
+
+#endif /* USE_ZSTD */
diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h
new file mode 100644
index 00000000000..2aaa6b100b1
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_zstd.h
+ * Zstd interface to compress_io.c routines
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/compress_zstd.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef COMPRESS_ZSTD_H
+#define COMPRESS_ZSTD_H
+
+#include "compress_io.h"
+
+extern void InitCompressorZstd(CompressorState *cs,
+ const pg_compress_specification compression_spec);
+extern void InitCompressFileHandleZstd(CompressFileHandle *CFH,
+ const pg_compress_specification compression_spec);
+
+#endif /* COMPRESS_ZSTD_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index b2fb7ac77fd..9d59a106f36 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -5,6 +5,7 @@ pg_dump_common_sources = files(
'compress_io.c',
'compress_lz4.c',
'compress_none.c',
+ 'compress_zstd.c',
'dumputils.c',
'parallel.c',
'pg_backup_archiver.c',
@@ -19,7 +20,7 @@ pg_dump_common_sources = files(
pg_dump_common = static_library('libpgdump_common',
pg_dump_common_sources,
c_pch: pch_postgres_fe_h,
- dependencies: [frontend_code, libpq, lz4, zlib],
+ dependencies: [frontend_code, libpq, lz4, zlib, zstd],
kwargs: internal_lib_args,
)
@@ -90,6 +91,7 @@ tests += {
'env': {
'GZIP_PROGRAM': gzip.path(),
'LZ4': program_lz4.found() ? program_lz4.path() : '',
+ 'ZSTD': program_zstd.found() ? program_zstd.path() : '',
'with_icu': icu.found() ? 'yes' : 'no',
},
'tests': [
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index ab77e373e91..e8ee6b1ad86 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -2120,7 +2120,7 @@ _discoverArchiveFormat(ArchiveHandle *AH)
/*
* Check if the specified archive is a directory. If so, check if
- * there's a "toc.dat" (or "toc.dat.{gz,lz4}") file in it.
+ * there's a "toc.dat" (or "toc.dat.{gz,lz4,zst}") file in it.
*/
if (stat(AH->fSpec, &st) == 0 && S_ISDIR(st.st_mode))
{
@@ -2131,10 +2131,17 @@ _discoverArchiveFormat(ArchiveHandle *AH)
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.gz"))
return AH->format;
#endif
+
#ifdef USE_LZ4
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.lz4"))
return AH->format;
#endif
+
+#ifdef USE_ZSTD
+ if (_fileExistsInDirectory(AH->fSpec, "toc.dat.zst"))
+ return AH->format;
+#endif
+
pg_fatal("directory \"%s\" does not appear to be a valid archive (\"toc.dat\" does not exist)",
AH->fSpec);
fh = NULL; /* keep compiler quiet */
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index abaaa3b10e3..2177d5ff425 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -785,6 +785,8 @@ _PrepParallelRestore(ArchiveHandle *AH)
strlcat(fname, ".gz", sizeof(fname));
else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4)
strlcat(fname, ".lz4", sizeof(fname));
+ else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ strlcat(fname, ".zst", sizeof(fname));
if (stat(fname, &st) == 0)
te->dataLength = st.st_size;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 6abbcff6834..05833a48460 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -61,6 +61,7 @@
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
+#include "compress_io.h"
#include "parallel.h"
#include "pg_backup_db.h"
#include "pg_backup_utils.h"
@@ -735,18 +736,13 @@ main(int argc, char **argv)
pg_fatal("invalid compression specification: %s",
error_detail);
- switch (compression_algorithm)
- {
- case PG_COMPRESSION_NONE:
- /* fallthrough */
- case PG_COMPRESSION_GZIP:
- /* fallthrough */
- case PG_COMPRESSION_LZ4:
- break;
- case PG_COMPRESSION_ZSTD:
- pg_fatal("compression with %s is not yet supported", "ZSTD");
- break;
- }
+ error_detail = supports_compression(compression_spec);
+ if (error_detail != NULL)
+ pg_fatal("%s", error_detail);
+
+ if (compression_spec.options & PG_COMPRESSION_OPTION_WORKERS)
+ pg_log_warning("compression option is not currently supported: \"%s\"",
+ "workers");
/*
* Custom and directory formats are compressed by default with gzip when
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 42215f82f7a..74f23ae7f74 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -54,8 +54,9 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir;
# those lines) to validate that part of the process.
my $supports_icu = ($ENV{with_icu} eq 'yes');
-my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1");
+my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
+my $supports_zstd = check_pg_config("#define USE_ZSTD 1");
my %pgdump_runs = (
binary_upgrade => {
@@ -213,6 +214,77 @@ my %pgdump_runs = (
},
},
+ compression_zstd_custom => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=custom',
+ '--compress=zstd', "--file=$tempdir/compression_zstd_custom.dump",
+ 'postgres',
+ ],
+ restore_cmd => [
+ 'pg_restore',
+ "--file=$tempdir/compression_zstd_custom.sql",
+ "$tempdir/compression_zstd_custom.dump",
+ ],
+ command_like => {
+ command => [
+ 'pg_restore',
+ '-l', "$tempdir/compression_zstd_custom.dump",
+ ],
+ expected => qr/Compression: zstd/,
+ name => 'data content is zstd compressed'
+ },
+ },
+
+ compression_zstd_dir => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--jobs=2',
+ '--format=directory', '--compress=zstd:1',
+ "--file=$tempdir/compression_zstd_dir", 'postgres',
+ ],
+ # Give coverage for manually compressed blob.toc files during
+ # restore.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-z', '-f', '--rm',
+ "$tempdir/compression_zstd_dir/blobs.toc",
+ "-o", "$tempdir/compression_zstd_dir/blobs.toc.zst",
+ ],
+ },
+ # Verify that data files were compressed
+ glob_patterns => [
+ "$tempdir/compression_zstd_dir/toc.dat",
+ "$tempdir/compression_zstd_dir/*.dat.zst",
+ ],
+ restore_cmd => [
+ 'pg_restore', '--jobs=2',
+ "--file=$tempdir/compression_zstd_dir.sql",
+ "$tempdir/compression_zstd_dir",
+ ],
+ },
+
+ compression_zstd_plain => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=plain', '--compress=zstd',
+ "--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
+ ],
+ # Decompress the generated file to run through the tests.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-d', '-f',
+ "$tempdir/compression_zstd_plain.sql.zst",
+ "-o", "$tempdir/compression_zstd_plain.sql",
+ ],
+ },
+ },
+
clean => {
dump_cmd => [
'pg_dump',
@@ -4648,10 +4720,11 @@ foreach my $run (sort keys %pgdump_runs)
my $test_key = $run;
my $run_db = 'postgres';
- # Skip command-level tests for gzip/lz4 if there is no support for it.
+ # Skip command-level tests for gzip/lz4/zstd if the tool is not supported
if ($pgdump_runs{$run}->{compile_option} &&
(($pgdump_runs{$run}->{compile_option} eq 'gzip' && !$supports_gzip) ||
- ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4)))
+ ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4) ||
+ ($pgdump_runs{$run}->{compile_option} eq 'zstd' && !$supports_zstd)))
{
note "$run: skipped due to no $pgdump_runs{$run}->{compile_option} support";
next;
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index 58039934756..10fb51585c9 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -154,6 +154,7 @@ do
test "$f" = src/bin/pg_dump/compress_io.h && continue
test "$f" = src/bin/pg_dump/compress_lz4.h && continue
test "$f" = src/bin/pg_dump/compress_none.h && continue
+ test "$f" = src/bin/pg_dump/compress_zstd.h && continue
test "$f" = src/bin/pg_dump/parallel.h && continue
test "$f" = src/bin/pg_dump/pg_backup_archiver.h && continue
test "$f" = src/bin/pg_dump/pg_dump.h && continue
--
2.34.1
[text/x-diff] 0002-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch (12.7K, ../../ZCdpzywVGl0BLan2@telsasoft.com/3-0002-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch)
download | inline diff:
From cead77201ffc76d9f6ea9b853467d846f1618a8a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 27 Mar 2022 11:55:01 -0500
Subject: [PATCH 2/4] zstd: support long distance mode in pg_dump/basebackup
First proposed here:
20220327205020.GM28503@telsasoft.com
---
doc/src/sgml/protocol.sgml | 10 +++-
doc/src/sgml/ref/pg_basebackup.sgml | 4 +-
doc/src/sgml/ref/pg_dump.sgml | 2 +
src/backend/backup/basebackup_zstd.c | 12 ++++
src/bin/pg_basebackup/bbstreamer_zstd.c | 13 +++++
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 9 ++-
src/bin/pg_dump/compress_zstd.c | 5 ++
src/bin/pg_dump/t/002_pg_dump.pl | 3 +-
src/bin/pg_verifybackup/t/008_untar.pl | 8 +++
src/bin/pg_verifybackup/t/010_client_untar.pl | 8 +++
src/common/compression.c | 57 ++++++++++++++++++-
src/include/common/compression.h | 2 +
12 files changed, 127 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 8b5e7b1ad7f..b11d9a6ba35 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2729,7 +2729,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
level. Otherwise, it should be a comma-separated list of items,
each of the form <replaceable>keyword</replaceable> or
<replaceable>keyword=value</replaceable>. Currently, the supported
- keywords are <literal>level</literal> and <literal>workers</literal>.
+ keywords are <literal>level</literal>, <literal>long</literal> and
+ <literal>workers</literal>.
</para>
<para>
@@ -2746,6 +2747,13 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<literal>3</literal>).
</para>
+ <para>
+ The <literal>long</literal> keyword enables long-distance matching
+ mode, for improved compression ratio, at the expense of higher memory
+ use. Long-distance mode is supported only for
+ <literal>zstd</literal>.
+ </para>
+
<para>
The <literal>workers</literal> keyword sets the number of threads
that should be used for parallel compression. Parallel compression
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index db3ad9cd5eb..79d3e657c32 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -424,8 +424,8 @@ PostgreSQL documentation
level. Otherwise, it should be a comma-separated list of items,
each of the form <literal>keyword</literal> or
<literal>keyword=value</literal>.
- Currently, the supported keywords are <literal>level</literal>
- and <literal>workers</literal>.
+ Currently, the supported keywords are <literal>level</literal>,
+ <literal>long</literal>, and <literal>workers</literal>.
The detail string cannot be used when the compression method
is specified as a plain integer.
</para>
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 8de38e0fd0d..e81e35c13b3 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -681,6 +681,8 @@ PostgreSQL documentation
as though it had been fed through <application>gzip</application>,
<application>lz4</application>, or <application>zstd</application>;
but the default is not to compress.
+ With zstd compression, <literal>long</literal> mode may improve the
+ compression ratio, at the cost of increased memory use.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/backend/backup/basebackup_zstd.c b/src/backend/backup/basebackup_zstd.c
index ac6cac178a0..1bb5820c884 100644
--- a/src/backend/backup/basebackup_zstd.c
+++ b/src/backend/backup/basebackup_zstd.c
@@ -118,6 +118,18 @@ bbsink_zstd_begin_backup(bbsink *sink)
compress->workers, ZSTD_getErrorName(ret)));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(mysink->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret)));
+ }
+
/*
* We need our own buffer, because we're going to pass different data to
* the next sink than what gets passed to us.
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index fe17d6df4ef..fba391e2a0f 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -106,6 +106,19 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, pg_compress_specification *comp
compress->workers, ZSTD_getErrorName(ret));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(streamer->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ {
+ pg_log_error("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret));
+ exit(1);
+ }
+ }
+
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
streamer->zstd_outBuf.size = streamer->base.bbs_buffer.maxlen;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index b60cb78a0d5..4d130a7f944 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -139,7 +139,14 @@ SKIP:
'gzip:workers=3',
'invalid compression specification: compression algorithm "gzip" does not accept a worker count',
'failure on worker count for gzip'
- ],);
+ ],
+ [
+ 'gzip:long',
+ 'invalid compression specification: compression algorithm "gzip" does not support long-distance mode',
+ 'failure on long mode for gzip'
+ ],
+ );
+
for my $cft (@compression_failure_tests)
{
my $cfail = quotemeta($client_fails . $cft->[1]);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
index c19d5262943..c7229ec2922 100644
--- a/src/bin/pg_dump/compress_zstd.c
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -80,6 +80,11 @@ _ZstdCStreamParams(pg_compress_specification compress)
_Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_compressionLevel,
compress.level, "level");
+ if (compress.options & PG_COMPRESSION_OPTION_LONG_DISTANCE)
+ _Zstd_CCtx_setParam_or_die(cstream,
+ ZSTD_c_enableLongDistanceMatching,
+ compress.long_distance, "long");
+
return cstream;
}
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 74f23ae7f74..bb898b06bb4 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -267,11 +267,12 @@ my %pgdump_runs = (
],
},
+ # Exercise long mode for test coverage
compression_zstd_plain => {
test_key => 'compression',
compile_option => 'zstd',
dump_cmd => [
- 'pg_dump', '--format=plain', '--compress=zstd',
+ 'pg_dump', '--format=plain', '--compress=zstd:long',
"--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
],
# Decompress the generated file to run through the tests.
diff --git a/src/bin/pg_verifybackup/t/008_untar.pl b/src/bin/pg_verifybackup/t/008_untar.pl
index 3007bbe8556..05754bc8ec7 100644
--- a/src/bin/pg_verifybackup/t/008_untar.pl
+++ b/src/bin/pg_verifybackup/t/008_untar.pl
@@ -49,6 +49,14 @@ my @test_configuration = (
'decompress_program' => $ENV{'ZSTD'},
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => [ '--compress', 'server-zstd:level=1,long' ],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => ['-d'],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
});
for my $tc (@test_configuration)
diff --git a/src/bin/pg_verifybackup/t/010_client_untar.pl b/src/bin/pg_verifybackup/t/010_client_untar.pl
index f3aa0f59e29..ac51a174d14 100644
--- a/src/bin/pg_verifybackup/t/010_client_untar.pl
+++ b/src/bin/pg_verifybackup/t/010_client_untar.pl
@@ -50,6 +50,14 @@ my @test_configuration = (
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
},
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => ['--compress', 'client-zstd:level=1,long'],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => [ '-d' ],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
{
'compression_method' => 'parallel zstd',
'backup_flags' => [ '--compress', 'client-zstd:workers=3' ],
diff --git a/src/common/compression.c b/src/common/compression.c
index 2d3e56b4d62..713a77c292d 100644
--- a/src/common/compression.c
+++ b/src/common/compression.c
@@ -12,7 +12,7 @@
* Otherwise, a compression specification is a comma-separated list of items,
* each having the form keyword or keyword=value.
*
- * Currently, the only supported keywords are "level" and "workers".
+ * Currently, the supported keywords are "level", "long", and "workers".
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
*
@@ -38,6 +38,8 @@
static int expect_integer_value(char *keyword, char *value,
pg_compress_specification *result);
+static bool expect_boolean_value(char *keyword, char *value,
+ pg_compress_specification *result);
/*
* Look up a compression algorithm by name. Returns true and sets *algorithm
@@ -232,6 +234,11 @@ parse_compress_specification(pg_compress_algorithm algorithm, char *specificatio
result->workers = expect_integer_value(keyword, value, result);
result->options |= PG_COMPRESSION_OPTION_WORKERS;
}
+ else if (strcmp(keyword, "long") == 0)
+ {
+ result->long_distance = expect_boolean_value(keyword, value, result);
+ result->options |= PG_COMPRESSION_OPTION_LONG_DISTANCE;
+ }
else
result->parse_error =
psprintf(_("unrecognized compression option: \"%s\""), keyword);
@@ -289,6 +296,43 @@ expect_integer_value(char *keyword, char *value, pg_compress_specification *resu
return ivalue;
}
+/*
+ * Parse 'value' as an boolean and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1. The caller must check result->parse_error to determine if
+ * the call was successful.
+ *
+ * Valid values are: yes, no, on, off, 1, 0.
+ *
+ * Inspired by ParseVariableBool().
+ */
+static bool
+expect_boolean_value(char *keyword, char *value, pg_compress_specification *result)
+{
+ if (value == NULL)
+ return true;
+
+ if (pg_strcasecmp(value, "yes") == 0)
+ return true;
+ if (pg_strcasecmp(value, "on") == 0)
+ return true;
+ if (pg_strcasecmp(value, "1") == 0)
+ return true;
+
+ if (pg_strcasecmp(value, "no") == 0)
+ return false;
+ if (pg_strcasecmp(value, "off") == 0)
+ return false;
+ if (pg_strcasecmp(value, "0") == 0)
+ return false;
+
+ result->parse_error =
+ psprintf(_("value for compression option \"%s\" must be a boolean"),
+ keyword);
+ return false;
+}
+
/*
* Returns NULL if the compression specification string was syntactically
* valid and semantically sensible. Otherwise, returns an error message.
@@ -354,6 +398,17 @@ validate_compress_specification(pg_compress_specification *spec)
get_compress_algorithm_name(spec->algorithm));
}
+ /*
+ * Of the compression algorithms that we currently support, only zstd
+ * supports long-distance mode.
+ */
+ if ((spec->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0 &&
+ (spec->algorithm != PG_COMPRESSION_ZSTD))
+ {
+ return psprintf(_("compression algorithm \"%s\" does not support long-distance mode"),
+ get_compress_algorithm_name(spec->algorithm));
+ }
+
return NULL;
}
diff --git a/src/include/common/compression.h b/src/include/common/compression.h
index b48c173022e..6cf8cf396a8 100644
--- a/src/include/common/compression.h
+++ b/src/include/common/compression.h
@@ -27,6 +27,7 @@ typedef enum pg_compress_algorithm
} pg_compress_algorithm;
#define PG_COMPRESSION_OPTION_WORKERS (1 << 0)
+#define PG_COMPRESSION_OPTION_LONG_DISTANCE (1 << 1)
typedef struct pg_compress_specification
{
@@ -34,6 +35,7 @@ typedef struct pg_compress_specification
unsigned options; /* OR of PG_COMPRESSION_OPTION constants */
int level;
int workers;
+ int long_distance;
char *parse_error; /* NULL if parsing was OK, else message */
} pg_compress_specification;
--
2.34.1
[text/x-diff] 0003-WIP-pg_dump-support-zstd-workers.patch (2.8K, ../../ZCdpzywVGl0BLan2@telsasoft.com/4-0003-WIP-pg_dump-support-zstd-workers.patch)
download | inline diff:
From 39b023ef6626dcd303aeee8e9c2418f107244f0f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Thu, 30 Mar 2023 17:48:57 -0500
Subject: [PATCH 3/4] WIP: pg_dump: support zstd workers
This is a separate commit since it's not essential; the zstd library is
frequently compiled without threading support, so the functionality
isn't very well-tested, and because use of zstd threads might
conceivably play poorly with pg_dump's use of threads under Windows.
Targetting postgres v17.
---
doc/src/sgml/ref/pg_dump.sgml | 8 ++++++--
src/bin/pg_dump/compress_zstd.c | 4 ++++
src/bin/pg_dump/pg_dump.c | 4 ----
3 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index e81e35c13b3..1d55ce05b21 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -681,8 +681,12 @@ PostgreSQL documentation
as though it had been fed through <application>gzip</application>,
<application>lz4</application>, or <application>zstd</application>;
but the default is not to compress.
- With zstd compression, <literal>long</literal> mode may improve the
- compression ratio, at the cost of increased memory use.
+ With zstd compression, <literal>long</literal> and
+ <literal>workers</literal> options may be specified to enable long-distance
+ matching and threaded workers, respectively.
+ Long distance mode may improve the compression ratio, at the cost of
+ increased memory use.
+ Threaded workers allow leveraging multiple CPUs during compression.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
index c7229ec2922..1b821f8ecb1 100644
--- a/src/bin/pg_dump/compress_zstd.c
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -85,6 +85,10 @@ _ZstdCStreamParams(pg_compress_specification compress)
ZSTD_c_enableLongDistanceMatching,
compress.long_distance, "long");
+ if (compress.options & PG_COMPRESSION_OPTION_WORKERS)
+ _Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
+ compress.workers, "workers");
+
return cstream;
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 05833a48460..c0c165c2940 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -740,10 +740,6 @@ main(int argc, char **argv)
if (error_detail != NULL)
pg_fatal("%s", error_detail);
- if (compression_spec.options & PG_COMPRESSION_OPTION_WORKERS)
- pg_log_warning("compression option is not currently supported: \"%s\"",
- "workers");
-
/*
* Custom and directory formats are compressed by default with gzip when
* available, not the others.
--
2.34.1
[text/x-diff] 0004-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch (4.4K, ../../ZCdpzywVGl0BLan2@telsasoft.com/5-0004-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch)
download | inline diff:
From a353ce7ecce1804170bbdf8c56313fab92376561 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 4 Jan 2023 21:21:53 -0600
Subject: [PATCH 4/4] TMP: pg_dump: use Zstd by default, for CI only
//-os-only: linux-meson
---
.cirrus.yml | 9 ++++++++-
src/bin/pg_dump/compress_zstd.c | 9 +++++++++
src/bin/pg_dump/pg_dump.c | 4 ++--
src/bin/pg_dump/t/002_pg_dump.pl | 14 +++++++-------
4 files changed, 26 insertions(+), 10 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 5b1747522f9..14402a0ad5c 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -267,6 +267,7 @@ LINUX_CONFIGURE_FEATURES: &LINUX_CONFIGURE_FEATURES >-
LINUX_MESON_FEATURES: &LINUX_MESON_FEATURES >-
-Dllvm=enabled
-Duuid=e2fs
+ -Dzstd=enabled
# Linux, both 32bit and 64bit
@@ -389,6 +390,9 @@ task:
configure_script: |
su postgres <<-EOF
+ mkdir subprojects
+ meson wrap install zstd
+ meson configure -D zstd:multithread=enabled --force-fallback-for=zstd
meson setup \
--buildtype=debug \
-Dcassert=true \
@@ -616,7 +620,10 @@ task:
# Use /DEBUG:FASTLINK to avoid high memory usage during linking
configure_script: |
vcvarsall x64
- meson setup --backend ninja --buildtype debug -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Db_pch=true -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" build
+ mkdir subprojects
+ meson wrap install zstd
+ meson configure -D zstd:multithread=enabled --force-fallback-for=zstd
+ meson setup --backend ninja --buildtype debug -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Db_pch=true -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" -D zstd=enabled build
build_script: |
vcvarsall x64
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
index 1b821f8ecb1..94dc16cff49 100644
--- a/src/bin/pg_dump/compress_zstd.c
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -88,6 +88,15 @@ _ZstdCStreamParams(pg_compress_specification compress)
if (compress.options & PG_COMPRESSION_OPTION_WORKERS)
_Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
compress.workers, "workers");
+ else
+ {
+ size_t res;
+
+ res = ZSTD_CCtx_setParameter(cstream, ZSTD_c_nbWorkers, 3);
+ if (ZSTD_isError(res))
+ pg_log_warning("could not set compression parameter: \"%s\": %s",
+ "workers", ZSTD_getErrorName(res));
+ }
return cstream;
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c0c165c2940..f146a29aeff 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -747,8 +747,8 @@ main(int argc, char **argv)
if ((archiveFormat == archCustom || archiveFormat == archDirectory) &&
!user_compression_defined)
{
-#ifdef HAVE_LIBZ
- parse_compress_specification(PG_COMPRESSION_GZIP, NULL,
+#ifdef USE_ZSTD
+ parse_compress_specification(PG_COMPRESSION_ZSTD, NULL,
&compression_spec);
#else
/* Nothing to do in the default case */
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index bb898b06bb4..0a635ae9fc3 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -385,10 +385,10 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_custom_format.dump", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
- name => 'data content is gzip-compressed by default if available',
+ name => 'data content is zstd-compressed by default if available',
},
},
@@ -410,16 +410,16 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_dir_format", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
name => 'data content is gzip-compressed by default',
},
glob_patterns => [
"$tempdir/defaults_dir_format/toc.dat",
"$tempdir/defaults_dir_format/blobs.toc",
- $supports_gzip ?
- "$tempdir/defaults_dir_format/*.dat.gz" :
+ $supports_zstd ?
+ "$tempdir/defaults_dir_format/*.dat.zst" :
"$tempdir/defaults_dir_format/*.dat",
],
},
--
2.34.1
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-04-01 00:11 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 46+ messages in thread
From: Tomas Vondra @ 2023-04-01 00:11 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On 4/1/23 01:16, Justin Pryzby wrote:
> On Tue, Mar 28, 2023 at 06:23:26PM +0200, Tomas Vondra wrote:
>> On 3/27/23 19:28, Justin Pryzby wrote:
>>> On Fri, Mar 17, 2023 at 03:43:31AM +0100, Tomas Vondra wrote:
>>>> On 3/16/23 05:50, Justin Pryzby wrote:
>>>>> On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
>>>>>> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <jchampion@timescale.com> wrote:
>>>>>>> I did some smoke testing against zstd's GitHub release on Windows. To
>>>>>>> build against it, I had to construct an import library, and put that
>>>>>>> and the DLL into the `lib` folder expected by the MSVC scripts...
>>>>>>> which makes me wonder if I've chosen a harder way than necessary?
>>>>>>
>>>>>> It looks like pg_dump's meson.build is missing dependencies on zstd
>>>>>> (meson couldn't find the headers in the subproject without them).
>>>>>
>>>>> I saw that this was added for LZ4, but I hadn't added it for zstd since
>>>>> I didn't run into an issue without it. Could you check that what I've
>>>>> added works for your case ?
>>>>>
>>>>>>> Parallel zstd dumps seem to work as expected, in that the resulting
>>>>>>> pg_restore output is identical to uncompressed dumps and nothing
>>>>>>> explodes. I haven't inspected the threading implementation for safety
>>>>>>> yet, as you mentioned.
>>>>>>
>>>>>> Hm. Best I can tell, the CloneArchive() machinery is supposed to be
>>>>>> handling safety for this, by isolating each thread's state. I don't feel
>>>>>> comfortable pronouncing this new addition safe or not, because I'm not
>>>>>> sure I understand what the comments in the format-specific _Clone()
>>>>>> callbacks are saying yet.
>>>>>
>>>>> My line of reasoning for unix is that pg_dump forks before any calls to
>>>>> zstd. Nothing zstd does ought to affect the pg_dump layer. But that
>>>>> doesn't apply to pg_dump under windows. This is an opened question. If
>>>>> there's no solid answer, I could disable/ignore the option (maybe only
>>>>> under windows).
>>>>
>>>> I may be missing something, but why would the patch affect this? Why
>>>> would it even affect safety of the parallel dump? And I don't see any
>>>> changes to the clone stuff ...
>>>
>>> zstd supports using threads during compression, with -Z zstd:workers=N.
>>> When unix forks, the child processes can't do anything to mess up the
>>> state of the parent processes.
>>>
>>> But windows pg_dump uses threads instead of forking, so it seems
>>> possible that the pg_dump -j threads that then spawn zstd threads could
>>> "leak threads" and break the main thread. I suspect there's no issue,
>>> but we still ought to verify that before declaring it safe.
>>
>> OK. I don't have access to a Windows machine so I can't test that. Is it
>> possible to disable the zstd threading, until we figure this out?
>
> I think that's what's best. I made it issue a warning if "workers" was
> specified. It could also be an error, or just ignored.
>
> I considered disabling workers only for windows, but realized that I
> haven't tested with threads myself - my local zstd package is compiled
> without threading, and I remember having some issue recompiling it with
> threading. Jacob's recipe for using meson wraps works well, but it
> still seems better to leave it as a future feature. I used that recipe
> to enabled zstd with threading on CI (except for linux/autoconf).
>
+1 to disable this if we're unsure it works correctly. I agree it's
better to just error out if workers are requested - I rather dislike
when a tool just ignores an explicit parameter. And AFAICS it's what
zstd does too, when someone requests workers on incompatible build.
FWIW I've been thinking about this a bit more and I don't quite see why
would the threading cause issues (except for Windows). I forgot
pg_basebackup already supports zstd, including the worker threading, so
why would it work there and not in pg_dump? Sure, pg_basebackup is not
parallel, but with separate pg_dump processes that shouldn't be an issue
(although I'm not sure when zstd creates threads).
The one thing I'm wondering about is at which point are the worker
threads spawned - but presumably not before the pg_dump processes fork.
I'll try building zstd with threading enabled, and do some tests over
the weekend.
>>>>> The function is first checking if it was passed a filename which already
>>>>> has a suffix. And if not, it searches through a list of suffixes,
>>>>> testing for an existing file with each suffix. The search with stat()
>>>>> doesn't happen if it has a suffix. I'm having trouble seeing how the
>>>>> hasSuffix() branch isn't dead code. Another opened question.
>>>>
>>>> AFAICS it's done this way because of this comment in pg_backup_directory
>>>>
>>>> * ...
>>>> * ".gz" suffix is added to the filenames. The TOC files are never
>>>> * compressed by pg_dump, however they are accepted with the .gz suffix
>>>> * too, in case the user has manually compressed them with 'gzip'.
>>>>
>>>> I haven't tried, but I believe that if you manually compress the
>>>> directory, it may hit this branch.
>>>
>>> That would make sense, but when I tried, it didn't work like that.
>>> The filenames were all uncompressed names. Maybe it worked differently
>>> in an older release. Or maybe it changed during development of the
>>> parallel-directory-dump patch and it's actually dead code.
>>
>> Interesting. Would be good to find out. I wonder if a little bit of
>> git-log digging could tell us more. Anyway, until we confirm it's dead
>> code, we should probably do what .gz does and have the same check for
>> .lz4 and .zst files.
>
> I found that hasSuffix() and cfopen() originated in the refactored patch
> Heikki's sent here; there's no history beyond that.
>
> https://www.postgresql.org/message-id/4D3954C7.9060503%40enterprisedb.com
>
> The patch published there appends the .gz within cfopen(), and the
> caller writes into the TOC the filename without ".gz". It seems like
> maybe a few hours prior, Heikki may have been appending the ".gz" suffix
> in the caller, and then writing the TOC with filename.gz.
>
> The only way I've been able to get a "filename.gz" passed to hasSuffix
> is to write a directory-format dump, with LOs, and without compression,
> and then compress the blobs with "gzip", and *also* edit the blobs.toc
> file to say ".gz" (which isn't necessary since, if the original file
> isn't found, the restore would search for files with compressed
> suffixes).
>
> So .. it's not *technically* unreachable, but I can't see why it'd be
> useful to support editing the *content* of the blob TOC (other than
> compressing it). I might give some weight to the idea if it were also
> possible to edit the non-blob TOC; but, it's a binary file, so no.
>
> For now, I made the change to make zstd and lz4 to behave the same here
> as .gz, unless Heikki has a memory or a git reflog going back far enough
> to further support the idea that the code path isn't useful.
>
Makes sense. Let's keep the same behavior for all compression methods,
and if it's unreachable we shall remove it from all. It's a trivial
amount of code, we can live with that.
> I'm going to set the patch as RFC as a hint to anyone who would want to
> make a final review.
>
OK.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2023-04-01 00:28 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 12:49 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
0 siblings, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2023-04-01 00:28 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Sat, Apr 01, 2023 at 02:11:12AM +0200, Tomas Vondra wrote:
> On 4/1/23 01:16, Justin Pryzby wrote:
> > On Tue, Mar 28, 2023 at 06:23:26PM +0200, Tomas Vondra wrote:
> >> On 3/27/23 19:28, Justin Pryzby wrote:
> >>> On Fri, Mar 17, 2023 at 03:43:31AM +0100, Tomas Vondra wrote:
> >>>> On 3/16/23 05:50, Justin Pryzby wrote:
> >>>>> On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
> >>>>>> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <jchampion@timescale.com> wrote:
> >>>>>>> I did some smoke testing against zstd's GitHub release on Windows. To
> >>>>>>> build against it, I had to construct an import library, and put that
> >>>>>>> and the DLL into the `lib` folder expected by the MSVC scripts...
> >>>>>>> which makes me wonder if I've chosen a harder way than necessary?
> >>>>>>
> >>>>>> It looks like pg_dump's meson.build is missing dependencies on zstd
> >>>>>> (meson couldn't find the headers in the subproject without them).
> >>>>>
> >>>>> I saw that this was added for LZ4, but I hadn't added it for zstd since
> >>>>> I didn't run into an issue without it. Could you check that what I've
> >>>>> added works for your case ?
> >>>>>
> >>>>>>> Parallel zstd dumps seem to work as expected, in that the resulting
> >>>>>>> pg_restore output is identical to uncompressed dumps and nothing
> >>>>>>> explodes. I haven't inspected the threading implementation for safety
> >>>>>>> yet, as you mentioned.
> >>>>>>
> >>>>>> Hm. Best I can tell, the CloneArchive() machinery is supposed to be
> >>>>>> handling safety for this, by isolating each thread's state. I don't feel
> >>>>>> comfortable pronouncing this new addition safe or not, because I'm not
> >>>>>> sure I understand what the comments in the format-specific _Clone()
> >>>>>> callbacks are saying yet.
> >>>>>
> >>>>> My line of reasoning for unix is that pg_dump forks before any calls to
> >>>>> zstd. Nothing zstd does ought to affect the pg_dump layer. But that
> >>>>> doesn't apply to pg_dump under windows. This is an opened question. If
> >>>>> there's no solid answer, I could disable/ignore the option (maybe only
> >>>>> under windows).
> >>>>
> >>>> I may be missing something, but why would the patch affect this? Why
> >>>> would it even affect safety of the parallel dump? And I don't see any
> >>>> changes to the clone stuff ...
> >>>
> >>> zstd supports using threads during compression, with -Z zstd:workers=N.
> >>> When unix forks, the child processes can't do anything to mess up the
> >>> state of the parent processes.
> >>>
> >>> But windows pg_dump uses threads instead of forking, so it seems
> >>> possible that the pg_dump -j threads that then spawn zstd threads could
> >>> "leak threads" and break the main thread. I suspect there's no issue,
> >>> but we still ought to verify that before declaring it safe.
> >>
> >> OK. I don't have access to a Windows machine so I can't test that. Is it
> >> possible to disable the zstd threading, until we figure this out?
> >
> > I think that's what's best. I made it issue a warning if "workers" was
> > specified. It could also be an error, or just ignored.
> >
> > I considered disabling workers only for windows, but realized that I
> > haven't tested with threads myself - my local zstd package is compiled
> > without threading, and I remember having some issue recompiling it with
> > threading. Jacob's recipe for using meson wraps works well, but it
> > still seems better to leave it as a future feature. I used that recipe
> > to enabled zstd with threading on CI (except for linux/autoconf).
>
> +1 to disable this if we're unsure it works correctly. I agree it's
> better to just error out if workers are requested - I rather dislike
> when a tool just ignores an explicit parameter. And AFAICS it's what
> zstd does too, when someone requests workers on incompatible build.
>
> FWIW I've been thinking about this a bit more and I don't quite see why
> would the threading cause issues (except for Windows). I forgot
> pg_basebackup already supports zstd, including the worker threading, so
> why would it work there and not in pg_dump? Sure, pg_basebackup is not
> parallel, but with separate pg_dump processes that shouldn't be an issue
> (although I'm not sure when zstd creates threads).
There's no concern at all except under windows (because on windows
pg_dump -j is implemented using threads rather than forking).
Especially since zstd:workers is already allowed in the basebackup
backend process.
> I'll try building zstd with threading enabled, and do some tests over
> the weekend.
Feel free to wait until v17 :)
I used "meson wraps" to get a local version with threading. Note that
if you want to use a zstd subproject, you may have to specify -D
zstd=enabled, or else meson may not enable the library at all.
Also, in order to introspect its settings, I had to do like this:
mkdir subprojects
meson wrap install zstd
meson subprojects download
mkdir build.meson
meson setup -C build.meson --force-fallback-for=zstd
--
Justin
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-04-01 12:49 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 13:36 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 46+ messages in thread
From: Tomas Vondra @ 2023-04-01 12:49 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On 4/1/23 02:28, Justin Pryzby wrote:
> On Sat, Apr 01, 2023 at 02:11:12AM +0200, Tomas Vondra wrote:
>> On 4/1/23 01:16, Justin Pryzby wrote:
>>> On Tue, Mar 28, 2023 at 06:23:26PM +0200, Tomas Vondra wrote:
>>>> On 3/27/23 19:28, Justin Pryzby wrote:
>>>>> On Fri, Mar 17, 2023 at 03:43:31AM +0100, Tomas Vondra wrote:
>>>>>> On 3/16/23 05:50, Justin Pryzby wrote:
>>>>>>> On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
>>>>>>>> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <jchampion@timescale.com> wrote:
>>>>>>>>> I did some smoke testing against zstd's GitHub release on Windows. To
>>>>>>>>> build against it, I had to construct an import library, and put that
>>>>>>>>> and the DLL into the `lib` folder expected by the MSVC scripts...
>>>>>>>>> which makes me wonder if I've chosen a harder way than necessary?
>>>>>>>>
>>>>>>>> It looks like pg_dump's meson.build is missing dependencies on zstd
>>>>>>>> (meson couldn't find the headers in the subproject without them).
>>>>>>>
>>>>>>> I saw that this was added for LZ4, but I hadn't added it for zstd since
>>>>>>> I didn't run into an issue without it. Could you check that what I've
>>>>>>> added works for your case ?
>>>>>>>
>>>>>>>>> Parallel zstd dumps seem to work as expected, in that the resulting
>>>>>>>>> pg_restore output is identical to uncompressed dumps and nothing
>>>>>>>>> explodes. I haven't inspected the threading implementation for safety
>>>>>>>>> yet, as you mentioned.
>>>>>>>>
>>>>>>>> Hm. Best I can tell, the CloneArchive() machinery is supposed to be
>>>>>>>> handling safety for this, by isolating each thread's state. I don't feel
>>>>>>>> comfortable pronouncing this new addition safe or not, because I'm not
>>>>>>>> sure I understand what the comments in the format-specific _Clone()
>>>>>>>> callbacks are saying yet.
>>>>>>>
>>>>>>> My line of reasoning for unix is that pg_dump forks before any calls to
>>>>>>> zstd. Nothing zstd does ought to affect the pg_dump layer. But that
>>>>>>> doesn't apply to pg_dump under windows. This is an opened question. If
>>>>>>> there's no solid answer, I could disable/ignore the option (maybe only
>>>>>>> under windows).
>>>>>>
>>>>>> I may be missing something, but why would the patch affect this? Why
>>>>>> would it even affect safety of the parallel dump? And I don't see any
>>>>>> changes to the clone stuff ...
>>>>>
>>>>> zstd supports using threads during compression, with -Z zstd:workers=N.
>>>>> When unix forks, the child processes can't do anything to mess up the
>>>>> state of the parent processes.
>>>>>
>>>>> But windows pg_dump uses threads instead of forking, so it seems
>>>>> possible that the pg_dump -j threads that then spawn zstd threads could
>>>>> "leak threads" and break the main thread. I suspect there's no issue,
>>>>> but we still ought to verify that before declaring it safe.
>>>>
>>>> OK. I don't have access to a Windows machine so I can't test that. Is it
>>>> possible to disable the zstd threading, until we figure this out?
>>>
>>> I think that's what's best. I made it issue a warning if "workers" was
>>> specified. It could also be an error, or just ignored.
>>>
>>> I considered disabling workers only for windows, but realized that I
>>> haven't tested with threads myself - my local zstd package is compiled
>>> without threading, and I remember having some issue recompiling it with
>>> threading. Jacob's recipe for using meson wraps works well, but it
>>> still seems better to leave it as a future feature. I used that recipe
>>> to enabled zstd with threading on CI (except for linux/autoconf).
>>
>> +1 to disable this if we're unsure it works correctly. I agree it's
>> better to just error out if workers are requested - I rather dislike
>> when a tool just ignores an explicit parameter. And AFAICS it's what
>> zstd does too, when someone requests workers on incompatible build.
>>
>> FWIW I've been thinking about this a bit more and I don't quite see why
>> would the threading cause issues (except for Windows). I forgot
>> pg_basebackup already supports zstd, including the worker threading, so
>> why would it work there and not in pg_dump? Sure, pg_basebackup is not
>> parallel, but with separate pg_dump processes that shouldn't be an issue
>> (although I'm not sure when zstd creates threads).
>
> There's no concern at all except under windows (because on windows
> pg_dump -j is implemented using threads rather than forking).
> Especially since zstd:workers is already allowed in the basebackup
> backend process.
>
If there are no concerns, why disable it outside Windows? I don't have a
good idea how beneficial the multi-threaded compression is, so I can't
quite judge the risk/benefits tradeoff.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 12:49 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2023-04-01 13:36 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 20:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
0 siblings, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2023-04-01 13:36 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Sat, Apr 01, 2023 at 02:49:44PM +0200, Tomas Vondra wrote:
> On 4/1/23 02:28, Justin Pryzby wrote:
> > On Sat, Apr 01, 2023 at 02:11:12AM +0200, Tomas Vondra wrote:
> >> On 4/1/23 01:16, Justin Pryzby wrote:
> >>> On Tue, Mar 28, 2023 at 06:23:26PM +0200, Tomas Vondra wrote:
> >>>> On 3/27/23 19:28, Justin Pryzby wrote:
> >>>>> On Fri, Mar 17, 2023 at 03:43:31AM +0100, Tomas Vondra wrote:
> >>>>>> On 3/16/23 05:50, Justin Pryzby wrote:
> >>>>>>> On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
> >>>>>>>> On Wed, Mar 8, 2023 at 10:59 AM Jacob Champion <jchampion@timescale.com> wrote:
> >>>>>>>>> I did some smoke testing against zstd's GitHub release on Windows. To
> >>>>>>>>> build against it, I had to construct an import library, and put that
> >>>>>>>>> and the DLL into the `lib` folder expected by the MSVC scripts...
> >>>>>>>>> which makes me wonder if I've chosen a harder way than necessary?
> >>>>>>>>
> >>>>>>>> It looks like pg_dump's meson.build is missing dependencies on zstd
> >>>>>>>> (meson couldn't find the headers in the subproject without them).
> >>>>>>>
> >>>>>>> I saw that this was added for LZ4, but I hadn't added it for zstd since
> >>>>>>> I didn't run into an issue without it. Could you check that what I've
> >>>>>>> added works for your case ?
> >>>>>>>
> >>>>>>>>> Parallel zstd dumps seem to work as expected, in that the resulting
> >>>>>>>>> pg_restore output is identical to uncompressed dumps and nothing
> >>>>>>>>> explodes. I haven't inspected the threading implementation for safety
> >>>>>>>>> yet, as you mentioned.
> >>>>>>>>
> >>>>>>>> Hm. Best I can tell, the CloneArchive() machinery is supposed to be
> >>>>>>>> handling safety for this, by isolating each thread's state. I don't feel
> >>>>>>>> comfortable pronouncing this new addition safe or not, because I'm not
> >>>>>>>> sure I understand what the comments in the format-specific _Clone()
> >>>>>>>> callbacks are saying yet.
> >>>>>>>
> >>>>>>> My line of reasoning for unix is that pg_dump forks before any calls to
> >>>>>>> zstd. Nothing zstd does ought to affect the pg_dump layer. But that
> >>>>>>> doesn't apply to pg_dump under windows. This is an opened question. If
> >>>>>>> there's no solid answer, I could disable/ignore the option (maybe only
> >>>>>>> under windows).
> >>>>>>
> >>>>>> I may be missing something, but why would the patch affect this? Why
> >>>>>> would it even affect safety of the parallel dump? And I don't see any
> >>>>>> changes to the clone stuff ...
> >>>>>
> >>>>> zstd supports using threads during compression, with -Z zstd:workers=N.
> >>>>> When unix forks, the child processes can't do anything to mess up the
> >>>>> state of the parent processes.
> >>>>>
> >>>>> But windows pg_dump uses threads instead of forking, so it seems
> >>>>> possible that the pg_dump -j threads that then spawn zstd threads could
> >>>>> "leak threads" and break the main thread. I suspect there's no issue,
> >>>>> but we still ought to verify that before declaring it safe.
> >>>>
> >>>> OK. I don't have access to a Windows machine so I can't test that. Is it
> >>>> possible to disable the zstd threading, until we figure this out?
> >>>
> >>> I think that's what's best. I made it issue a warning if "workers" was
> >>> specified. It could also be an error, or just ignored.
> >>>
> >>> I considered disabling workers only for windows, but realized that I
> >>> haven't tested with threads myself - my local zstd package is compiled
> >>> without threading, and I remember having some issue recompiling it with
> >>> threading. Jacob's recipe for using meson wraps works well, but it
> >>> still seems better to leave it as a future feature. I used that recipe
> >>> to enabled zstd with threading on CI (except for linux/autoconf).
> >>
> >> +1 to disable this if we're unsure it works correctly. I agree it's
> >> better to just error out if workers are requested - I rather dislike
> >> when a tool just ignores an explicit parameter. And AFAICS it's what
> >> zstd does too, when someone requests workers on incompatible build.
> >>
> >> FWIW I've been thinking about this a bit more and I don't quite see why
> >> would the threading cause issues (except for Windows). I forgot
> >> pg_basebackup already supports zstd, including the worker threading, so
> >> why would it work there and not in pg_dump? Sure, pg_basebackup is not
> >> parallel, but with separate pg_dump processes that shouldn't be an issue
> >> (although I'm not sure when zstd creates threads).
> >
> > There's no concern at all except under windows (because on windows
> > pg_dump -j is implemented using threads rather than forking).
> > Especially since zstd:workers is already allowed in the basebackup
> > backend process.
>
> If there are no concerns, why disable it outside Windows? I don't have a
> good idea how beneficial the multi-threaded compression is, so I can't
> quite judge the risk/benefits tradeoff.
Because it's a minor/fringe feature, and it's annoying to have platform
differences (would we plan on relaxing the restriction in v17, or is it
more likely we'd forget ?).
I realized how little I've tested with zstd workers myself. And I think
on cirrusci, the macos and freebsd tasks have zstd libraries with
threading support, but it wasn't being exercised (because using :workers
would cause the patch to fail unless it's supported everywhere). So I
updated the "for CI only" patch to 1) use meson wraps to compile zstd
library with threading on linux and windows; and, 2) use zstd:workers=3
"opportunistically" (but avoid failing if threads are not supported,
since the autoconf task still doesn't have access to a library with
thread support). That's a great step, but it still seems bad that the
thread stuff has been little exercised until now. (Also, the windows
task failed; I think that's due to a transient network issue).
Feel free to mess around with threads (but I'd much rather see the patch
progress for zstd:long).
--
Justin
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 12:49 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 13:36 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-04-01 20:26 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-03 19:17 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 46+ messages in thread
From: Tomas Vondra @ 2023-04-01 20:26 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On 4/1/23 15:36, Justin Pryzby wrote:
>
> ...
>
>> If there are no concerns, why disable it outside Windows? I don't have a
>> good idea how beneficial the multi-threaded compression is, so I can't
>> quite judge the risk/benefits tradeoff.
>
> Because it's a minor/fringe feature, and it's annoying to have platform
> differences (would we plan on relaxing the restriction in v17, or is it
> more likely we'd forget ?).
>
> I realized how little I've tested with zstd workers myself. And I think
> on cirrusci, the macos and freebsd tasks have zstd libraries with
> threading support, but it wasn't being exercised (because using :workers
> would cause the patch to fail unless it's supported everywhere). So I
> updated the "for CI only" patch to 1) use meson wraps to compile zstd
> library with threading on linux and windows; and, 2) use zstd:workers=3
> "opportunistically" (but avoid failing if threads are not supported,
> since the autoconf task still doesn't have access to a library with
> thread support). That's a great step, but it still seems bad that the
> thread stuff has been little exercised until now. (Also, the windows
> task failed; I think that's due to a transient network issue).
>
Agreed, let's leave the threading for PG17, depending on how beneficial
it turns out to be for pg_dump.
> Feel free to mess around with threads (but I'd much rather see the patch
> progress for zstd:long).
OK, understood. The long mode patch is pretty simple. IIUC it does not
change the format, i.e. in the worst case we could leave it for PG17
too. Correct?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 12:49 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 13:36 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 20:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2023-04-03 19:17 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-03 21:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
0 siblings, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2023-04-03 19:17 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Sat, Apr 01, 2023 at 10:26:01PM +0200, Tomas Vondra wrote:
> > Feel free to mess around with threads (but I'd much rather see the patch
> > progress for zstd:long).
>
> OK, understood. The long mode patch is pretty simple. IIUC it does not
> change the format, i.e. in the worst case we could leave it for PG17
> too. Correct?
Right, libzstd only has one "format", which is the same as what's used
by the commandline tool. zstd:long doesn't change the format of the
output: the library just uses a larger memory buffer to allow better
compression. There's no format change for zstd:workers, either.
--
Justin
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 12:49 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 13:36 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 20:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-03 19:17 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-04-03 21:26 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-04 03:04 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 46+ messages in thread
From: Tomas Vondra @ 2023-04-03 21:26 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On 4/3/23 21:17, Justin Pryzby wrote:
> On Sat, Apr 01, 2023 at 10:26:01PM +0200, Tomas Vondra wrote:
>>> Feel free to mess around with threads (but I'd much rather see the patch
>>> progress for zstd:long).
>>
>> OK, understood. The long mode patch is pretty simple. IIUC it does not
>> change the format, i.e. in the worst case we could leave it for PG17
>> too. Correct?
>
> Right, libzstd only has one "format", which is the same as what's used
> by the commandline tool. zstd:long doesn't change the format of the
> output: the library just uses a larger memory buffer to allow better
> compression. There's no format change for zstd:workers, either.
>
OK. I plan to do a bit more review/testing on this, and get it committed
over the next day or two, likely including the long mode. One thing I
noticed today is that maybe long_distance should be a bool, not int.
Yes, ZSTD_c_enableLongDistanceMatching() accepts int, but it'd be
cleaner to cast the value during a call and keep it bool otherwise.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 12:49 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 13:36 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 20:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-03 19:17 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-03 21:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2023-04-04 03:04 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-05 19:42 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
0 siblings, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2023-04-04 03:04 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Mon, Apr 03, 2023 at 11:26:09PM +0200, Tomas Vondra wrote:
> On 4/3/23 21:17, Justin Pryzby wrote:
> > On Sat, Apr 01, 2023 at 10:26:01PM +0200, Tomas Vondra wrote:
> >>> Feel free to mess around with threads (but I'd much rather see the patch
> >>> progress for zstd:long).
> >>
> >> OK, understood. The long mode patch is pretty simple. IIUC it does not
> >> change the format, i.e. in the worst case we could leave it for PG17
> >> too. Correct?
> >
> > Right, libzstd only has one "format", which is the same as what's used
> > by the commandline tool. zstd:long doesn't change the format of the
> > output: the library just uses a larger memory buffer to allow better
> > compression. There's no format change for zstd:workers, either.
>
> OK. I plan to do a bit more review/testing on this, and get it committed
> over the next day or two, likely including the long mode. One thing I
> noticed today is that maybe long_distance should be a bool, not int.
> Yes, ZSTD_c_enableLongDistanceMatching() accepts int, but it'd be
> cleaner to cast the value during a call and keep it bool otherwise.
Thanks for noticing. Evidently I wrote it using "int" to get the
feature working, and then later wrote the bool parsing bits but never
changed the data structure.
This also updates a few comments, indentation, removes a useless
assertion, and updates the warning about zstd:workers.
--
Justin
Attachments:
[text/x-diff] 0001-pg_dump-zstd-compression.patch (30.6K, ../../ZCuTotQ38nwA%2FhLY@telsasoft.com/2-0001-pg_dump-zstd-compression.patch)
download | inline diff:
From df0eb4d3c4799f24e58f1e5b0a9470e5af355ad6 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 7 Jan 2023 15:45:06 -0600
Subject: [PATCH 1/4] pg_dump: zstd compression
Previously proposed at: 20201221194924.GI30237@telsasoft.com
---
doc/src/sgml/ref/pg_dump.sgml | 13 +-
src/bin/pg_dump/Makefile | 2 +
src/bin/pg_dump/compress_io.c | 66 ++--
src/bin/pg_dump/compress_zstd.c | 537 ++++++++++++++++++++++++++
src/bin/pg_dump/compress_zstd.h | 25 ++
src/bin/pg_dump/meson.build | 4 +-
src/bin/pg_dump/pg_backup_archiver.c | 9 +-
src/bin/pg_dump/pg_backup_directory.c | 2 +
src/bin/pg_dump/pg_dump.c | 20 +-
src/bin/pg_dump/t/002_pg_dump.pl | 79 +++-
src/tools/pginclude/cpluspluscheck | 1 +
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 705 insertions(+), 54 deletions(-)
create mode 100644 src/bin/pg_dump/compress_zstd.c
create mode 100644 src/bin/pg_dump/compress_zstd.h
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 77299878e02..8de38e0fd0d 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -330,8 +330,9 @@ PostgreSQL documentation
machine-readable format that <application>pg_restore</application>
can read. A directory format archive can be manipulated with
standard Unix tools; for example, files in an uncompressed archive
- can be compressed with the <application>gzip</application> or
- <application>lz4</application> tools.
+ can be compressed with the <application>gzip</application>,
+ <application>lz4</application>, or
+ <application>zstd</application> tools.
This format is compressed by default using <literal>gzip</literal>
and also supports parallel dumps.
</para>
@@ -655,7 +656,8 @@ PostgreSQL documentation
<para>
Specify the compression method and/or the compression level to use.
The compression method can be set to <literal>gzip</literal>,
- <literal>lz4</literal>, or <literal>none</literal> for no compression.
+ <literal>lz4</literal>, <literal>zstd</literal>,
+ or <literal>none</literal> for no compression.
A compression detail string can optionally be specified. If the
detail string is an integer, it specifies the compression level.
Otherwise, it should be a comma-separated list of items, each of the
@@ -676,8 +678,9 @@ PostgreSQL documentation
individual table-data segments, and the default is to compress using
<literal>gzip</literal> at a moderate level. For plain text output,
setting a nonzero compression level causes the entire output file to be compressed,
- as though it had been fed through <application>gzip</application> or
- <application>lz4</application>; but the default is not to compress.
+ as though it had been fed through <application>gzip</application>,
+ <application>lz4</application>, or <application>zstd</application>;
+ but the default is not to compress.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index eb8f59459a1..24de7593a6a 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -18,6 +18,7 @@ include $(top_builddir)/src/Makefile.global
export GZIP_PROGRAM=$(GZIP)
export LZ4
+export ZSTD
export with_icu
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
@@ -29,6 +30,7 @@ OBJS = \
compress_io.o \
compress_lz4.o \
compress_none.o \
+ compress_zstd.o \
dumputils.o \
parallel.o \
pg_backup_archiver.o \
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index 0972a4f934a..4f06bb024f9 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -52,8 +52,8 @@
*
* InitDiscoverCompressFileHandle tries to infer the compression by the
* filename suffix. If the suffix is not yet known then it tries to simply
- * open the file and if it fails, it tries to open the same file with the .gz
- * suffix, and then again with the .lz4 suffix.
+ * open the file and if it fails, it tries to open the same file with
+ * compressed suffixes.
*
* IDENTIFICATION
* src/bin/pg_dump/compress_io.c
@@ -69,6 +69,7 @@
#include "compress_io.h"
#include "compress_lz4.h"
#include "compress_none.h"
+#include "compress_zstd.h"
#include "pg_backup_utils.h"
/*----------------------
@@ -77,7 +78,8 @@
*/
/*
- * Checks whether a compression algorithm is supported.
+ * Checks whether support for a compression algorithm is implemented in
+ * pg_dump/restore.
*
* On success returns NULL, otherwise returns a malloc'ed string which can be
* used by the caller in an error message.
@@ -98,6 +100,10 @@ supports_compression(const pg_compress_specification compression_spec)
if (algorithm == PG_COMPRESSION_LZ4)
supported = true;
#endif
+#ifdef USE_ZSTD
+ if (algorithm == PG_COMPRESSION_ZSTD)
+ supported = true;
+#endif
if (!supported)
return psprintf("this build does not support compression with %s",
@@ -130,6 +136,8 @@ AllocateCompressor(const pg_compress_specification compression_spec,
InitCompressorGzip(cs, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressorLZ4(cs, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressorZstd(cs, compression_spec);
return cs;
}
@@ -196,20 +204,30 @@ InitCompressFileHandle(const pg_compress_specification compression_spec)
InitCompressFileHandleGzip(CFH, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressFileHandleLZ4(CFH, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressFileHandleZstd(CFH, compression_spec);
return CFH;
}
+static bool
+check_compressed_file(const char *path, char **fname, char *ext)
+{
+ free_keep_errno(*fname);
+ *fname = psprintf("%s.%s", path, ext);
+ return (access(*fname, F_OK) == 0);
+}
+
/*
* Open a file for reading. 'path' is the file to open, and 'mode' should
* be either "r" or "rb".
*
* If the file at 'path' contains the suffix of a supported compression method,
- * currently this includes ".gz" and ".lz4", then this compression will be used
+ * currently this includes ".gz", ".lz4" and ".zst", then this compression will be used
* throughout. Otherwise the compression will be inferred by iteratively trying
* to open the file at 'path', first as is, then by appending known compression
* suffixes. So if you pass "foo" as 'path', this will open either "foo" or
- * "foo.gz" or "foo.lz4", trying in that order.
+ * "foo.{gz,lz4,zst}", trying in that order.
*
* On failure, return NULL with an error code in errno.
*/
@@ -229,36 +247,20 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode)
if (hasSuffix(fname, ".gz"))
compression_spec.algorithm = PG_COMPRESSION_GZIP;
+ else if (hasSuffix(fname, ".lz4"))
+ compression_spec.algorithm = PG_COMPRESSION_LZ4;
+ else if (hasSuffix(fname, ".zst"))
+ compression_spec.algorithm = PG_COMPRESSION_ZSTD;
else
{
- bool exists;
-
- exists = (stat(path, &st) == 0);
- /* avoid unused warning if it is not built with compression */
- if (exists)
+ if (stat(path, &st) == 0)
compression_spec.algorithm = PG_COMPRESSION_NONE;
-#ifdef HAVE_LIBZ
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.gz", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_GZIP;
- }
-#endif
-#ifdef USE_LZ4
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.lz4", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_LZ4;
- }
-#endif
+ else if (check_compressed_file(path, &fname, "gz"))
+ compression_spec.algorithm = PG_COMPRESSION_GZIP;
+ else if (check_compressed_file(path, &fname, "lz4"))
+ compression_spec.algorithm = PG_COMPRESSION_LZ4;
+ else if (check_compressed_file(path, &fname, "zst"))
+ compression_spec.algorithm = PG_COMPRESSION_ZSTD;
}
CFH = InitCompressFileHandle(compression_spec);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
new file mode 100644
index 00000000000..cf85c3a4c93
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -0,0 +1,537 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_zstd.c
+ * Routines for archivers to write a Zstd compressed data stream.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/compress_zstd.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "pg_backup_utils.h"
+#include "compress_zstd.h"
+
+#ifndef USE_ZSTD
+
+void
+InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+#else
+
+#include <zstd.h>
+
+typedef struct ZstdCompressorState
+{
+ /* This is a normal file to which we read/write compressed data */
+ FILE *fp;
+
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+
+ /* pointer to a static string like from strerror(), for Zstd_write() */
+ const char *zstderror;
+} ZstdCompressorState;
+
+static ZSTD_CStream *_ZstdCStreamParams(pg_compress_specification compress);
+static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen);
+static void ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs);
+
+static void
+_Zstd_CCtx_setParam_or_die(ZSTD_CStream *cstream,
+ ZSTD_cParameter param, int value, char *paramname)
+{
+ size_t res;
+
+ res = ZSTD_CCtx_setParameter(cstream, param, value);
+ if (ZSTD_isError(res))
+ pg_fatal("could not set compression parameter: \"%s\": %s",
+ paramname, ZSTD_getErrorName(res));
+}
+
+/* Return a compression stream with parameters set per argument */
+static ZSTD_CStream *
+_ZstdCStreamParams(pg_compress_specification compress)
+{
+ ZSTD_CStream *cstream;
+
+ cstream = ZSTD_createCStream();
+ if (cstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ _Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_compressionLevel,
+ compress.level, "level");
+
+ return cstream;
+}
+
+/* Helper function for WriteDataToArchiveZstd and EndCompressorZstd */
+static void
+_ZstdWriteCommon(ArchiveHandle *AH, CompressorState *cs, bool flush)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ /* Loop while there's any input or until flushed */
+ while (input->pos != input->size || flush)
+ {
+ size_t res;
+
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output,
+ input, flush ? ZSTD_e_end : ZSTD_e_continue);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ /*
+ * Extra paranoia: avoid zero-length chunks, since a zero length chunk
+ * is the EOF marker in the custom format. This should never happen
+ * but...
+ */
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ if (res == 0)
+ break; /* End of frame or all input consumed */
+ }
+}
+
+static void
+EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ if (cs->readF != NULL)
+ {
+ Assert(zstdcs->cstream == NULL);
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ }
+ else if (cs->writeF != NULL)
+ {
+ Assert(zstdcs->dstream == NULL);
+ _ZstdWriteCommon(AH, cs, true);
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ pg_free(zstdcs);
+}
+
+static void
+WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ zstdcs->input.src = data;
+ zstdcs->input.size = dLen;
+ zstdcs->input.pos = 0;
+
+ _ZstdWriteCommon(AH, cs, false);
+}
+
+static void
+ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ size_t input_allocated_size = ZSTD_DStreamInSize();
+ size_t res;
+
+ for (;;)
+ {
+ size_t cnt;
+
+ /*
+ * Read compressed data. Note that readF can resize the buffer; the
+ * new size is tracked and used for future loops.
+ */
+ input->size = input_allocated_size;
+ cnt = cs->readF(AH, (char **) unconstify(void **, &input->src), &input->size);
+
+ /* ensure that readF didn't *shrink* the buffer */
+ Assert(input->size >= input_allocated_size);
+ input_allocated_size = input->size;
+ input->size = cnt;
+ input->pos = 0;
+
+ if (cnt == 0)
+ break;
+
+ /* Now decompress */
+ while (input->pos < input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ /*
+ * then write the decompressed data to the output handle
+ */
+ ((char *) output->dst)[output->pos] = '\0';
+ ahwrite(output->dst, 1, output->pos, AH);
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+ }
+}
+
+/* Public routine that supports Zstd compressed data I/O */
+void
+InitCompressorZstd(CompressorState *cs,
+ const pg_compress_specification compression_spec)
+{
+ ZstdCompressorState *zstdcs;
+
+ cs->readData = ReadDataFromArchiveZstd;
+ cs->writeData = WriteDataToArchiveZstd;
+ cs->end = EndCompressorZstd;
+
+ cs->compression_spec = compression_spec;
+
+ zstdcs = (ZstdCompressorState *) pg_malloc0(sizeof(*zstdcs));
+ cs->private_data = zstdcs;
+
+ /* We expect that exactly one of readF/writeF is specified */
+ Assert((cs->readF == NULL) != (cs->writeF == NULL));
+
+ if (cs->readF != NULL)
+ {
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ zstdcs->input.size = ZSTD_DStreamInSize();
+ zstdcs->input.src = pg_malloc(zstdcs->input.size);
+
+ /*
+ * output.size is the buffer size we tell zstd it can output to.
+ * Allocate an additional byte such that ReadDataFromArchiveZstd() can
+ * call ahwrite() with a null-terminated string, which is an optimized
+ * case in ExecuteSqlCommandBuf().
+ */
+ zstdcs->output.size = ZSTD_DStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size + 1);
+ }
+ else if (cs->writeF != NULL)
+ {
+ zstdcs->cstream = _ZstdCStreamParams(cs->compression_spec);
+
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size);
+ zstdcs->output.pos = 0;
+ }
+}
+
+/*
+ * Compressed stream API
+ */
+
+static bool
+Zstd_read(void *ptr, size_t size, size_t *rdsize, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t input_allocated_size = ZSTD_DStreamInSize();
+ size_t res,
+ cnt;
+
+ output->size = size;
+ output->dst = ptr;
+ output->pos = 0;
+
+ for (;;)
+ {
+ Assert(input->pos <= input->size);
+ Assert(input->size <= input_allocated_size);
+
+ /*
+ * If the input is completely consumed, start back at the beginning
+ */
+ if (input->pos == input->size)
+ {
+ /* input->size is size produced by "fread" */
+ input->size = 0;
+ /* input->pos is position consumed by decompress */
+ input->pos = 0;
+ }
+
+ /* read compressed data if we must produce more input */
+ if (input->pos == input->size)
+ {
+ cnt = fread(unconstify(void *, input->src), 1, input_allocated_size, zstdcs->fp);
+ input->size = cnt;
+
+ Assert(cnt <= input_allocated_size);
+
+ /* If we have no more input to consume, we're done */
+ if (cnt == 0)
+ break;
+ }
+
+ while (input->pos < input->size)
+ {
+ /* now decompress */
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ if (output->pos == output->size)
+ break; /* No more room for output */
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+
+ if (output->pos == output->size)
+ break; /* We read all the data that fits */
+ }
+
+ if (rdsize != NULL)
+ *rdsize = output->pos;
+
+ return true;
+}
+
+static bool
+Zstd_write(const void *ptr, size_t size, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t res,
+ cnt;
+
+ input->src = ptr;
+ input->size = size;
+ input->pos = 0;
+
+ /* Consume all input, to be flushed later */
+ while (input->pos != input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_continue);
+ if (ZSTD_isError(res))
+ {
+ zstdcs->zstderror = ZSTD_getErrorName(res);
+ return false;
+ }
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ {
+ zstdcs->zstderror = strerror(errno);
+ return false;
+ }
+ }
+
+ return size;
+}
+
+static int
+Zstd_getc(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ int ret;
+
+ if (CFH->read_func(&ret, 1, NULL, CFH) != 1)
+ {
+ if (feof(zstdcs->fp))
+ pg_fatal("could not read from input file: end of file");
+ else
+ pg_fatal("could not read from input file: %m");
+ }
+ return ret;
+}
+
+static char *
+Zstd_gets(char *buf, int len, CompressFileHandle *CFH)
+{
+ int i;
+
+ Assert(len > 0);
+
+ /*
+ * Read one byte at a time until newline or EOF. This is only used to read
+ * the list of LOs, and the I/O is buffered anyway.
+ */
+ for (i = 0; i < len - 1; ++i)
+ {
+ size_t readsz;
+
+ if (!CFH->read_func(&buf[i], 1, &readsz, CFH))
+ break;
+ if (readsz != 1)
+ break;
+ if (buf[i] == '\n')
+ {
+ ++i;
+ break;
+ }
+ }
+ buf[i] = '\0';
+ return i > 0 ? buf : NULL;
+}
+
+static bool
+Zstd_close(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ if (zstdcs->cstream)
+ {
+ size_t res,
+ cnt;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ /* Loop until the compression buffers are fully consumed */
+ for (;;)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_end);
+ if (ZSTD_isError(res))
+ {
+ zstdcs->zstderror = ZSTD_getErrorName(res);
+ return false;
+ }
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ {
+ zstdcs->zstderror = strerror(errno);
+ return false;
+ }
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ if (zstdcs->dstream)
+ {
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ }
+
+ if (fclose(zstdcs->fp) != 0)
+ return false;
+
+ pg_free(zstdcs);
+ return true;
+}
+
+static bool
+Zstd_eof(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ return feof(zstdcs->fp);
+}
+
+static bool
+Zstd_open(const char *path, int fd, const char *mode,
+ CompressFileHandle *CFH)
+{
+ FILE *fp;
+ ZstdCompressorState *zstdcs;
+
+ if (fd >= 0)
+ fp = fdopen(fd, mode);
+ else
+ fp = fopen(path, mode);
+
+ if (fp == NULL)
+ return false;
+
+ zstdcs = (ZstdCompressorState *) pg_malloc0(sizeof(*zstdcs));
+ CFH->private_data = zstdcs;
+ zstdcs->fp = fp;
+
+ if (mode[0] == 'r')
+ {
+ zstdcs->input.src = pg_malloc0(ZSTD_DStreamInSize());
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+ }
+ else if (mode[0] == 'w' || mode[0] == 'a')
+ {
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc0(zstdcs->output.size);
+ zstdcs->cstream = _ZstdCStreamParams(CFH->compression_spec);
+ if (zstdcs->cstream == NULL)
+ pg_fatal("could not initialize compression library");
+ }
+ else
+ pg_fatal("unhandled mode");
+
+ return true;
+}
+
+static bool
+Zstd_open_write(const char *path, const char *mode, CompressFileHandle *CFH)
+{
+ char fname[MAXPGPATH];
+
+ sprintf(fname, "%s.zst", path);
+ return CFH->open_func(fname, -1, mode, CFH);
+}
+
+static const char *
+Zstd_get_error(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ return zstdcs->zstderror;
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH,
+ const pg_compress_specification compression_spec)
+{
+ CFH->open_func = Zstd_open;
+ CFH->open_write_func = Zstd_open_write;
+ CFH->read_func = Zstd_read;
+ CFH->write_func = Zstd_write;
+ CFH->gets_func = Zstd_gets;
+ CFH->getc_func = Zstd_getc;
+ CFH->close_func = Zstd_close;
+ CFH->eof_func = Zstd_eof;
+ CFH->get_error_func = Zstd_get_error;
+
+ CFH->compression_spec = compression_spec;
+
+ CFH->private_data = NULL;
+}
+
+#endif /* USE_ZSTD */
diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h
new file mode 100644
index 00000000000..2aaa6b100b1
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_zstd.h
+ * Zstd interface to compress_io.c routines
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/compress_zstd.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef COMPRESS_ZSTD_H
+#define COMPRESS_ZSTD_H
+
+#include "compress_io.h"
+
+extern void InitCompressorZstd(CompressorState *cs,
+ const pg_compress_specification compression_spec);
+extern void InitCompressFileHandleZstd(CompressFileHandle *CFH,
+ const pg_compress_specification compression_spec);
+
+#endif /* COMPRESS_ZSTD_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index b2fb7ac77fd..9d59a106f36 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -5,6 +5,7 @@ pg_dump_common_sources = files(
'compress_io.c',
'compress_lz4.c',
'compress_none.c',
+ 'compress_zstd.c',
'dumputils.c',
'parallel.c',
'pg_backup_archiver.c',
@@ -19,7 +20,7 @@ pg_dump_common_sources = files(
pg_dump_common = static_library('libpgdump_common',
pg_dump_common_sources,
c_pch: pch_postgres_fe_h,
- dependencies: [frontend_code, libpq, lz4, zlib],
+ dependencies: [frontend_code, libpq, lz4, zlib, zstd],
kwargs: internal_lib_args,
)
@@ -90,6 +91,7 @@ tests += {
'env': {
'GZIP_PROGRAM': gzip.path(),
'LZ4': program_lz4.found() ? program_lz4.path() : '',
+ 'ZSTD': program_zstd.found() ? program_zstd.path() : '',
'with_icu': icu.found() ? 'yes' : 'no',
},
'tests': [
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index ab77e373e91..e8ee6b1ad86 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -2120,7 +2120,7 @@ _discoverArchiveFormat(ArchiveHandle *AH)
/*
* Check if the specified archive is a directory. If so, check if
- * there's a "toc.dat" (or "toc.dat.{gz,lz4}") file in it.
+ * there's a "toc.dat" (or "toc.dat.{gz,lz4,zst}") file in it.
*/
if (stat(AH->fSpec, &st) == 0 && S_ISDIR(st.st_mode))
{
@@ -2131,10 +2131,17 @@ _discoverArchiveFormat(ArchiveHandle *AH)
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.gz"))
return AH->format;
#endif
+
#ifdef USE_LZ4
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.lz4"))
return AH->format;
#endif
+
+#ifdef USE_ZSTD
+ if (_fileExistsInDirectory(AH->fSpec, "toc.dat.zst"))
+ return AH->format;
+#endif
+
pg_fatal("directory \"%s\" does not appear to be a valid archive (\"toc.dat\" does not exist)",
AH->fSpec);
fh = NULL; /* keep compiler quiet */
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index abaaa3b10e3..2177d5ff425 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -785,6 +785,8 @@ _PrepParallelRestore(ArchiveHandle *AH)
strlcat(fname, ".gz", sizeof(fname));
else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4)
strlcat(fname, ".lz4", sizeof(fname));
+ else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ strlcat(fname, ".zst", sizeof(fname));
if (stat(fname, &st) == 0)
te->dataLength = st.st_size;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 6abbcff6834..a426984046b 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -56,6 +56,7 @@
#include "catalog/pg_type_d.h"
#include "common/connect.h"
#include "common/relpath.h"
+#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
@@ -735,18 +736,13 @@ main(int argc, char **argv)
pg_fatal("invalid compression specification: %s",
error_detail);
- switch (compression_algorithm)
- {
- case PG_COMPRESSION_NONE:
- /* fallthrough */
- case PG_COMPRESSION_GZIP:
- /* fallthrough */
- case PG_COMPRESSION_LZ4:
- break;
- case PG_COMPRESSION_ZSTD:
- pg_fatal("compression with %s is not yet supported", "ZSTD");
- break;
- }
+ error_detail = supports_compression(compression_spec);
+ if (error_detail != NULL)
+ pg_fatal("%s", error_detail);
+
+ if (compression_spec.options & PG_COMPRESSION_OPTION_WORKERS)
+ pg_log_warning("compression option \"%s\" is not currently supported by pg_dump",
+ "workers");
/*
* Custom and directory formats are compressed by default with gzip when
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 42215f82f7a..74f23ae7f74 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -54,8 +54,9 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir;
# those lines) to validate that part of the process.
my $supports_icu = ($ENV{with_icu} eq 'yes');
-my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1");
+my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
+my $supports_zstd = check_pg_config("#define USE_ZSTD 1");
my %pgdump_runs = (
binary_upgrade => {
@@ -213,6 +214,77 @@ my %pgdump_runs = (
},
},
+ compression_zstd_custom => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=custom',
+ '--compress=zstd', "--file=$tempdir/compression_zstd_custom.dump",
+ 'postgres',
+ ],
+ restore_cmd => [
+ 'pg_restore',
+ "--file=$tempdir/compression_zstd_custom.sql",
+ "$tempdir/compression_zstd_custom.dump",
+ ],
+ command_like => {
+ command => [
+ 'pg_restore',
+ '-l', "$tempdir/compression_zstd_custom.dump",
+ ],
+ expected => qr/Compression: zstd/,
+ name => 'data content is zstd compressed'
+ },
+ },
+
+ compression_zstd_dir => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--jobs=2',
+ '--format=directory', '--compress=zstd:1',
+ "--file=$tempdir/compression_zstd_dir", 'postgres',
+ ],
+ # Give coverage for manually compressed blob.toc files during
+ # restore.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-z', '-f', '--rm',
+ "$tempdir/compression_zstd_dir/blobs.toc",
+ "-o", "$tempdir/compression_zstd_dir/blobs.toc.zst",
+ ],
+ },
+ # Verify that data files were compressed
+ glob_patterns => [
+ "$tempdir/compression_zstd_dir/toc.dat",
+ "$tempdir/compression_zstd_dir/*.dat.zst",
+ ],
+ restore_cmd => [
+ 'pg_restore', '--jobs=2',
+ "--file=$tempdir/compression_zstd_dir.sql",
+ "$tempdir/compression_zstd_dir",
+ ],
+ },
+
+ compression_zstd_plain => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=plain', '--compress=zstd',
+ "--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
+ ],
+ # Decompress the generated file to run through the tests.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-d', '-f',
+ "$tempdir/compression_zstd_plain.sql.zst",
+ "-o", "$tempdir/compression_zstd_plain.sql",
+ ],
+ },
+ },
+
clean => {
dump_cmd => [
'pg_dump',
@@ -4648,10 +4720,11 @@ foreach my $run (sort keys %pgdump_runs)
my $test_key = $run;
my $run_db = 'postgres';
- # Skip command-level tests for gzip/lz4 if there is no support for it.
+ # Skip command-level tests for gzip/lz4/zstd if the tool is not supported
if ($pgdump_runs{$run}->{compile_option} &&
(($pgdump_runs{$run}->{compile_option} eq 'gzip' && !$supports_gzip) ||
- ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4)))
+ ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4) ||
+ ($pgdump_runs{$run}->{compile_option} eq 'zstd' && !$supports_zstd)))
{
note "$run: skipped due to no $pgdump_runs{$run}->{compile_option} support";
next;
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index 58039934756..10fb51585c9 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -154,6 +154,7 @@ do
test "$f" = src/bin/pg_dump/compress_io.h && continue
test "$f" = src/bin/pg_dump/compress_lz4.h && continue
test "$f" = src/bin/pg_dump/compress_none.h && continue
+ test "$f" = src/bin/pg_dump/compress_zstd.h && continue
test "$f" = src/bin/pg_dump/parallel.h && continue
test "$f" = src/bin/pg_dump/pg_backup_archiver.h && continue
test "$f" = src/bin/pg_dump/pg_dump.h && continue
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 5c0410869f7..065acb6f50b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3937,3 +3937,4 @@ yyscan_t
z_stream
z_streamp
zic_t
+ZSTD_CStream
--
2.34.1
[text/x-diff] 0002-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch (12.7K, ../../ZCuTotQ38nwA%2FhLY@telsasoft.com/3-0002-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch)
download | inline diff:
From c82621cb1b1d5ab70dee2245be06ca29cabf8c35 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 27 Mar 2022 11:55:01 -0500
Subject: [PATCH 2/4] zstd: support long distance mode in pg_dump/basebackup
First proposed here:
20220327205020.GM28503@telsasoft.com
---
doc/src/sgml/protocol.sgml | 10 +++-
doc/src/sgml/ref/pg_basebackup.sgml | 4 +-
doc/src/sgml/ref/pg_dump.sgml | 2 +
src/backend/backup/basebackup_zstd.c | 12 ++++
src/bin/pg_basebackup/bbstreamer_zstd.c | 13 +++++
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 9 ++-
src/bin/pg_dump/compress_zstd.c | 5 ++
src/bin/pg_dump/t/002_pg_dump.pl | 3 +-
src/bin/pg_verifybackup/t/008_untar.pl | 8 +++
src/bin/pg_verifybackup/t/010_client_untar.pl | 8 +++
src/common/compression.c | 57 ++++++++++++++++++-
src/include/common/compression.h | 2 +
12 files changed, 127 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 8b5e7b1ad7f..b11d9a6ba35 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2729,7 +2729,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
level. Otherwise, it should be a comma-separated list of items,
each of the form <replaceable>keyword</replaceable> or
<replaceable>keyword=value</replaceable>. Currently, the supported
- keywords are <literal>level</literal> and <literal>workers</literal>.
+ keywords are <literal>level</literal>, <literal>long</literal> and
+ <literal>workers</literal>.
</para>
<para>
@@ -2746,6 +2747,13 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<literal>3</literal>).
</para>
+ <para>
+ The <literal>long</literal> keyword enables long-distance matching
+ mode, for improved compression ratio, at the expense of higher memory
+ use. Long-distance mode is supported only for
+ <literal>zstd</literal>.
+ </para>
+
<para>
The <literal>workers</literal> keyword sets the number of threads
that should be used for parallel compression. Parallel compression
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index db3ad9cd5eb..79d3e657c32 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -424,8 +424,8 @@ PostgreSQL documentation
level. Otherwise, it should be a comma-separated list of items,
each of the form <literal>keyword</literal> or
<literal>keyword=value</literal>.
- Currently, the supported keywords are <literal>level</literal>
- and <literal>workers</literal>.
+ Currently, the supported keywords are <literal>level</literal>,
+ <literal>long</literal>, and <literal>workers</literal>.
The detail string cannot be used when the compression method
is specified as a plain integer.
</para>
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 8de38e0fd0d..e81e35c13b3 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -681,6 +681,8 @@ PostgreSQL documentation
as though it had been fed through <application>gzip</application>,
<application>lz4</application>, or <application>zstd</application>;
but the default is not to compress.
+ With zstd compression, <literal>long</literal> mode may improve the
+ compression ratio, at the cost of increased memory use.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/backend/backup/basebackup_zstd.c b/src/backend/backup/basebackup_zstd.c
index ac6cac178a0..1bb5820c884 100644
--- a/src/backend/backup/basebackup_zstd.c
+++ b/src/backend/backup/basebackup_zstd.c
@@ -118,6 +118,18 @@ bbsink_zstd_begin_backup(bbsink *sink)
compress->workers, ZSTD_getErrorName(ret)));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(mysink->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret)));
+ }
+
/*
* We need our own buffer, because we're going to pass different data to
* the next sink than what gets passed to us.
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index fe17d6df4ef..fba391e2a0f 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -106,6 +106,19 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, pg_compress_specification *comp
compress->workers, ZSTD_getErrorName(ret));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(streamer->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ {
+ pg_log_error("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret));
+ exit(1);
+ }
+ }
+
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
streamer->zstd_outBuf.size = streamer->base.bbs_buffer.maxlen;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index b60cb78a0d5..4d130a7f944 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -139,7 +139,14 @@ SKIP:
'gzip:workers=3',
'invalid compression specification: compression algorithm "gzip" does not accept a worker count',
'failure on worker count for gzip'
- ],);
+ ],
+ [
+ 'gzip:long',
+ 'invalid compression specification: compression algorithm "gzip" does not support long-distance mode',
+ 'failure on long mode for gzip'
+ ],
+ );
+
for my $cft (@compression_failure_tests)
{
my $cfail = quotemeta($client_fails . $cft->[1]);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
index cf85c3a4c93..49a877ce010 100644
--- a/src/bin/pg_dump/compress_zstd.c
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -80,6 +80,11 @@ _ZstdCStreamParams(pg_compress_specification compress)
_Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_compressionLevel,
compress.level, "level");
+ if (compress.options & PG_COMPRESSION_OPTION_LONG_DISTANCE)
+ _Zstd_CCtx_setParam_or_die(cstream,
+ ZSTD_c_enableLongDistanceMatching,
+ compress.long_distance, "long");
+
return cstream;
}
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 74f23ae7f74..bb898b06bb4 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -267,11 +267,12 @@ my %pgdump_runs = (
],
},
+ # Exercise long mode for test coverage
compression_zstd_plain => {
test_key => 'compression',
compile_option => 'zstd',
dump_cmd => [
- 'pg_dump', '--format=plain', '--compress=zstd',
+ 'pg_dump', '--format=plain', '--compress=zstd:long',
"--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
],
# Decompress the generated file to run through the tests.
diff --git a/src/bin/pg_verifybackup/t/008_untar.pl b/src/bin/pg_verifybackup/t/008_untar.pl
index 3007bbe8556..05754bc8ec7 100644
--- a/src/bin/pg_verifybackup/t/008_untar.pl
+++ b/src/bin/pg_verifybackup/t/008_untar.pl
@@ -49,6 +49,14 @@ my @test_configuration = (
'decompress_program' => $ENV{'ZSTD'},
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => [ '--compress', 'server-zstd:level=1,long' ],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => ['-d'],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
});
for my $tc (@test_configuration)
diff --git a/src/bin/pg_verifybackup/t/010_client_untar.pl b/src/bin/pg_verifybackup/t/010_client_untar.pl
index f3aa0f59e29..ac51a174d14 100644
--- a/src/bin/pg_verifybackup/t/010_client_untar.pl
+++ b/src/bin/pg_verifybackup/t/010_client_untar.pl
@@ -50,6 +50,14 @@ my @test_configuration = (
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
},
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => ['--compress', 'client-zstd:level=1,long'],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => [ '-d' ],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
{
'compression_method' => 'parallel zstd',
'backup_flags' => [ '--compress', 'client-zstd:workers=3' ],
diff --git a/src/common/compression.c b/src/common/compression.c
index 2d3e56b4d62..35a7cade645 100644
--- a/src/common/compression.c
+++ b/src/common/compression.c
@@ -12,7 +12,7 @@
* Otherwise, a compression specification is a comma-separated list of items,
* each having the form keyword or keyword=value.
*
- * Currently, the only supported keywords are "level" and "workers".
+ * Currently, the supported keywords are "level", "long", and "workers".
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
*
@@ -38,6 +38,8 @@
static int expect_integer_value(char *keyword, char *value,
pg_compress_specification *result);
+static bool expect_boolean_value(char *keyword, char *value,
+ pg_compress_specification *result);
/*
* Look up a compression algorithm by name. Returns true and sets *algorithm
@@ -232,6 +234,11 @@ parse_compress_specification(pg_compress_algorithm algorithm, char *specificatio
result->workers = expect_integer_value(keyword, value, result);
result->options |= PG_COMPRESSION_OPTION_WORKERS;
}
+ else if (strcmp(keyword, "long") == 0)
+ {
+ result->long_distance = expect_boolean_value(keyword, value, result);
+ result->options |= PG_COMPRESSION_OPTION_LONG_DISTANCE;
+ }
else
result->parse_error =
psprintf(_("unrecognized compression option: \"%s\""), keyword);
@@ -289,6 +296,43 @@ expect_integer_value(char *keyword, char *value, pg_compress_specification *resu
return ivalue;
}
+/*
+ * Parse 'value' as a boolean and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1. The caller must check result->parse_error to determine if
+ * the call was successful.
+ *
+ * Valid values are: yes, no, on, off, 1, 0.
+ *
+ * Inspired by ParseVariableBool().
+ */
+static bool
+expect_boolean_value(char *keyword, char *value, pg_compress_specification *result)
+{
+ if (value == NULL)
+ return true;
+
+ if (pg_strcasecmp(value, "yes") == 0)
+ return true;
+ if (pg_strcasecmp(value, "on") == 0)
+ return true;
+ if (pg_strcasecmp(value, "1") == 0)
+ return true;
+
+ if (pg_strcasecmp(value, "no") == 0)
+ return false;
+ if (pg_strcasecmp(value, "off") == 0)
+ return false;
+ if (pg_strcasecmp(value, "0") == 0)
+ return false;
+
+ result->parse_error =
+ psprintf(_("value for compression option \"%s\" must be a boolean"),
+ keyword);
+ return false;
+}
+
/*
* Returns NULL if the compression specification string was syntactically
* valid and semantically sensible. Otherwise, returns an error message.
@@ -354,6 +398,17 @@ validate_compress_specification(pg_compress_specification *spec)
get_compress_algorithm_name(spec->algorithm));
}
+ /*
+ * Of the compression algorithms that we currently support, only zstd
+ * supports long-distance mode.
+ */
+ if ((spec->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0 &&
+ (spec->algorithm != PG_COMPRESSION_ZSTD))
+ {
+ return psprintf(_("compression algorithm \"%s\" does not support long-distance mode"),
+ get_compress_algorithm_name(spec->algorithm));
+ }
+
return NULL;
}
diff --git a/src/include/common/compression.h b/src/include/common/compression.h
index b48c173022e..38aae9dd873 100644
--- a/src/include/common/compression.h
+++ b/src/include/common/compression.h
@@ -27,6 +27,7 @@ typedef enum pg_compress_algorithm
} pg_compress_algorithm;
#define PG_COMPRESSION_OPTION_WORKERS (1 << 0)
+#define PG_COMPRESSION_OPTION_LONG_DISTANCE (1 << 1)
typedef struct pg_compress_specification
{
@@ -34,6 +35,7 @@ typedef struct pg_compress_specification
unsigned options; /* OR of PG_COMPRESSION_OPTION constants */
int level;
int workers;
+ bool long_distance;
char *parse_error; /* NULL if parsing was OK, else message */
} pg_compress_specification;
--
2.34.1
[text/x-diff] 0003-WIP-pg_dump-support-zstd-workers.patch (2.8K, ../../ZCuTotQ38nwA%2FhLY@telsasoft.com/4-0003-WIP-pg_dump-support-zstd-workers.patch)
download | inline diff:
From 1b78e26b5901de39c24677362c78391cb7c39b6a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Thu, 30 Mar 2023 17:48:57 -0500
Subject: [PATCH 3/4] WIP: pg_dump: support zstd workers
This is a separate commit since it's not essential; the zstd library is
frequently compiled without threading support, so the functionality
isn't very well-tested, and because use of zstd threads might
conceivably play poorly with pg_dump's use of threads under Windows.
Targetting postgres v17.
---
doc/src/sgml/ref/pg_dump.sgml | 8 ++++++--
src/bin/pg_dump/compress_zstd.c | 4 ++++
src/bin/pg_dump/pg_dump.c | 4 ----
3 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index e81e35c13b3..1d55ce05b21 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -681,8 +681,12 @@ PostgreSQL documentation
as though it had been fed through <application>gzip</application>,
<application>lz4</application>, or <application>zstd</application>;
but the default is not to compress.
- With zstd compression, <literal>long</literal> mode may improve the
- compression ratio, at the cost of increased memory use.
+ With zstd compression, <literal>long</literal> and
+ <literal>workers</literal> options may be specified to enable long-distance
+ matching and threaded workers, respectively.
+ Long distance mode may improve the compression ratio, at the cost of
+ increased memory use.
+ Threaded workers allow leveraging multiple CPUs during compression.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
index 49a877ce010..f1f84ad69c4 100644
--- a/src/bin/pg_dump/compress_zstd.c
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -85,6 +85,10 @@ _ZstdCStreamParams(pg_compress_specification compress)
ZSTD_c_enableLongDistanceMatching,
compress.long_distance, "long");
+ if (compress.options & PG_COMPRESSION_OPTION_WORKERS)
+ _Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
+ compress.workers, "workers");
+
return cstream;
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a426984046b..240dcdb0223 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -740,10 +740,6 @@ main(int argc, char **argv)
if (error_detail != NULL)
pg_fatal("%s", error_detail);
- if (compression_spec.options & PG_COMPRESSION_OPTION_WORKERS)
- pg_log_warning("compression option \"%s\" is not currently supported by pg_dump",
- "workers");
-
/*
* Custom and directory formats are compressed by default with gzip when
* available, not the others.
--
2.34.1
[text/x-diff] 0004-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch (4.4K, ../../ZCuTotQ38nwA%2FhLY@telsasoft.com/5-0004-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch)
download | inline diff:
From 6d8d1d6474b4a02689e21e59240188d4e621ef2a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 4 Jan 2023 21:21:53 -0600
Subject: [PATCH 4/4] TMP: pg_dump: use Zstd by default, for CI only
//-os-only: linux-meson
---
.cirrus.yml | 9 ++++++++-
src/bin/pg_dump/compress_zstd.c | 9 +++++++++
src/bin/pg_dump/pg_dump.c | 4 ++--
src/bin/pg_dump/t/002_pg_dump.pl | 14 +++++++-------
4 files changed, 26 insertions(+), 10 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 5b1747522f9..14402a0ad5c 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -267,6 +267,7 @@ LINUX_CONFIGURE_FEATURES: &LINUX_CONFIGURE_FEATURES >-
LINUX_MESON_FEATURES: &LINUX_MESON_FEATURES >-
-Dllvm=enabled
-Duuid=e2fs
+ -Dzstd=enabled
# Linux, both 32bit and 64bit
@@ -389,6 +390,9 @@ task:
configure_script: |
su postgres <<-EOF
+ mkdir subprojects
+ meson wrap install zstd
+ meson configure -D zstd:multithread=enabled --force-fallback-for=zstd
meson setup \
--buildtype=debug \
-Dcassert=true \
@@ -616,7 +620,10 @@ task:
# Use /DEBUG:FASTLINK to avoid high memory usage during linking
configure_script: |
vcvarsall x64
- meson setup --backend ninja --buildtype debug -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Db_pch=true -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" build
+ mkdir subprojects
+ meson wrap install zstd
+ meson configure -D zstd:multithread=enabled --force-fallback-for=zstd
+ meson setup --backend ninja --buildtype debug -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Db_pch=true -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" -D zstd=enabled build
build_script: |
vcvarsall x64
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
index f1f84ad69c4..c7be670b4a3 100644
--- a/src/bin/pg_dump/compress_zstd.c
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -88,6 +88,15 @@ _ZstdCStreamParams(pg_compress_specification compress)
if (compress.options & PG_COMPRESSION_OPTION_WORKERS)
_Zstd_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
compress.workers, "workers");
+ else
+ {
+ size_t res;
+
+ res = ZSTD_CCtx_setParameter(cstream, ZSTD_c_nbWorkers, 3);
+ if (ZSTD_isError(res))
+ pg_log_warning("could not set compression parameter: \"%s\": %s",
+ "workers", ZSTD_getErrorName(res));
+ }
return cstream;
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 240dcdb0223..b90a1087f30 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -747,8 +747,8 @@ main(int argc, char **argv)
if ((archiveFormat == archCustom || archiveFormat == archDirectory) &&
!user_compression_defined)
{
-#ifdef HAVE_LIBZ
- parse_compress_specification(PG_COMPRESSION_GZIP, NULL,
+#ifdef USE_ZSTD
+ parse_compress_specification(PG_COMPRESSION_ZSTD, NULL,
&compression_spec);
#else
/* Nothing to do in the default case */
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index bb898b06bb4..0a635ae9fc3 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -385,10 +385,10 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_custom_format.dump", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
- name => 'data content is gzip-compressed by default if available',
+ name => 'data content is zstd-compressed by default if available',
},
},
@@ -410,16 +410,16 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_dir_format", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
name => 'data content is gzip-compressed by default',
},
glob_patterns => [
"$tempdir/defaults_dir_format/toc.dat",
"$tempdir/defaults_dir_format/blobs.toc",
- $supports_gzip ?
- "$tempdir/defaults_dir_format/*.dat.gz" :
+ $supports_zstd ?
+ "$tempdir/defaults_dir_format/*.dat.zst" :
"$tempdir/defaults_dir_format/*.dat",
],
},
--
2.34.1
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 12:49 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 13:36 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 20:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-03 19:17 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-03 21:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-04 03:04 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-04-05 19:42 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-06 15:34 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
0 siblings, 1 reply; 46+ messages in thread
From: Tomas Vondra @ 2023-04-05 19:42 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On 4/4/23 05:04, Justin Pryzby wrote:
> On Mon, Apr 03, 2023 at 11:26:09PM +0200, Tomas Vondra wrote:
>> On 4/3/23 21:17, Justin Pryzby wrote:
>>> On Sat, Apr 01, 2023 at 10:26:01PM +0200, Tomas Vondra wrote:
>>>>> Feel free to mess around with threads (but I'd much rather see the patch
>>>>> progress for zstd:long).
>>>>
>>>> OK, understood. The long mode patch is pretty simple. IIUC it does not
>>>> change the format, i.e. in the worst case we could leave it for PG17
>>>> too. Correct?
>>>
>>> Right, libzstd only has one "format", which is the same as what's used
>>> by the commandline tool. zstd:long doesn't change the format of the
>>> output: the library just uses a larger memory buffer to allow better
>>> compression. There's no format change for zstd:workers, either.
>>
>> OK. I plan to do a bit more review/testing on this, and get it committed
>> over the next day or two, likely including the long mode. One thing I
>> noticed today is that maybe long_distance should be a bool, not int.
>> Yes, ZSTD_c_enableLongDistanceMatching() accepts int, but it'd be
>> cleaner to cast the value during a call and keep it bool otherwise.
>
> Thanks for noticing. Evidently I wrote it using "int" to get the
> feature working, and then later wrote the bool parsing bits but never
> changed the data structure.
>
> This also updates a few comments, indentation, removes a useless
> assertion, and updates the warning about zstd:workers.
>
Thanks. I've cleaned up the 0001 a little bit (a couple comment
improvements), updated the commit message and pushed it. I plan to take
care of the 0002 (long distance mode) tomorrow, and that'll be it for
PG16 I think.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 12:49 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 13:36 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 20:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-03 19:17 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-03 21:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-04 03:04 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-05 19:42 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2023-04-06 15:34 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-06 16:10 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 46+ messages in thread
From: Tomas Vondra @ 2023-04-06 15:34 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On 4/5/23 21:42, Tomas Vondra wrote:
> On 4/4/23 05:04, Justin Pryzby wrote:
>> On Mon, Apr 03, 2023 at 11:26:09PM +0200, Tomas Vondra wrote:
>>> On 4/3/23 21:17, Justin Pryzby wrote:
>>>> On Sat, Apr 01, 2023 at 10:26:01PM +0200, Tomas Vondra wrote:
>>>>>> Feel free to mess around with threads (but I'd much rather see the patch
>>>>>> progress for zstd:long).
>>>>>
>>>>> OK, understood. The long mode patch is pretty simple. IIUC it does not
>>>>> change the format, i.e. in the worst case we could leave it for PG17
>>>>> too. Correct?
>>>>
>>>> Right, libzstd only has one "format", which is the same as what's used
>>>> by the commandline tool. zstd:long doesn't change the format of the
>>>> output: the library just uses a larger memory buffer to allow better
>>>> compression. There's no format change for zstd:workers, either.
>>>
>>> OK. I plan to do a bit more review/testing on this, and get it committed
>>> over the next day or two, likely including the long mode. One thing I
>>> noticed today is that maybe long_distance should be a bool, not int.
>>> Yes, ZSTD_c_enableLongDistanceMatching() accepts int, but it'd be
>>> cleaner to cast the value during a call and keep it bool otherwise.
>>
>> Thanks for noticing. Evidently I wrote it using "int" to get the
>> feature working, and then later wrote the bool parsing bits but never
>> changed the data structure.
>>
>> This also updates a few comments, indentation, removes a useless
>> assertion, and updates the warning about zstd:workers.
>>
>
> Thanks. I've cleaned up the 0001 a little bit (a couple comment
> improvements), updated the commit message and pushed it. I plan to take
> care of the 0002 (long distance mode) tomorrow, and that'll be it for
> PG16 I think.
I looked at the long mode patch again, updated the commit message and
pushed it. I was wondering if long_mode should really be bool -
logically it is, but ZSTD_CCtx_setParameter() expects int. But I think
that's fine.
I think that's all for PG16 in this patch series. If there's more we
want to do, it'll have to wait for PG17 - Justin, can you update and
submit the patches that you think are relevant for the next CF?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 12:49 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 13:36 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 20:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-03 19:17 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-03 21:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-04 03:04 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-05 19:42 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-06 15:34 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2023-04-06 16:10 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-07 02:09 ` Re: zstd compression for pg_dump Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 46+ messages in thread
From: Justin Pryzby @ 2023-04-06 16:10 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Thu, Apr 06, 2023 at 05:34:30PM +0200, Tomas Vondra wrote:
> I looked at the long mode patch again, updated the commit message and
> pushed it. I was wondering if long_mode should really be bool -
> logically it is, but ZSTD_CCtx_setParameter() expects int. But I think
> that's fine.
Thanks!
> I think that's all for PG16 in this patch series. If there's more we want to
> do, it'll have to wait for PG17 -
Yes
> Justin, can you update and submit the patches that you think are relevant for
> the next CF?
Yeah.
It sounds like a shiny new feature, but it's not totally clear if it's safe
here or even how useful it is. (It might be like my patch for
wal_compression=zstd:level, and Michael's for toast_compression=zstd, neither
of which saw any support).
Last year's basebackup thread had some interesting comments about safety of
threads, although pg_dump's considerations may be different.
The patch itself is trivial, so it'd be fine to wait until PG16 is released to
get some experience. If someone else wanted to do that, it'd be fine with me.
--
Justin
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-31 23:16 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 12:49 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 13:36 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 20:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-03 19:17 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-03 21:26 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-04 03:04 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-04-05 19:42 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-06 15:34 ` Re: zstd compression for pg_dump Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-06 16:10 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-04-07 02:09 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 46+ messages in thread
From: Tom Lane @ 2023-04-07 02:09 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>; Jacob Champion <jchampion@timescale.com>; pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
Justin Pryzby <pryzby@telsasoft.com> writes:
> On Thu, Apr 06, 2023 at 05:34:30PM +0200, Tomas Vondra wrote:
>> I think that's all for PG16 in this patch series. If there's more we want to
>> do, it'll have to wait for PG17 -
> Yes
Shouldn't the CF entry be closed as committed? It's certainly
making the cfbot unhappy because the patch-of-record doesn't apply.
regards, tom lane
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Re: zstd compression for pg_dump Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-03-28 22:33 ` Jacob Champion <jchampion@timescale.com>
1 sibling, 0 replies; 46+ messages in thread
From: Jacob Champion @ 2023-03-28 22:33 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@postgresql.org, gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>
On Wed, Mar 15, 2023 at 9:50 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> On Fri, Mar 10, 2023 at 12:48:13PM -0800, Jacob Champion wrote:
> > It looks like pg_dump's meson.build is missing dependencies on zstd
> > (meson couldn't find the headers in the subproject without them).
>
> I saw that this was added for LZ4, but I hadn't added it for zstd since
> I didn't run into an issue without it. Could you check that what I've
> added works for your case ?
I thought I replied to this, sorry -- your newest patchset builds
correctly with subprojects, so the new dependency looks good to me.
Thanks!
> > Hm. Best I can tell, the CloneArchive() machinery is supposed to be
> > handling safety for this, by isolating each thread's state. I don't feel
> > comfortable pronouncing this new addition safe or not, because I'm not
> > sure I understand what the comments in the format-specific _Clone()
> > callbacks are saying yet.
>
> My line of reasoning for unix is that pg_dump forks before any calls to
> zstd. Nothing zstd does ought to affect the pg_dump layer. But that
> doesn't apply to pg_dump under windows. This is an opened question. If
> there's no solid answer, I could disable/ignore the option (maybe only
> under windows).
To (maybe?) move this forward a bit, note that pg_backup_custom's
_Clone() function makes sure that there is no active compressor state
at the beginning of the new thread. pg_backup_directory's
implementation has no such provision. And I don't think it can,
because the parent thread might have concurrently set one up -- see
the directory-specific implementation of _CloseArchive(). Perhaps we
should just NULL out those fields after the copy, instead?
To illustrate why I think this is tough to characterize: if I've read
the code correctly, the _Clone() and CloneArchive() implementations
are running concurrently with code that is actively modifying the
ArchiveHandle and the lclContext. So safety is only ensured to the
extent that we keep track of which fields threads are allowed to
touch, and I don't have that mental model.
--Jacob
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: zstd compression for pg_dump
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-26 01:22 ` Re: zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
@ 2023-03-05 17:47 ` Justin Pryzby <pryzby@telsasoft.com>
1 sibling, 0 replies; 46+ messages in thread
From: Justin Pryzby @ 2023-03-05 17:47 UTC (permalink / raw)
To: pgsql-hackers; +Cc: gkokolatos@pm.me, Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Thomas Munro <thomas.munro@gmail.com>; Dipesh Pandit <dipesh.pandit@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Mark Dilger <mark.dilger@enterprisedb.com>; Jacob Champion <jchampion@timescale.com>
On Sat, Feb 25, 2023 at 07:22:27PM -0600, Justin Pryzby wrote:
> On Fri, Feb 24, 2023 at 01:18:40PM -0600, Justin Pryzby wrote:
> > This is a draft patch - review is welcome and would help to get this
> > ready to be considererd for v16, if desired.
> >
> > I'm going to add this thread to the old CF entry.
> > https://commitfest.postgresql.org/31/2888/
>
> This resolves cfbot warnings: windows and cppcheck.
> And refactors zstd routines.
> And updates docs.
> And includes some fixes for earlier patches that these patches conflicts
> with/depends on.
This rebases over the TAP and doc fixes to LZ4.
And adds necessary ENV to makefile and meson.
And adds an annoying boilerplate header.
And removes supports_compression(), which is what I think Tomas meant
when referring to "annoying unsupported cases".
And updates zstd.c: fix an off-by-one, allocate in init depending on
readF/writeF, do not reset the input buffer on each iteration, and show
parameter name in errors.
I'd appreciate help checking that this is doing the right things and
works correctly with zstd threaded workers. The zstd API says: "use one
different context per thread for parallel execution" and "For parallel
execution, use one separate ZSTD_CStream per thread".
https://github.com/facebook/zstd/blob/dev/lib/zstd.h
I understand that to mean that, if pg_dump *itself* were using threads,
then each thread would need to call ZSTD_createCStream(). pg_dump isn't
threaded, so there's nothing special needed, right?
Except that, under windows, pg_dump -Fd -j actually uses threads instead
of forking. I *think* that's still safe, since the pgdump threads are
created *before* calling zstd functions (see _PrintTocData and
_StartData of the custom and directory formats), so it happens naturally
that there's a separate zstd stream for each thread of pgdump.
--
Justin
Attachments:
[text/x-diff] 0001-pg_dump-zstd-compression.patch (28.8K, ../../20230305174758.GA11636@telsasoft.com/2-0001-pg_dump-zstd-compression.patch)
download | inline diff:
From 6a8b88a3dd37d24ebfdaa6a96505476b8a1efe92 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 7 Jan 2023 15:45:06 -0600
Subject: [PATCH 1/3] pg_dump: zstd compression
Previously proposed at: 20201221194924.GI30237@telsasoft.com
---
doc/src/sgml/ref/pg_dump.sgml | 8 +-
src/bin/pg_dump/Makefile | 2 +
src/bin/pg_dump/compress_io.c | 79 ++--
src/bin/pg_dump/compress_zstd.c | 515 ++++++++++++++++++++++++++
src/bin/pg_dump/compress_zstd.h | 9 +
src/bin/pg_dump/meson.build | 2 +
src/bin/pg_dump/pg_backup_archiver.c | 28 +-
src/bin/pg_dump/pg_backup_directory.c | 2 +
src/bin/pg_dump/pg_dump.c | 13 -
src/bin/pg_dump/t/002_pg_dump.pl | 79 +++-
src/tools/pginclude/cpluspluscheck | 1 +
11 files changed, 654 insertions(+), 84 deletions(-)
create mode 100644 src/bin/pg_dump/compress_zstd.c
create mode 100644 src/bin/pg_dump/compress_zstd.h
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 334e4b7fd14..1fb66be1818 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -330,8 +330,9 @@ PostgreSQL documentation
machine-readable format that <application>pg_restore</application>
can read. A directory format archive can be manipulated with
standard Unix tools; for example, files in an uncompressed archive
- can be compressed with the <application>gzip</application> or
- <application>lz4</application> tools.
+ can be compressed with the <application>gzip</application>,
+ <application>lz4</application>, or
+ <application>zstd</application> tools.
This format is compressed by default using <literal>gzip</literal>
and also supports parallel dumps.
</para>
@@ -655,7 +656,8 @@ PostgreSQL documentation
<para>
Specify the compression method and/or the compression level to use.
The compression method can be set to <literal>gzip</literal>,
- <literal>lz4</literal>, or <literal>none</literal> for no compression.
+ <literal>lz4</literal>, <literal>zstd</literal>,
+ or <literal>none</literal> for no compression.
A compression detail string can optionally be specified. If the
detail string is an integer, it specifies the compression level.
Otherwise, it should be a comma-separated list of items, each of the
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index eb8f59459a1..24de7593a6a 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -18,6 +18,7 @@ include $(top_builddir)/src/Makefile.global
export GZIP_PROGRAM=$(GZIP)
export LZ4
+export ZSTD
export with_icu
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
@@ -29,6 +30,7 @@ OBJS = \
compress_io.o \
compress_lz4.o \
compress_none.o \
+ compress_zstd.o \
dumputils.o \
parallel.o \
pg_backup_archiver.o \
diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c
index ce06f1eac9c..a3c2f36bb67 100644
--- a/src/bin/pg_dump/compress_io.c
+++ b/src/bin/pg_dump/compress_io.c
@@ -52,8 +52,8 @@
*
* InitDiscoverCompressFileHandle tries to infer the compression by the
* filename suffix. If the suffix is not yet known then it tries to simply
- * open the file and if it fails, it tries to open the same file with the .gz
- * suffix, and then again with the .lz4 suffix.
+ * open the file and if it fails, it tries to open the same file with
+ * compressed suffixes.
*
* IDENTIFICATION
* src/bin/pg_dump/compress_io.c
@@ -69,6 +69,7 @@
#include "compress_io.h"
#include "compress_lz4.h"
#include "compress_none.h"
+#include "compress_zstd.h"
#include "pg_backup_utils.h"
/*----------------------
@@ -76,36 +77,6 @@
*----------------------
*/
-/*
- * Checks whether a compression algorithm is supported.
- *
- * On success returns NULL, otherwise returns a malloc'ed string which can be
- * used by the caller in an error message.
- */
-char *
-supports_compression(const pg_compress_specification compression_spec)
-{
- const pg_compress_algorithm algorithm = compression_spec.algorithm;
- bool supported = false;
-
- if (algorithm == PG_COMPRESSION_NONE)
- supported = true;
-#ifdef HAVE_LIBZ
- if (algorithm == PG_COMPRESSION_GZIP)
- supported = true;
-#endif
-#ifdef USE_LZ4
- if (algorithm == PG_COMPRESSION_LZ4)
- supported = true;
-#endif
-
- if (!supported)
- return psprintf("this build does not support compression with %s",
- get_compress_algorithm_name(algorithm));
-
- return NULL;
-}
-
/*----------------------
* Compressor API
*----------------------
@@ -130,6 +101,8 @@ AllocateCompressor(const pg_compress_specification compression_spec,
InitCompressorGzip(cs, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressorLZ4(cs, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressorZstd(cs, compression_spec);
return cs;
}
@@ -196,20 +169,30 @@ InitCompressFileHandle(const pg_compress_specification compression_spec)
InitCompressFileHandleGzip(CFH, compression_spec);
else if (compression_spec.algorithm == PG_COMPRESSION_LZ4)
InitCompressFileHandleLZ4(CFH, compression_spec);
+ else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ InitCompressFileHandleZstd(CFH, compression_spec);
return CFH;
}
+static bool
+check_compressed_file(const char *path, char **fname, char *ext)
+{
+ free_keep_errno(*fname);
+ *fname = psprintf("%s.%s", path, ext);
+ return (access(*fname, F_OK) == 0);
+}
+
/*
* Open a file for reading. 'path' is the file to open, and 'mode' should
* be either "r" or "rb".
*
* If the file at 'path' contains the suffix of a supported compression method,
- * currently this includes ".gz" and ".lz4", then this compression will be used
+ * currently this includes ".gz", ".lz4" and ".zst", then this compression will be used
* throughout. Otherwise the compression will be inferred by iteratively trying
* to open the file at 'path', first as is, then by appending known compression
* suffixes. So if you pass "foo" as 'path', this will open either "foo" or
- * "foo.gz" or "foo.lz4", trying in that order.
+ * "foo.{gz,lz4,zst}", trying in that order.
*
* On failure, return NULL with an error code in errno.
*/
@@ -237,28 +220,12 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode)
/* avoid unused warning if it is not built with compression */
if (exists)
compression_spec.algorithm = PG_COMPRESSION_NONE;
-#ifdef HAVE_LIBZ
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.gz", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_GZIP;
- }
-#endif
-#ifdef USE_LZ4
- if (!exists)
- {
- free_keep_errno(fname);
- fname = psprintf("%s.lz4", path);
- exists = (stat(fname, &st) == 0);
-
- if (exists)
- compression_spec.algorithm = PG_COMPRESSION_LZ4;
- }
-#endif
+ else if (check_compressed_file(path, &fname, "gz"))
+ compression_spec.algorithm = PG_COMPRESSION_GZIP;
+ else if (check_compressed_file(path, &fname, "lz4"))
+ compression_spec.algorithm = PG_COMPRESSION_LZ4;
+ else if (check_compressed_file(path, &fname, "zst"))
+ compression_spec.algorithm = PG_COMPRESSION_ZSTD;
}
CFH = InitCompressFileHandle(compression_spec);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
new file mode 100644
index 00000000000..d4c54d6a1dd
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -0,0 +1,515 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_zstd.c
+ * Routines for archivers to write a Zstd compressed data stream.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/compress_zstd.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "pg_backup_utils.h"
+#include "compress_zstd.h"
+
+#ifndef USE_ZSTD
+
+void
+InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec)
+{
+ pg_fatal("this build does not support compression with %s", "ZSTD");
+}
+
+#else
+
+#include <zstd.h>
+
+typedef struct ZstdCompressorState
+{
+ /* This is a normal file to which we read/write compressed data */
+ FILE *fp;
+
+ ZSTD_CStream *cstream;
+ ZSTD_DStream *dstream;
+ ZSTD_outBuffer output;
+ ZSTD_inBuffer input;
+
+ /* pointer to a static string like from strerror(), for Zstd_write() */
+ const char *zstderror;
+} ZstdCompressorState;
+
+static ZSTD_CStream *ZstdCStreamParams(pg_compress_specification compress);
+static void EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs);
+static void WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen);
+static void ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs);
+
+static void
+ZSTD_CCtx_setParam_or_die(ZSTD_CStream *cstream,
+ ZSTD_cParameter param, int value, char *paramname)
+{
+ size_t res;
+
+ res = ZSTD_CCtx_setParameter(cstream, param, value);
+ if (ZSTD_isError(res))
+ pg_fatal("could not set compression parameter: \"%s\": %s",
+ paramname, ZSTD_getErrorName(res));
+}
+
+/* Return a compression stream with parameters set per argument */
+static ZSTD_CStream *
+ZstdCStreamParams(pg_compress_specification compress)
+{
+ ZSTD_CStream *cstream;
+
+ cstream = ZSTD_createCStream();
+ if (cstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_compressionLevel,
+ compress.level, "level");
+
+ if (compress.options & PG_COMPRESSION_OPTION_WORKERS)
+ ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
+ compress.workers, "workers");
+
+ return cstream;
+}
+
+/* Helper function for EndCompressorZstd and WriteDataToArchiveZstd */
+static void
+ZstdWriteCommon(ArchiveHandle *AH, CompressorState *cs, bool flush)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ /* Loop while there's any input or until flushed */
+ while (input->pos != input->size || flush)
+ {
+ size_t res;
+
+ res = ZSTD_compressStream2(zstdcs->cstream, output,
+ input, flush ? ZSTD_e_end : ZSTD_e_continue);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not compress data: %s", ZSTD_getErrorName(res));
+
+ /*
+ * Extra paranoia: avoid zero-length chunks, since a zero length chunk
+ * is the EOF marker in the custom format. This should never happen
+ * but...
+ */
+ if (output->pos > 0)
+ cs->writeF(AH, output->dst, output->pos);
+
+ output->pos = 0;
+
+ if (res == 0)
+ break;
+ }
+}
+
+void
+EndCompressorZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ if (zstdcs->cstream != NULL)
+ {
+ ZstdWriteCommon(AH, cs, true);
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ if (zstdcs->dstream != NULL)
+ {
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ pg_free(zstdcs->output.dst);
+ }
+
+ pg_free(zstdcs);
+}
+
+static void
+WriteDataToArchiveZstd(ArchiveHandle *AH, CompressorState *cs,
+ const void *data, size_t dLen)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ zstdcs->input.src = data;
+ zstdcs->input.size = dLen;
+ zstdcs->input.pos = 0;
+
+ ZstdWriteCommon(AH, cs, false);
+}
+
+/* Read data from a compressed zstd archive */
+static void
+ReadDataFromArchiveZstd(ArchiveHandle *AH, CompressorState *cs)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) cs->private_data;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ size_t input_allocated_size = ZSTD_DStreamInSize();
+ size_t res;
+
+ for (;;)
+ {
+ size_t cnt;
+
+ /*
+ * Read compressed data. Note that readF can resize the buffer; the
+ * new size is tracked and used for future loops.
+ */
+ input->size = input_allocated_size;
+ cnt = cs->readF(AH, (char **) unconstify(void **, &input->src), &input->size);
+ input_allocated_size = input->size;
+ input->size = cnt;
+ input->pos = 0;
+
+ if (cnt == 0)
+ break;
+
+ /* Now decompress */
+ while (input->pos < input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ /* write to output handle */
+ ((char *) output->dst)[output->pos] = '\0';
+ ahwrite(output->dst, 1, output->pos, AH);
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+ }
+}
+
+/* Public routines that support Zstd compressed data I/O */
+void
+InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec)
+{
+ ZstdCompressorState *zstdcs;
+
+ cs->readData = ReadDataFromArchiveZstd;
+ cs->writeData = WriteDataToArchiveZstd;
+ cs->end = EndCompressorZstd;
+
+ cs->compression_spec = compression_spec;
+
+ cs->private_data = pg_malloc0(sizeof(ZstdCompressorState));
+ zstdcs = (ZstdCompressorState *) cs->private_data;
+
+ if (cs->readF != NULL)
+ {
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+
+ zstdcs->input.size = ZSTD_DStreamInSize();
+ zstdcs->input.src = pg_malloc(zstdcs->input.size);
+
+ zstdcs->output.size = ZSTD_DStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size + 1);
+ }
+
+ if (cs->writeF != NULL)
+ {
+ zstdcs->cstream = ZstdCStreamParams(cs->compression_spec);
+
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc(zstdcs->output.size);
+ zstdcs->output.pos = 0;
+ }
+}
+
+/*
+ * Compressed stream API
+ */
+
+static size_t
+Zstd_read(void *ptr, size_t size, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t input_allocated_size = ZSTD_DStreamInSize();
+ size_t res,
+ cnt;
+
+ output->size = size;
+ output->dst = ptr;
+ output->pos = 0;
+
+ for (;;)
+ {
+ Assert(input->pos <= input->size);
+ Assert(input->size <= input_allocated_size);
+
+ /* If the input is completely consumed, start back at the beginning */
+ if (input->pos == input->size)
+ {
+ /* input->size is size produced by "fread" */
+ input->size = 0;
+ /* input->pos is position consumed by decompress */
+ input->pos = 0;
+ }
+
+ /* read compressed data if we must produce more input */
+ if (input->pos == input->size)
+ {
+ cnt = fread(unconstify(void *, input->src), 1, input_allocated_size, zstdcs->fp);
+ input->size = cnt;
+
+ Assert(cnt >= 0);
+ Assert(input->size <= input_allocated_size);
+
+ /* If we have no input to consume, we're done */
+ if (cnt == 0)
+ break;
+ }
+
+ while (input->pos < input->size)
+ {
+ /* decompress */
+ res = ZSTD_decompressStream(zstdcs->dstream, output, input);
+
+ if (ZSTD_isError(res))
+ pg_fatal("could not decompress data: %s", ZSTD_getErrorName(res));
+
+ if (output->pos == output->size)
+ break; /* No more room for output */
+
+ if (res == 0)
+ break; /* End of frame */
+ }
+
+ if (output->pos == output->size)
+ break; /* We read all the data that fits */
+ }
+
+ return output->pos;
+}
+
+static size_t
+Zstd_write(const void *ptr, size_t size, CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+ size_t res,
+ cnt;
+
+ input->src = ptr;
+ input->size = size;
+ input->pos = 0;
+
+ /* Consume all input, to be flushed later */
+ while (input->pos != input->size)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_continue);
+ if (ZSTD_isError(res))
+ {
+ zstdcs->zstderror = ZSTD_getErrorName(res);
+ return -1;
+ }
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ {
+ zstdcs->zstderror = strerror(errno);
+ return -1;
+ }
+ }
+
+ return size;
+}
+
+static int
+Zstd_getc(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ int ret;
+
+ if (CFH->read_func(&ret, 1, CFH) != 1)
+ {
+ if (feof(zstdcs->fp))
+ pg_fatal("could not read from input file: end of file");
+ else
+ pg_fatal("could not read from input file: %m");
+ }
+ return ret;
+}
+
+static char *
+Zstd_gets(char *buf, int len, CompressFileHandle *CFH)
+{
+ int i,
+ res;
+
+ /*
+ * Read one byte at a time until newline or EOF. This is only used to read
+ * the list of LOs, and the I/O is buffered anyway.
+ */
+ for (i = 0; i < len - 1; ++i)
+ {
+ res = CFH->read_func(&buf[i], 1, CFH);
+ if (res != 1)
+ break;
+ if (buf[i] == '\n')
+ {
+ ++i;
+ break;
+ }
+ }
+ buf[i] = '\0';
+ return i > 0 ? buf : NULL;
+}
+
+static int
+Zstd_close(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+ int result;
+
+ if (zstdcs->cstream)
+ {
+ size_t res,
+ cnt;
+ ZSTD_inBuffer *input = &zstdcs->input;
+ ZSTD_outBuffer *output = &zstdcs->output;
+
+ /* Loop until the compression buffers are fully consumed */
+ for (;;)
+ {
+ output->pos = 0;
+ res = ZSTD_compressStream2(zstdcs->cstream, output, input, ZSTD_e_end);
+ if (ZSTD_isError(res))
+ {
+ zstdcs->zstderror = ZSTD_getErrorName(res);
+ return -1;
+ }
+
+ cnt = fwrite(output->dst, 1, output->pos, zstdcs->fp);
+ if (cnt != output->pos)
+ {
+ zstdcs->zstderror = strerror(errno);
+ return -1;
+ }
+
+ if (res == 0)
+ break;
+ }
+
+ ZSTD_freeCStream(zstdcs->cstream);
+ pg_free(zstdcs->output.dst);
+ }
+
+ if (zstdcs->dstream)
+ {
+ ZSTD_freeDStream(zstdcs->dstream);
+ pg_free(unconstify(void *, zstdcs->input.src));
+ }
+
+ result = fclose(zstdcs->fp);
+ pg_free(zstdcs);
+ return result;
+}
+
+static int
+Zstd_eof(CompressFileHandle *CFH)
+{
+ ZstdCompressorState *zstdcs = (ZstdCompressorState *) CFH->private_data;
+
+ return feof(zstdcs->fp);
+}
+
+static int
+Zstd_open(const char *path, int fd, const char *mode,
+ CompressFileHandle *CFH)
+{
+ FILE *fp;
+ ZstdCompressorState *zstdcs;
+
+ if (fd >= 0)
+ fp = fdopen(fd, mode);
+ else
+ fp = fopen(path, mode);
+
+ if (fp == NULL)
+ return 1;
+
+ CFH->private_data = pg_malloc0(sizeof(ZstdCompressorState));
+ zstdcs = (ZstdCompressorState *) CFH->private_data;
+ zstdcs->fp = fp;
+
+ if (mode[0] == 'r')
+ {
+ zstdcs->input.src = pg_malloc0(ZSTD_DStreamInSize());
+ zstdcs->dstream = ZSTD_createDStream();
+ if (zstdcs->dstream == NULL)
+ pg_fatal("could not initialize compression library");
+ }
+ else if (mode[0] == 'w' || mode[0] == 'a')
+ {
+ zstdcs->output.size = ZSTD_CStreamOutSize();
+ zstdcs->output.dst = pg_malloc0(zstdcs->output.size);
+ zstdcs->cstream = ZstdCStreamParams(CFH->compression_spec);
+ }
+ else
+ pg_fatal("unhandled mode");
+
+ return 0;
+}
+
+static int
+Zstd_open_write(const char *path, const char *mode, CompressFileHandle *CFH)
+{
+ char fname[MAXPGPATH];
+
+ sprintf(fname, "%s.zst", path);
+ return CFH->open_func(fname, -1, mode, CFH);
+}
+
+static const char *
+Zstd_get_error(CompressFileHandle *CFH)
+{
+ return strerror(errno);
+}
+
+void
+InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec)
+{
+ CFH->open_func = Zstd_open;
+ CFH->open_write_func = Zstd_open_write;
+ CFH->read_func = Zstd_read;
+ CFH->write_func = Zstd_write;
+ CFH->gets_func = Zstd_gets;
+ CFH->getc_func = Zstd_getc;
+ CFH->close_func = Zstd_close;
+ CFH->eof_func = Zstd_eof;
+ CFH->get_error_func = Zstd_get_error;
+
+ CFH->compression_spec = compression_spec;
+
+ CFH->private_data = NULL;
+}
+
+#endif /* USE_ZSTD */
diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h
new file mode 100644
index 00000000000..f36698b4c26
--- /dev/null
+++ b/src/bin/pg_dump/compress_zstd.h
@@ -0,0 +1,9 @@
+#ifndef COMPRESS_ZSTD_H
+#define COMPRESS_ZSTD_H
+
+#include "compress_io.h"
+
+extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec);
+extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec);
+
+#endif /* COMPRESS_ZSTD_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 0da476a4c34..c332ef87787 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -5,6 +5,7 @@ pg_dump_common_sources = files(
'compress_io.c',
'compress_lz4.c',
'compress_none.c',
+ 'compress_zstd.c',
'dumputils.c',
'parallel.c',
'pg_backup_archiver.c',
@@ -90,6 +91,7 @@ tests += {
'env': {
'GZIP_PROGRAM': gzip.path(),
'LZ4': program_lz4.found() ? program_lz4.path() : '',
+ 'ZSTD': program_zstd.found() ? program_zstd.path() : '',
},
'tests': [
't/001_basic.pl',
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 61ebb8fe85d..6fda84d8774 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -388,10 +388,12 @@ RestoreArchive(Archive *AHX)
{
if (te->hadDumper && (te->reqs & REQ_DATA) != 0)
{
- char *errmsg = supports_compression(AH->compression_spec);
- if (errmsg)
+ pg_compress_specification compress_spec;
+ parse_compress_specification(AH->compression_spec.algorithm,
+ NULL, &compress_spec);
+ if (compress_spec.parse_error != NULL)
pg_fatal("cannot restore from compressed archive (%s)",
- errmsg);
+ compress_spec.parse_error);
else
break;
}
@@ -2075,7 +2077,7 @@ _discoverArchiveFormat(ArchiveHandle *AH)
/*
* Check if the specified archive is a directory. If so, check if
- * there's a "toc.dat" (or "toc.dat.{gz,lz4}") file in it.
+ * there's a "toc.dat" (or "toc.dat.{gz,lz4,zst}") file in it.
*/
if (stat(AH->fSpec, &st) == 0 && S_ISDIR(st.st_mode))
{
@@ -2086,10 +2088,17 @@ _discoverArchiveFormat(ArchiveHandle *AH)
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.gz"))
return AH->format;
#endif
+
#ifdef USE_LZ4
if (_fileExistsInDirectory(AH->fSpec, "toc.dat.lz4"))
return AH->format;
#endif
+
+#ifdef USE_ZSTD
+ if (_fileExistsInDirectory(AH->fSpec, "toc.dat.zst"))
+ return AH->format;
+#endif
+
pg_fatal("directory \"%s\" does not appear to be a valid archive (\"toc.dat\" does not exist)",
AH->fSpec);
fh = NULL; /* keep compiler quiet */
@@ -3674,7 +3683,7 @@ WriteHead(ArchiveHandle *AH)
void
ReadHead(ArchiveHandle *AH)
{
- char *errmsg;
+ pg_compress_specification compress_spec;
char vmaj,
vmin,
vrev;
@@ -3745,12 +3754,13 @@ ReadHead(ArchiveHandle *AH)
else
AH->compression_spec.algorithm = PG_COMPRESSION_GZIP;
- errmsg = supports_compression(AH->compression_spec);
- if (errmsg)
+ parse_compress_specification(AH->compression_spec.algorithm,
+ NULL, &compress_spec);
+ if (compress_spec.parse_error != NULL)
{
pg_log_warning("archive is compressed, but this installation does not support compression (%s) -- no data will be available",
- errmsg);
- pg_free(errmsg);
+ compress_spec.parse_error);
+ pg_free(compress_spec.parse_error);
}
if (AH->version >= K_VERS_1_4)
diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c
index 41c2b733e3e..29845340859 100644
--- a/src/bin/pg_dump/pg_backup_directory.c
+++ b/src/bin/pg_dump/pg_backup_directory.c
@@ -785,6 +785,8 @@ _PrepParallelRestore(ArchiveHandle *AH)
strlcat(fname, ".gz", sizeof(fname));
else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4)
strlcat(fname, ".lz4", sizeof(fname));
+ else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD)
+ strlcat(fname, ".zst", sizeof(fname));
if (stat(fname, &st) == 0)
te->dataLength = st.st_size;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 74d806c77ba..08edeef2e3d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -710,19 +710,6 @@ main(int argc, char **argv)
pg_fatal("invalid compression specification: %s",
error_detail);
- switch (compression_algorithm)
- {
- case PG_COMPRESSION_NONE:
- /* fallthrough */
- case PG_COMPRESSION_GZIP:
- /* fallthrough */
- case PG_COMPRESSION_LZ4:
- break;
- case PG_COMPRESSION_ZSTD:
- pg_fatal("compression with %s is not yet supported", "ZSTD");
- break;
- }
-
/*
* Custom and directory formats are compressed by default with gzip when
* available, not the others.
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 14cd0d2d503..85dcc144f83 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -54,8 +54,9 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir;
# those lines) to validate that part of the process.
my $supports_icu = ($ENV{with_icu} eq 'yes');
-my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1");
+my $supports_lz4 = check_pg_config("#define USE_LZ4 1");
+my $supports_zstd = check_pg_config("#define USE_ZSTD 1");
my %pgdump_runs = (
binary_upgrade => {
@@ -213,6 +214,77 @@ my %pgdump_runs = (
},
},
+ compression_zstd_custom => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=custom',
+ '--compress=zstd', "--file=$tempdir/compression_zstd_custom.dump",
+ 'postgres',
+ ],
+ restore_cmd => [
+ 'pg_restore',
+ "--file=$tempdir/compression_zstd_custom.sql",
+ "$tempdir/compression_zstd_custom.dump",
+ ],
+ command_like => {
+ command => [
+ 'pg_restore',
+ '-l', "$tempdir/compression_zstd_custom.dump",
+ ],
+ expected => qr/Compression: zstd/,
+ name => 'data content is zstd compressed'
+ },
+ },
+
+ compression_zstd_dir => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--jobs=2',
+ '--format=directory', '--compress=zstd:1',
+ "--file=$tempdir/compression_zstd_dir", 'postgres',
+ ],
+ # Give coverage for manually compressed blob.toc files during
+ # restore.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-z', '-f', '--rm',
+ "$tempdir/compression_zstd_dir/blobs.toc",
+ "-o", "$tempdir/compression_zstd_dir/blobs.toc.zst",
+ ],
+ },
+ # Verify that data files were compressed
+ glob_patterns => [
+ "$tempdir/compression_zstd_dir/toc.dat",
+ "$tempdir/compression_zstd_dir/*.dat.zst",
+ ],
+ restore_cmd => [
+ 'pg_restore', '--jobs=2',
+ "--file=$tempdir/compression_zstd_dir.sql",
+ "$tempdir/compression_zstd_dir",
+ ],
+ },
+
+ compression_zstd_plain => {
+ test_key => 'compression',
+ compile_option => 'zstd',
+ dump_cmd => [
+ 'pg_dump', '--format=plain', '--compress=zstd',
+ "--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
+ ],
+ # Decompress the generated file to run through the tests.
+ compress_cmd => {
+ program => $ENV{'ZSTD'},
+ args => [
+ '-d', '-f',
+ "$tempdir/compression_zstd_plain.sql.zst",
+ "-o", "$tempdir/compression_zstd_plain.sql",
+ ],
+ },
+ },
+
clean => {
dump_cmd => [
'pg_dump',
@@ -4271,10 +4343,11 @@ foreach my $run (sort keys %pgdump_runs)
my $test_key = $run;
my $run_db = 'postgres';
- # Skip command-level tests for gzip/lz4 if there is no support for it.
+ # Skip command-level tests for gzip/lz4/zstd if the tool is not supported
if ($pgdump_runs{$run}->{compile_option} &&
(($pgdump_runs{$run}->{compile_option} eq 'gzip' && !$supports_gzip) ||
- ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4)))
+ ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4) ||
+ ($pgdump_runs{$run}->{compile_option} eq 'zstd' && !$supports_zstd)))
{
note "$run: skipped due to no $pgdump_runs{$run}->{compile_option} support";
next;
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index 58039934756..10fb51585c9 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -154,6 +154,7 @@ do
test "$f" = src/bin/pg_dump/compress_io.h && continue
test "$f" = src/bin/pg_dump/compress_lz4.h && continue
test "$f" = src/bin/pg_dump/compress_none.h && continue
+ test "$f" = src/bin/pg_dump/compress_zstd.h && continue
test "$f" = src/bin/pg_dump/parallel.h && continue
test "$f" = src/bin/pg_dump/pg_backup_archiver.h && continue
test "$f" = src/bin/pg_dump/pg_dump.h && continue
--
2.34.1
[text/x-diff] 0002-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch (2.2K, ../../20230305174758.GA11636@telsasoft.com/3-0002-TMP-pg_dump-use-Zstd-by-default-for-CI-only.patch)
download | inline diff:
From 002cf327f7309c7d2b94919626b6a972ca875ed4 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 4 Jan 2023 21:21:53 -0600
Subject: [PATCH 2/3] TMP: pg_dump: use Zstd by default, for CI only
//-os-only: warnings
---
src/bin/pg_dump/pg_dump.c | 4 ++--
src/bin/pg_dump/t/002_pg_dump.pl | 14 +++++++-------
2 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 08edeef2e3d..2f35b059a75 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -717,8 +717,8 @@ main(int argc, char **argv)
if ((archiveFormat == archCustom || archiveFormat == archDirectory) &&
!user_compression_defined)
{
-#ifdef HAVE_LIBZ
- parse_compress_specification(PG_COMPRESSION_GZIP, NULL,
+#ifdef USE_ZSTD
+ parse_compress_specification(PG_COMPRESSION_ZSTD, NULL,
&compression_spec);
#else
/* Nothing to do in the default case */
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 85dcc144f83..11a28e77f54 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -384,10 +384,10 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_custom_format.dump", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
- name => 'data content is gzip-compressed by default if available',
+ name => 'data content is zstd-compressed by default if available',
},
},
@@ -409,16 +409,16 @@ my %pgdump_runs = (
command_like => {
command =>
[ 'pg_restore', '-l', "$tempdir/defaults_dir_format", ],
- expected => $supports_gzip ?
- qr/Compression: gzip/ :
+ expected => $supports_zstd ?
+ qr/Compression: zstd/ :
qr/Compression: none/,
name => 'data content is gzip-compressed by default',
},
glob_patterns => [
"$tempdir/defaults_dir_format/toc.dat",
"$tempdir/defaults_dir_format/blobs.toc",
- $supports_gzip ?
- "$tempdir/defaults_dir_format/*.dat.gz" :
+ $supports_zstd ?
+ "$tempdir/defaults_dir_format/*.dat.zst" :
"$tempdir/defaults_dir_format/*.dat",
],
},
--
2.34.1
[text/x-diff] 0003-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch (13.1K, ../../20230305174758.GA11636@telsasoft.com/4-0003-zstd-support-long-distance-mode-in-pg_dump-basebacku.patch)
download | inline diff:
From b60bb3ce2cc7e6babc6f8b88a7a5c1f481b79345 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 27 Mar 2022 11:55:01 -0500
Subject: [PATCH 3/3] zstd: support long distance mode in pg_dump/basebackup
First proposed here:
20220327205020.GM28503@telsasoft.com
//-os-only: freebsd
---
doc/src/sgml/protocol.sgml | 10 +++-
doc/src/sgml/ref/pg_basebackup.sgml | 4 +-
doc/src/sgml/ref/pg_dump.sgml | 7 ++-
src/backend/backup/basebackup_zstd.c | 12 ++++
src/bin/pg_basebackup/bbstreamer_zstd.c | 13 +++++
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 9 ++-
src/bin/pg_dump/compress_zstd.c | 5 ++
src/bin/pg_dump/t/002_pg_dump.pl | 3 +-
src/bin/pg_verifybackup/t/008_untar.pl | 8 +++
src/bin/pg_verifybackup/t/010_client_untar.pl | 8 +++
src/common/compression.c | 57 ++++++++++++++++++-
src/include/common/compression.h | 2 +
12 files changed, 130 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 73b7f4432f3..05a887cd092 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2747,7 +2747,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
level. Otherwise, it should be a comma-separated list of items,
each of the form <replaceable>keyword</replaceable> or
<replaceable>keyword=value</replaceable>. Currently, the supported
- keywords are <literal>level</literal> and <literal>workers</literal>.
+ keywords are <literal>level</literal>, <literal>long</literal> and
+ <literal>workers</literal>.
</para>
<para>
@@ -2764,6 +2765,13 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<literal>3</literal>).
</para>
+ <para>
+ The <literal>long</literal> keyword enables long-distance matching
+ mode, for improved compression ratio, at the expense of higher memory
+ use. Long-distance mode is supported only for
+ <literal>zstd</literal>.
+ </para>
+
<para>
The <literal>workers</literal> keyword sets the number of threads
that should be used for parallel compression. Parallel compression
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index db3ad9cd5eb..79d3e657c32 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -424,8 +424,8 @@ PostgreSQL documentation
level. Otherwise, it should be a comma-separated list of items,
each of the form <literal>keyword</literal> or
<literal>keyword=value</literal>.
- Currently, the supported keywords are <literal>level</literal>
- and <literal>workers</literal>.
+ Currently, the supported keywords are <literal>level</literal>,
+ <literal>long</literal>, and <literal>workers</literal>.
The detail string cannot be used when the compression method
is specified as a plain integer.
</para>
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 1fb66be1818..261c18e14cc 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -678,8 +678,11 @@ PostgreSQL documentation
individual table-data segments, and the default is to compress using
<literal>gzip</literal> at a moderate level. For plain text output,
setting a nonzero compression level causes the entire output file to be compressed,
- as though it had been fed through <application>gzip</application> or
- <application>lz4</application>; but the default is not to compress.
+ as though it had been fed through <application>gzip</application>, or
+ <application>lz4</application>; or <application>zstd</application>,
+ but the default is not to compress.
+ With zstd compression, <literal>long</literal> mode may improve the
+ compression ratio, at the cost of increased memory use.
</para>
<para>
The tar archive format currently does not support compression at all.
diff --git a/src/backend/backup/basebackup_zstd.c b/src/backend/backup/basebackup_zstd.c
index ac6cac178a0..1bb5820c884 100644
--- a/src/backend/backup/basebackup_zstd.c
+++ b/src/backend/backup/basebackup_zstd.c
@@ -118,6 +118,18 @@ bbsink_zstd_begin_backup(bbsink *sink)
compress->workers, ZSTD_getErrorName(ret)));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(mysink->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret)));
+ }
+
/*
* We need our own buffer, because we're going to pass different data to
* the next sink than what gets passed to us.
diff --git a/src/bin/pg_basebackup/bbstreamer_zstd.c b/src/bin/pg_basebackup/bbstreamer_zstd.c
index fe17d6df4ef..fba391e2a0f 100644
--- a/src/bin/pg_basebackup/bbstreamer_zstd.c
+++ b/src/bin/pg_basebackup/bbstreamer_zstd.c
@@ -106,6 +106,19 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, pg_compress_specification *comp
compress->workers, ZSTD_getErrorName(ret));
}
+ if ((compress->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0)
+ {
+ ret = ZSTD_CCtx_setParameter(streamer->cctx,
+ ZSTD_c_enableLongDistanceMatching,
+ compress->long_distance);
+ if (ZSTD_isError(ret))
+ {
+ pg_log_error("could not set compression flag for %s: %s",
+ "long", ZSTD_getErrorName(ret));
+ exit(1);
+ }
+ }
+
/* Initialize the ZSTD output buffer. */
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;
streamer->zstd_outBuf.size = streamer->base.bbs_buffer.maxlen;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index b60cb78a0d5..4d130a7f944 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -139,7 +139,14 @@ SKIP:
'gzip:workers=3',
'invalid compression specification: compression algorithm "gzip" does not accept a worker count',
'failure on worker count for gzip'
- ],);
+ ],
+ [
+ 'gzip:long',
+ 'invalid compression specification: compression algorithm "gzip" does not support long-distance mode',
+ 'failure on long mode for gzip'
+ ],
+ );
+
for my $cft (@compression_failure_tests)
{
my $cfail = quotemeta($client_fails . $cft->[1]);
diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c
index d4c54d6a1dd..09378f62939 100644
--- a/src/bin/pg_dump/compress_zstd.c
+++ b/src/bin/pg_dump/compress_zstd.c
@@ -84,6 +84,11 @@ ZstdCStreamParams(pg_compress_specification compress)
ZSTD_CCtx_setParam_or_die(cstream, ZSTD_c_nbWorkers,
compress.workers, "workers");
+ if (compress.options & PG_COMPRESSION_OPTION_LONG_DISTANCE)
+ ZSTD_CCtx_setParam_or_die(cstream,
+ ZSTD_c_enableLongDistanceMatching,
+ compress.long_distance, "long");
+
return cstream;
}
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 11a28e77f54..634acf39840 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -267,11 +267,12 @@ my %pgdump_runs = (
],
},
+ # Exercise long mode for test coverage
compression_zstd_plain => {
test_key => 'compression',
compile_option => 'zstd',
dump_cmd => [
- 'pg_dump', '--format=plain', '--compress=zstd',
+ 'pg_dump', '--format=plain', '--compress=zstd:long',
"--file=$tempdir/compression_zstd_plain.sql.zst", 'postgres',
],
# Decompress the generated file to run through the tests.
diff --git a/src/bin/pg_verifybackup/t/008_untar.pl b/src/bin/pg_verifybackup/t/008_untar.pl
index 3007bbe8556..05754bc8ec7 100644
--- a/src/bin/pg_verifybackup/t/008_untar.pl
+++ b/src/bin/pg_verifybackup/t/008_untar.pl
@@ -49,6 +49,14 @@ my @test_configuration = (
'decompress_program' => $ENV{'ZSTD'},
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => [ '--compress', 'server-zstd:level=1,long' ],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => ['-d'],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
});
for my $tc (@test_configuration)
diff --git a/src/bin/pg_verifybackup/t/010_client_untar.pl b/src/bin/pg_verifybackup/t/010_client_untar.pl
index f3aa0f59e29..ac51a174d14 100644
--- a/src/bin/pg_verifybackup/t/010_client_untar.pl
+++ b/src/bin/pg_verifybackup/t/010_client_untar.pl
@@ -50,6 +50,14 @@ my @test_configuration = (
'decompress_flags' => ['-d'],
'enabled' => check_pg_config("#define USE_ZSTD 1")
},
+ {
+ 'compression_method' => 'zstd',
+ 'backup_flags' => ['--compress', 'client-zstd:level=1,long'],
+ 'backup_archive' => 'base.tar.zst',
+ 'decompress_program' => $ENV{'ZSTD'},
+ 'decompress_flags' => [ '-d' ],
+ 'enabled' => check_pg_config("#define USE_ZSTD 1")
+ },
{
'compression_method' => 'parallel zstd',
'backup_flags' => [ '--compress', 'client-zstd:workers=3' ],
diff --git a/src/common/compression.c b/src/common/compression.c
index 2d3e56b4d62..713a77c292d 100644
--- a/src/common/compression.c
+++ b/src/common/compression.c
@@ -12,7 +12,7 @@
* Otherwise, a compression specification is a comma-separated list of items,
* each having the form keyword or keyword=value.
*
- * Currently, the only supported keywords are "level" and "workers".
+ * Currently, the supported keywords are "level", "long", and "workers".
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
*
@@ -38,6 +38,8 @@
static int expect_integer_value(char *keyword, char *value,
pg_compress_specification *result);
+static bool expect_boolean_value(char *keyword, char *value,
+ pg_compress_specification *result);
/*
* Look up a compression algorithm by name. Returns true and sets *algorithm
@@ -232,6 +234,11 @@ parse_compress_specification(pg_compress_algorithm algorithm, char *specificatio
result->workers = expect_integer_value(keyword, value, result);
result->options |= PG_COMPRESSION_OPTION_WORKERS;
}
+ else if (strcmp(keyword, "long") == 0)
+ {
+ result->long_distance = expect_boolean_value(keyword, value, result);
+ result->options |= PG_COMPRESSION_OPTION_LONG_DISTANCE;
+ }
else
result->parse_error =
psprintf(_("unrecognized compression option: \"%s\""), keyword);
@@ -289,6 +296,43 @@ expect_integer_value(char *keyword, char *value, pg_compress_specification *resu
return ivalue;
}
+/*
+ * Parse 'value' as an boolean and return the result.
+ *
+ * If parsing fails, set result->parse_error to an appropriate message
+ * and return -1. The caller must check result->parse_error to determine if
+ * the call was successful.
+ *
+ * Valid values are: yes, no, on, off, 1, 0.
+ *
+ * Inspired by ParseVariableBool().
+ */
+static bool
+expect_boolean_value(char *keyword, char *value, pg_compress_specification *result)
+{
+ if (value == NULL)
+ return true;
+
+ if (pg_strcasecmp(value, "yes") == 0)
+ return true;
+ if (pg_strcasecmp(value, "on") == 0)
+ return true;
+ if (pg_strcasecmp(value, "1") == 0)
+ return true;
+
+ if (pg_strcasecmp(value, "no") == 0)
+ return false;
+ if (pg_strcasecmp(value, "off") == 0)
+ return false;
+ if (pg_strcasecmp(value, "0") == 0)
+ return false;
+
+ result->parse_error =
+ psprintf(_("value for compression option \"%s\" must be a boolean"),
+ keyword);
+ return false;
+}
+
/*
* Returns NULL if the compression specification string was syntactically
* valid and semantically sensible. Otherwise, returns an error message.
@@ -354,6 +398,17 @@ validate_compress_specification(pg_compress_specification *spec)
get_compress_algorithm_name(spec->algorithm));
}
+ /*
+ * Of the compression algorithms that we currently support, only zstd
+ * supports long-distance mode.
+ */
+ if ((spec->options & PG_COMPRESSION_OPTION_LONG_DISTANCE) != 0 &&
+ (spec->algorithm != PG_COMPRESSION_ZSTD))
+ {
+ return psprintf(_("compression algorithm \"%s\" does not support long-distance mode"),
+ get_compress_algorithm_name(spec->algorithm));
+ }
+
return NULL;
}
diff --git a/src/include/common/compression.h b/src/include/common/compression.h
index b48c173022e..6cf8cf396a8 100644
--- a/src/include/common/compression.h
+++ b/src/include/common/compression.h
@@ -27,6 +27,7 @@ typedef enum pg_compress_algorithm
} pg_compress_algorithm;
#define PG_COMPRESSION_OPTION_WORKERS (1 << 0)
+#define PG_COMPRESSION_OPTION_LONG_DISTANCE (1 << 1)
typedef struct pg_compress_specification
{
@@ -34,6 +35,7 @@ typedef struct pg_compress_specification
unsigned options; /* OR of PG_COMPRESSION_OPTION constants */
int level;
int workers;
+ int long_distance;
char *parse_error; /* NULL if parsing was OK, else message */
} pg_compress_specification;
--
2.34.1
^ permalink raw reply [nested|flat] 46+ messages in thread
end of thread, other threads:[~2023-04-07 02:09 UTC | newest]
Thread overview: 46+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-12-21 19:49 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2020-12-21 20:02 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-12-22 02:32 ` Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 02:53 ` Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 06:04 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-04 07:06 ` Justin Pryzby <pryzby@telsasoft.com>
2021-01-04 10:17 ` Daniil Zakhlystov <usernamedt@yandex-team.ru>
2021-03-12 22:31 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2021-01-10 21:06 ` Andreas Karlsson <andreas@proxel.se>
2021-04-08 15:19 ` David Steele <david@pgmasters.net>
2023-02-24 19:18 zstd compression for pg_dump Justin Pryzby <pryzby@telsasoft.com>
2023-02-25 04:44 ` Michael Paquier <michael@paquier.xyz>
2023-02-25 04:50 ` Justin Pryzby <pryzby@telsasoft.com>
2023-02-25 10:31 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-02-25 14:47 ` Euler Taveira <euler@eulerto.com>
2023-02-26 01:22 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 18:32 ` Jacob Champion <jchampion@timescale.com>
2023-03-03 18:55 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-03 21:38 ` Jacob Champion <jchampion@timescale.com>
2023-03-04 16:57 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-08 18:59 ` Jacob Champion <jchampion@timescale.com>
2023-03-10 20:48 ` Jacob Champion <jchampion@timescale.com>
2023-03-16 04:50 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-17 02:43 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 16:20 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-27 17:28 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-28 16:23 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-28 18:03 ` Kirk Wolak <wolakk@gmail.com>
2023-03-28 18:45 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-03-29 13:35 ` Justin Pryzby <pryzby@telsasoft.com>
2023-03-29 15:10 ` Jacob Champion <jchampion@timescale.com>
2023-03-31 23:16 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 00:11 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 00:28 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 12:49 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-01 13:36 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-01 20:26 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-03 19:17 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-03 21:26 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-04 03:04 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-05 19:42 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-06 15:34 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2023-04-06 16:10 ` Justin Pryzby <pryzby@telsasoft.com>
2023-04-07 02:09 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-03-28 22:33 ` Jacob Champion <jchampion@timescale.com>
2023-03-05 17:47 ` Justin Pryzby <pryzby@telsasoft.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox