pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Tomas Vondra <tomas.vondra@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: pg_dump: Add support for zstd compression
Date: Wed, 05 Apr 2023 19:40:41 +0000
Message-ID: <E1pk8zo-001XX9-M3@gemulon.postgresql.org> (raw)

pg_dump: Add support for zstd compression

Allow pg_dump to use the zstd compression, in addition to gzip/lz4. Bulk
of the new compression method is implemented in compress_zstd.{c,h},
covering the pg_dump compression APIs. The rest of the patch adds test
and makes various places aware of the new compression method.

The zstd library (which this patch relies on) supports multithreaded
compression since version 1.5. We however disallow that feature for now,
as it might interfere with parallel backups on platforms that rely on
threads (e.g. Windows). This can be improved / relaxed in the future.

This also fixes a minor issue in InitDiscoverCompressFileHandle(), which
was not updated to check if the file already has the .lz4 extension.

Adding zstd compression was originally proposed in 2020 (see the second
thread), but then was reworked to use the new compression API introduced
in e9960732a9. I've considered both threads when compiling the list of
reviewers.

Author: Justin Pryzby
Reviewed-by: Tomas Vondra, Jacob Champion, Andreas Karlsson
Discussion: https://postgr.es/m/20230224191840.GD1653@telsasoft.com
Discussion: https://postgr.es/m/20201221194924.GI30237@telsasoft.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/84adc8e20f54e93a003cd316fa1eb9b03e393288

Modified Files
--------------
doc/src/sgml/ref/pg_dump.sgml         |  13 +-
src/bin/pg_dump/Makefile              |   2 +
src/bin/pg_dump/compress_io.c         |  72 +++--
src/bin/pg_dump/compress_zstd.c       | 537 ++++++++++++++++++++++++++++++++++
src/bin/pg_dump/compress_zstd.h       |  25 ++
src/bin/pg_dump/meson.build           |   4 +-
src/bin/pg_dump/pg_backup_archiver.c  |   6 +-
src/bin/pg_dump/pg_backup_directory.c |   2 +
src/bin/pg_dump/pg_dump.c             |  25 +-
src/bin/pg_dump/t/002_pg_dump.pl      |  79 ++++-
src/tools/pginclude/cpluspluscheck    |   1 +
src/tools/pgindent/typedefs.list      |   1 +
12 files changed, 713 insertions(+), 54 deletions(-)



Message-ID: <E1pk8zo-001XX9-M3@gemulon.postgresql.org>
Permalink:  ../E1pk8zo-001XX9-M3@gemulon.postgresql.org/
Also on:    postgresql.org/message-id/E1pk8zo-001XX9-M3@gemulon.postgresql.org

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-committers@postgresql.org
  Cc: tomas.vondra@postgresql.org, pgsql-committers@lists.postgresql.org
  Subject: Re: pgsql: pg_dump: Add support for zstd compression
  In-Reply-To: <E1pk8zo-001XX9-M3@gemulon.postgresql.org>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox