pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pgsql: Rework compression options of pg_receivewal
2+ messages / 1 participants
[nested] [flat]

* pgsql: Rework compression options of pg_receivewal
@ 2021-11-04 02:14 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 2+ messages in thread

From: Michael Paquier @ 2021-11-04 02:14 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Rework compression options of pg_receivewal

pg_receivewal includes since cada1af the option --compress, to allow the
compression of WAL segments using gzip, with a value of 0 (the default)
meaning that no compression can be used.

This commit introduces a new option, called --compression-method, able
to use as values "none", the default, and "gzip", to make things more
extensible.  The case of --compress=0 becomes fuzzy with this option
layer, so we have made the choice to make pg_receivewal return an error
when using "none" and a non-zero compression level, meaning that the
authorized values of --compress are now [1,9] instead of [0,9].  Not
specifying --compress with "gzip" as compression method makes
pg_receivewal use the default of zlib instead (Z_DEFAULT_COMPRESSION).

The code in charge of finding the streaming start LSN when scanning the
existing archives is refactored and made more extensible.  While on it,
rename "compression" to "compression_level" in walmethods.c, to reduce
the confusion with the introduction of the compression method, even if
the tar method used by pg_basebackup does not rely on the compression
method (yet, at least), but just on the compression level (this area
could be improved more, actually).

This is in preparation for an upcoming patch that adds LZ4 support to
pg_receivewal.

Author: Georgios Kokolatos
Reviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip Kumar,
Robert Haas
Discussion: https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP...

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/d62bcc8b07f921bad105c7a826702c117ea7be58

Modified Files
--------------
doc/src/sgml/ref/pg_receivewal.sgml          |  29 ++++-
src/bin/pg_basebackup/pg_basebackup.c        |   7 +-
src/bin/pg_basebackup/pg_receivewal.c        | 167 +++++++++++++++++++--------
src/bin/pg_basebackup/receivelog.c           |   2 +-
src/bin/pg_basebackup/t/020_pg_receivewal.pl |  16 ++-
src/bin/pg_basebackup/walmethods.c           |  96 ++++++++-------
src/bin/pg_basebackup/walmethods.h           |  16 ++-
src/tools/pgindent/typedefs.list             |   1 +
8 files changed, 231 insertions(+), 103 deletions(-)



^ permalink  raw  reply  [nested|flat] 2+ messages in thread

* pgsql: Rework compression options of pg_receivewal
@ 2022-04-13 02:10 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 2+ messages in thread

From: Michael Paquier @ 2022-04-13 02:10 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Rework compression options of pg_receivewal

Since babbbb5 and the introduction of LZ4 in pg_receivewal, the
compression of the WAL archived is controlled by two options:
- --compression-method with "gzip", "none" or "lz4" as possible value.
- --compress=N to specify a compression level.  This includes a
backward-incompatible change where a value of 0 leads to a failure
instead of no compression enforced.

This commit takes advantage of a4b5754 and 3603f7c to rework the
compression options of pg_receivewal, as of:
- The removal of --compression-method.
- The extenction of --compress to use the same grammar as pg_basebackup,
with a METHOD:DETAIL format, where a METHOD is "gzip", "none" or "lz4"
and a DETAIL is a comma-separated list of options, the only keyword
supported is now "level" to control the compression level.  If only an
integer is specified as value of this option, "none" is implied on 0
and "gzip" is implied otherwise.  This brings back --compress to be
backward-compatible with ~14, while still supporting LZ4.

This has also the advantage of centralizing the set of checks used by
pg_receivewal to validate its compression options.

Author: Michael Paquier
Reviewed-by: Robert Haas, Georgios Kokolatos
Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/042a923ad53dfbe39a9d5012d6c3cf3c9c338884

Modified Files
--------------
doc/src/sgml/ref/pg_receivewal.sgml          |  45 +++++-----
src/bin/pg_basebackup/pg_receivewal.c        | 127 +++++++++++++++++++--------
src/bin/pg_basebackup/t/020_pg_receivewal.pl |  28 ++----
3 files changed, 121 insertions(+), 79 deletions(-)



^ permalink  raw  reply  [nested|flat] 2+ messages in thread


end of thread, other threads:[~2022-04-13 02:10 UTC | newest]

Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-11-04 02:14 pgsql: Rework compression options of pg_receivewal Michael Paquier <michael@paquier.xyz>
2022-04-13 02:10 pgsql: Rework compression options of pg_receivewal Michael Paquier <michael@paquier.xyz>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox