pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Michael Paquier <michael@paquier.xyz>
To: gkokolatos@pm.me
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Introduce pg_receivewal gzip compression tests
Date: Tue, 13 Jul 2021 16:37:53 +0900
Message-ID: <YO1C0THrAPCkZUXW@paquier.xyz> (raw)
In-Reply-To: <thcj3N2o9EjmIdnlO26DWs8S0XhzHcNkGVwjY4GeIjlahutYHUdwhs8OyK1Bo_Znre7faKxsxfqI5BflZtTTL_8P--6bKPEuoAgOOloLjXc=@pm.me>
References: <07BK3Mk5aEOsTwGaY77qBVyf9GjoEzn8TMgHLyPGfEFPIpTEmoQuP2P4c7teesjSg-LPeUafsp1flnPeQYINMSMB_UpggJDoduB5EDYBqaQ=@protonmail.com>
	<YOvkR0F6i4AmmmUe@paquier.xyz>
	<qQjgSaFwOzw7iFfsrbo4dRn_AbfxhEl7kXpEXvsIrljGCkoocIcI_2N8vbcE85-6BgKF_C5rYPW8ZH_STh9UCmweG5kEboj64LOjElPOYRc=@pm.me>
	<YOwhwqCi+ii1dzF1@paquier.xyz>
	<nr63G0R6veCrLY30QMwxYA2aCawclWJopZiKEDN8HtKwIoIqZ79fVmSE2GxTyPD1Mk9FzRdfLCrc-BSGO6vTlDT_E2-aqtWeEiNn-qsDDzk=@pm.me>
	<9JCA0-R5_v1ayPfI36e8HMfoVP4FPcL2PqDfC09iyeaWYFyXsfKjnIuk2EYxIVBMkiqwOPSvDHOj3jnuWCaKwmHqAkyPpcPWi78EtPulyc0=@pm.me>
	<YOzyCMLW1I3mV8lD@paquier.xyz>
	<thcj3N2o9EjmIdnlO26DWs8S0XhzHcNkGVwjY4GeIjlahutYHUdwhs8OyK1Bo_Znre7faKxsxfqI5BflZtTTL_8P--6bKPEuoAgOOloLjXc=@pm.me>

On Tue, Jul 13, 2021 at 06:36:59AM +0000, gkokolatos@pm.me wrote:
> I am sorry this was not so clear. It is indeed running twice the binary
> with different flags. However the goal is not to check the flags, but
> to make certain that the partial file has now been completed. That is
> why there was code asserting that the previous FILENAME.gz.partial file
> after the second invocation is converted to FILENAME.gz.

The first run you are adding checks the same thing thanks to
pg_switch_wal(), where pg_receivewal completes the generation of
000000010000000000000002.gz and finishes with
000000010000000000000003.gz.partial.

> Additionally the second invocation of pg_receivewal is extending the
> coverage of FindStreamingStart().

Hmm.  It looks like a waste in runtime once we mix LZ4 in that as that
would mean 5 runs of pg_receivewal, but we really need only three of
them with --endpos:
- One with ZLIB compression.
- One with LZ4 compression.
- One without compression.

Do you think that we could take advantage of what is now the only run
of pg_receivewal --endpos for that?  We could make the ZLIB checks run
first, conditionally, and then let the last command with --endpos
perform a full scan of the contents in $stream_dir with the .gz files
already in place.  The addition of LZ4 would be an extra conditional
block similar to what's introduced in ZLIB, running before the last
command without compression.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

view thread (29+ messages)  latest in thread

Message-ID: <YO1C0THrAPCkZUXW@paquier.xyz>
Permalink:  ../YO1C0THrAPCkZUXW@paquier.xyz/
Also on:    postgresql.org/message-id/YO1C0THrAPCkZUXW@paquier.xyz

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: michael@paquier.xyz, gkokolatos@pm.me, pgsql-hackers@lists.postgresql.org
  Subject: Re: Introduce pg_receivewal gzip compression tests
  In-Reply-To: <YO1C0THrAPCkZUXW@paquier.xyz>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox