Subject: Re: AdvanceXLInsertBuffer vs. WAL segment compressibility
To: Amit Kapila <amit.kapila16@gmail.com>
References: <579297F8.7020107@anastigmatix.net>
 <CAA4eK1L7UvOS8TpUvoMuGZ4Mid=Mef-Cq5Q0M-2oYSHC-CSikw@mail.gmail.com>
 <57962061.9010506@anastigmatix.net>
 <CAB7nPqQ5NS5E-rWzd76MQHBEoc5cGHjSrUs4W6EiTxwWEx338Q@mail.gmail.com>
 <5796D9EB.6070407@anastigmatix.net>
 <CAA4eK1K_qmdeVoitjY0vyLcEBRu=o5Mwqz-+6hA+nG=KnS_kHA@mail.gmail.com>
Cc: Michael Paquier <michael.paquier@gmail.com>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>
From: Chapman Flack <chap@anastigmatix.net>
Message-ID: <57976646.70502@anastigmatix.net>
Date: Tue, 26 Jul 2016 09:31:50 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.8.0
MIME-Version: 1.0
In-Reply-To: 
 <CAA4eK1K_qmdeVoitjY0vyLcEBRu=o5Mwqz-+6hA+nG=KnS_kHA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

On 07/26/2016 08:48 AM, Amit Kapila wrote:

> general, if you have a very low WAL activity, then the final size of
> compressed WAL shouldn't be much even if you use gzip.  It seems your

9.5 pg_xlog, low activity test cluster (segment switches forced
only by checkpoint timeouts), compressed with gzip -9:

$ for i in 0*; do echo -n "$i  " && gzip -9 <$i | wc -c; done
000000010000000100000042  27072
000000010000000100000043  27075
000000010000000100000044  27077
000000010000000100000045  27073
000000010000000100000046  27075

Log from live pre-9.4 cluster, low-activity time of day, delta
compression using rsync:

2016-07-26 03:54:02 EDT (walship) INFO: using 2.39s user, 0.4s system,
9.11s on
wall:
231 byte 000000010000004600000029_000000010000004600000021_fwd
...
2016-07-26 04:54:01 EDT (walship) INFO: using 2.47s user, 0.4s system,
8.43s on
wall:
232 byte 00000001000000460000002A_000000010000004600000022_fwd
...
2016-07-26 05:54:02 EDT (walship) INFO: using 2.56s user, 0.29s system,
9.44s on
 wall:
230 byte 00000001000000460000002B_000000010000004600000023_fwd

So when I say "factor of 100", I'm understating slightly. (Those
timings, for the curious, include sending a copy offsite via ssh.)

> everything zero. Now, it might be possible to selectively initialize
> the fields that doesn't harm the methodology for archive you are using
> considering there is no other impact of same in code. However, it

Indeed, it is only the one header field that duplicates the low-
order part of the (hex) file name that breaks delta compression,
because it has always been incremented even when nothing else is
different, and it's scattered 2000 times through the file.
Would it break anything for *that* to be zero in dummy blocks?

-Chap


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers