Subject: Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility
To: Daniel Gustafsson <daniel@yesql.se>
References: <579297F8.7020107@anastigmatix.net>
 <CA+TgmoaJR3s6CymT2wa9L5QR0pKY0cBthfece8MtxaBADMXw0Q@mail.gmail.com>
 <5797D953.6040806@anastigmatix.net> <20160802183347.GD32575@momjian.us>
 <CA+TgmobKFW5eVx4gd9TyiXv9iL_5DcC0-6UuJLXvwG1Ft1Pn-g@mail.gmail.com>
 <20943eba-bbb9-390c-01b1-e6704ce33c7b@iki.fi>
 <594C9457.6000708@anastigmatix.net> <59506148.1010708@anastigmatix.net>
 <2b9ed9a4-56a6-a4db-0abf-3527d993e2d4@iki.fi>
 <fe79f7c5-2bc5-f763-8908-78675ba33eba@anastigmatix.net>
 <d190140f-b6e8-5e2f-c663-c8f9617da18e@iki.fi>
 <CAB7nPqQr1D3gjx8CmHYQREidG2eeGHzx2yQ97B3t2L99GE=+kQ@mail.gmail.com>
 <5A92F0C0.7070706@anastigmatix.net>
 <C0523BD1-485D-48B5-982E-8D63E2D001B7@yesql.se>
 <5AADB130.2020206@anastigmatix.net> <4597.1521338981@sss.pgh.pa.us>
 <46FFB930-B0F3-400A-85ED-D9948C97FADF@yesql.se>
 <5AAEE00E.9050503@anastigmatix.net>
 <21C25ED8-4ED8-4AD4-B551-3E3987ECB18F@yesql.se>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Michael Paquier
 <michael.paquier@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>,
 Robert Haas <robertmhaas@gmail.com>, Bruce Momjian <bruce@momjian.us>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>
From: Chapman Flack <chap@anastigmatix.net>
Message-ID: <5AAEFF11.3070903@anastigmatix.net>
Date: Sun, 18 Mar 2018 20:06:41 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.7.0
MIME-Version: 1.0
In-Reply-To: <21C25ED8-4ED8-4AD4-B551-3E3987ECB18F@yesql.se>
Content-Type: multipart/mixed;
 boundary="------------060900020403040800010400"
Precedence: bulk

This is a multi-part message in MIME format.
--------------060900020403040800010400
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

On 03/18/18 19:28, Daniel Gustafsson wrote:
> It seems expensive to regex over BLCKSZ, but it’s probably the safest option
> and it’s not a performance critical codepath.  Feel free to whack the test
> patch over the head with the above diff.

Both patches in a single email for cfbot's enjoyment, coming right up.

-Chap

--------------060900020403040800010400
Content-Type: text/x-patch;
 name="0002-Add-test-for-ensuring-WAL-segment-is-zeroed-out.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="0002-Add-test-for-ensuring-WAL-segment-is-zeroed-out.patch"

From b2d36e3e4f4cd003cd7564a217bb15c19e1da5e2 Mon Sep 17 00:00:00 2001
From: Chapman Flack <chap@anastigmatix.net>
Date: Sun, 18 Mar 2018 20:03:04 -0400
Subject: [PATCH] Add test for ensuring WAL segment is zeroed out

Switch to a new WAL file and ensure the last block in the now
switched-away-from file is completely zeroed out.

This adds a mode to check_pg_config() where the match from the
regexp can be returned, not just the literal count of matches.
Only one match is returned (the first), but given the structure
of pg_config.h that's likely to be enough.
---
 .travis.yml                          | 47 ++++++++++++++++++++++++++++++++++++
 src/test/perl/TestLib.pm             | 18 ++++++++++++--
 src/test/recovery/t/002_archiving.pl | 19 ++++++++++++++-
 3 files changed, 81 insertions(+), 3 deletions(-)
 create mode 100644 .travis.yml

diff --git a/.travis.yml b/.travis.yml
new file mode 100644
index 0000000..386bb60
--- /dev/null
+++ b/.travis.yml
@@ -0,0 +1,47 @@
+
+sudo: required
+addons:
+  apt:
+    packages:
+      - gdb
+      - lcov
+      - libipc-run-perl
+      - libperl-dev
+      - libpython-dev
+      - tcl-dev
+      - libldap2-dev
+      - libicu-dev
+      - docbook
+      - docbook-dsssl
+      - docbook-xsl
+      - libxml2-utils
+      - openjade1.3
+      - opensp
+      - xsltproc
+language: c
+cache: ccache
+before_install:
+  - echo '/tmp/%e-%s-%p.core' | sudo tee /proc/sys/kernel/core_pattern
+  - echo "deb http://archive.ubuntu.com/ubuntu xenial main" | sudo tee /etc/apt/sources.list.d/xenial.list > /dev/null
+  - |
+    sudo tee -a /etc/apt/preferences.d/trusty > /dev/null <<EOF
+    Package: *
+    Pin: release n=xenial
+    Pin-Priority: 1
+    
+    Package: make
+    Pin: release n=xenial
+    Pin-Priority: 500
+    EOF
+  - sudo apt-get update && sudo apt-get install make
+script: ./configure --enable-debug --enable-cassert --enable-coverage --enable-tap-tests --with-tcl --with-python --with-perl --with-ldap --with-icu && make -j4 all contrib docs && make -Otarget -j3 check-world
+after_success:
+  - bash <(curl -s https://codecov.io/bash)
+after_failure:
+  - for f in ` find . -name regression.diffs ` ; do echo "========= Contents of $f" ; head -1000 $f ; done
+  - |
+    for corefile in $(find /tmp/ -name '*.core' 2>/dev/null) ; do
+      binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECFN | perl -pe "s/^.*\"(.*)\"\$/\$1/g")
+      echo dumping $corefile for $binary
+      gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile
+    done
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 7d017d4..8b9796c 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -227,15 +227,29 @@ sub append_to_file
 # that.
 sub check_pg_config
 {
-	my ($regexp) = @_;
+	my ($regexp, $value) = @_;
 	my ($stdout, $stderr);
+	my $match;
 	my $result = IPC::Run::run [ 'pg_config', '--includedir' ], '>',
 	  \$stdout, '2>', \$stderr
 	  or die "could not execute pg_config";
 	chomp($stdout);
 
 	open my $pg_config_h, '<', "$stdout/pg_config.h" or die "$!";
-	my $match = (grep {/^$regexp/} <$pg_config_h>);
+	if (defined $value)
+	{
+		while (<$pg_config_h>)
+		{
+			$_ =~ m/^$regexp/;
+			next unless defined $1;
+			$match = $1;
+			last;
+		}
+	}
+	else
+	{
+		$match = (grep {/^$regexp/} <$pg_config_h>);
+	}
 	close $pg_config_h;
 	return $match;
 }
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
index e1bd3c9..9551db1 100644
--- a/src/test/recovery/t/002_archiving.pl
+++ b/src/test/recovery/t/002_archiving.pl
@@ -3,7 +3,7 @@ use strict;
 use warnings;
 use PostgresNode;
 use TestLib;
-use Test::More tests => 1;
+use Test::More tests => 2;
 use File::Copy;
 
 # Initialize master node, doing archives
@@ -49,3 +49,20 @@ $node_standby->poll_query_until('postgres', $caughtup_query)
 my $result =
   $node_standby->safe_psql('postgres', "SELECT count(*) FROM tab_int");
 is($result, qq(1000), 'check content from archives');
+
+# Add some content, switch WAL file and read the last segment of the WAL
+# file which should be zeroed out
+my $current_file = $node_master->safe_psql('postgres',
+	"SELECT pg_walfile_name(pg_current_wal_lsn())");
+$node_master->safe_psql('postgres', "INSERT INTO tab_int VALUES (1)");
+$node_master->safe_psql('postgres', "SELECT pg_switch_wal()");
+
+my $xlog_blcksz = check_pg_config('#define XLOG_BLCKSZ (\d+)', 1);
+
+open my $fh, '<:raw', $node_master->data_dir . '/pg_wal/' . $current_file;
+# SEEK_END is defined as 2, but is not allowed when using strict mode
+seek $fh, ($xlog_blcksz * -1), 2;
+my $len = read($fh, my $bytes, $xlog_blcksz);
+close $fh;
+
+ok($bytes =~ /\A\x00*+\z/, 'make sure wal segment is zeroed');
-- 
2.7.3


--------------060900020403040800010400
Content-Type: text/plain; charset=UTF-8;
 name="0001-Zero-headers-of-unused-pages-after-WAL-switch.patch"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="0001-Zero-headers-of-unused-pages-after-WAL-switch.patch"

RnJvbSAzNTY4NGJkYWExZDI3YzNmNGI3MTk4NTQxZjNjOTJiYjJiNGNiMmY0IE1vbiBTZXAg
MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBDaGFwbWFuIEZsYWNrIDxjaGFwQGFuYXN0aWdtYXRp
eC5uZXQ+CkRhdGU6IFN1biwgMjUgRmViIDIwMTggMTE6NDQ6NDcgLTA1MDAKU3ViamVjdDog
W1BBVENIXSBaZXJvIGhlYWRlcnMgb2YgdW51c2VkIHBhZ2VzIGFmdGVyIFdBTCBzd2l0Y2gu
CgpXaGVuIHdyaXRpbmcgemVyb2VkIHBhZ2VzIHRvIHRoZSByZW1haW5kZXIgb2YgYSBXQUwg
c2VnbWVudAphZnRlciBhIFdBTCBzd2l0Y2gsIGVuc3VyZSB0aGF0IHRoZSBoZWFkZXJzIG9m
IHRob3NlIHBhZ2VzIGFyZQphbHNvIHplcm9lZCwgYXMgdGhlaXIgaW5pdGlhbGl6ZWQgdmFs
dWVzIG90aGVyd2lzZSByZWR1Y2UgdGhlCmNvbXByZXNzaWJpbGl0eSBvZiB0aGUgV0FMIHNl
Z21lbnQgZmlsZSBieSBnZW5lcmFsIHRvb2xzLgotLS0KIHNyYy9iYWNrZW5kL2FjY2Vzcy90
cmFuc2FtL3hsb2cuYyB8IDExICsrKysrKysrKystCiAxIGZpbGUgY2hhbmdlZCwgMTAgaW5z
ZXJ0aW9ucygrKSwgMSBkZWxldGlvbigtKQoKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2Fj
Y2Vzcy90cmFuc2FtL3hsb2cuYyBiL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2cu
YwppbmRleCA0N2E2YzRkLi5hOTFlYzdiIDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9hY2Nl
c3MvdHJhbnNhbS94bG9nLmMKKysrIGIvc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGxv
Zy5jCkBAIC0xNTU2LDcgKzE1NTYsMTYgQEAgQ29weVhMb2dSZWNvcmRUb1dBTChpbnQgd3Jp
dGVfbGVuLCBib29sIGlzTG9nU3dpdGNoLCBYTG9nUmVjRGF0YSAqcmRhdGEsCiAJCXsKIAkJ
CS8qIGluaXRpYWxpemUgdGhlIG5leHQgcGFnZSAoaWYgbm90IGluaXRpYWxpemVkIGFscmVh
ZHkpICovCiAJCQlXQUxJbnNlcnRMb2NrVXBkYXRlSW5zZXJ0aW5nQXQoQ3VyclBvcyk7Ci0J
CQlBZHZhbmNlWExJbnNlcnRCdWZmZXIoQ3VyclBvcywgZmFsc2UpOworCQkJLyoKKwkJCSAq
IEZpZWxkcyBpbiB0aGUgcGFnZSBoZWFkZXIgcHJlaW5pdGlhbGl6ZWQgYnkgQWR2YW5jZVhM
SW5zZXJ0QnVmZmVyCisJCQkgKiB0byBub25jb25zdGFudCB2YWx1ZXMgcmVkdWNlIHRoZSBj
b21wcmVzc2liaWxpdHkgb2YgV0FMIHNlZ21lbnRzLAorCQkJICogYW5kIGFyZW4ndCBuZWVk
ZWQgaW4gdGhlIGZyZWVzcGFjZSBmb2xsb3dpbmcgYSBzd2l0Y2ggcmVjb3JkLgorCQkJICog
UmUtemVybyB0aGF0IGhlYWRlciBhcmVhLiBUaGlzIGlzIG5vdCBwZXJmb3JtYW5jZS1jcml0
aWNhbCwgYXMKKwkJCSAqIHRoZSBtb3JlIGVtcHR5IHBhZ2VzIHRoZXJlIGFyZSBmb3IgdGhp
cyBsb29wIHRvIHRvdWNoLCB0aGUgbGVzcworCQkJICogYnVzeSB0aGUgc3lzdGVtIGlzLgor
CQkJICovCisJCQljdXJycG9zID0gR2V0WExvZ0J1ZmZlcihDdXJyUG9zKTsKKwkJCU1lbVNl
dChjdXJycG9zLCAwLCBTaXplT2ZYTG9nU2hvcnRQSEQpOwogCQkJQ3VyclBvcyArPSBYTE9H
X0JMQ0tTWjsKIAkJfQogCX0KLS0gCjIuNy4zCgo=
--------------060900020403040800010400--