From: Chapman Flack <chap@anastigmatix.net>
To: Daniel Gustafsson <daniel@yesql.se>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Michael Paquier <michael.paquier@gmail.com>
Cc: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Bruce Momjian <bruce@momjian.us>
Cc: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility
Date: Sun, 18 Mar 2018 20:06:41 -0400
Message-ID: <5AAEFF11.3070903@anastigmatix.net> (raw)
In-Reply-To: <21C25ED8-4ED8-4AD4-B551-3E3987ECB18F@yesql.se>
References: <579297F8.7020107@anastigmatix.net>
<CA+TgmoaJR3s6CymT2wa9L5QR0pKY0cBthfece8MtxaBADMXw0Q@mail.gmail.com>
<5797D953.6040806@anastigmatix.net>
<20160802183347.GD32575@momjian.us>
<CA+TgmobKFW5eVx4gd9TyiXv9iL_5DcC0-6UuJLXvwG1Ft1Pn-g@mail.gmail.com>
<20943eba-bbb9-390c-01b1-e6704ce33c7b@iki.fi>
<594C9457.6000708@anastigmatix.net>
<59506148.1010708@anastigmatix.net>
<2b9ed9a4-56a6-a4db-0abf-3527d993e2d4@iki.fi>
<fe79f7c5-2bc5-f763-8908-78675ba33eba@anastigmatix.net>
<d190140f-b6e8-5e2f-c663-c8f9617da18e@iki.fi>
<CAB7nPqQr1D3gjx8CmHYQREidG2eeGHzx2yQ97B3t2L99GE=+kQ@mail.gmail.com>
<5A92F0C0.7070706@anastigmatix.net>
<C0523BD1-485D-48B5-982E-8D63E2D001B7@yesql.se>
<5AADB130.2020206@anastigmatix.net>
<4597.1521338981@sss.pgh.pa.us>
<46FFB930-B0F3-400A-85ED-D9948C97FADF@yesql.se>
<5AAEE00E.9050503@anastigmatix.net>
<21C25ED8-4ED8-4AD4-B551-3E3987ECB18F@yesql.se>
On 03/18/18 19:28, Daniel Gustafsson wrote:
> It seems expensive to regex over BLCKSZ, but it’s probably the safest option
> and it’s not a performance critical codepath. Feel free to whack the test
> patch over the head with the above diff.
Both patches in a single email for cfbot's enjoyment, coming right up.
-Chap
Attachments:download all (2 files, .tar.gz)0002-Add-test-for-ensuring-WAL-segment-is-zeroed-out.patch (text/x-patch, 4.7K) download
From b2d36e3e4f4cd003cd7564a217bb15c19e1da5e2 Mon Sep 17 00:00:00 2001
From: Chapman Flack <chap@anastigmatix.net>
Date: Sun, 18 Mar 2018 20:03:04 -0400
Subject: [PATCH] Add test for ensuring WAL segment is zeroed out
Switch to a new WAL file and ensure the last block in the now
switched-away-from file is completely zeroed out.
This adds a mode to check_pg_config() where the match from the
regexp can be returned, not just the literal count of matches.
Only one match is returned (the first), but given the structure
of pg_config.h that's likely to be enough.
---
.travis.yml | 47 ++++++++++++++++++++++++++++++++++++
src/test/perl/TestLib.pm | 18 ++++++++++++--
src/test/recovery/t/002_archiving.pl | 19 ++++++++++++++-
3 files changed, 81 insertions(+), 3 deletions(-)
create mode 100644 .travis.yml
diff --git a/.travis.yml b/.travis.ymlnew file mode 100644index 0000000..386bb60--- /dev/null+++ b/.travis.yml@@ -0,0 +1,47 @@++sudo: required+addons:+ apt:+ packages:+ - gdb+ - lcov+ - libipc-run-perl+ - libperl-dev+ - libpython-dev+ - tcl-dev+ - libldap2-dev+ - libicu-dev+ - docbook+ - docbook-dsssl+ - docbook-xsl+ - libxml2-utils+ - openjade1.3+ - opensp+ - xsltproc+language: c+cache: ccache+before_install:+ - echo '/tmp/%e-%s-%p.core' | sudo tee /proc/sys/kernel/core_pattern+ - echo "deb http://archive.ubuntu.com/ubuntu xenial main" | sudo tee /etc/apt/sources.list.d/xenial.list > /dev/null+ - |+ sudo tee -a /etc/apt/preferences.d/trusty > /dev/null <<EOF+ Package: *+ Pin: release n=xenial+ Pin-Priority: 1+ + Package: make+ Pin: release n=xenial+ Pin-Priority: 500+ EOF+ - sudo apt-get update && sudo apt-get install make+script: ./configure --enable-debug --enable-cassert --enable-coverage --enable-tap-tests --with-tcl --with-python --with-perl --with-ldap --with-icu && make -j4 all contrib docs && make -Otarget -j3 check-world+after_success:+ - bash <(curl -s https://codecov.io/bash)+after_failure:+ - for f in ` find . -name regression.diffs ` ; do echo "========= Contents of $f" ; head -1000 $f ; done+ - |+ for corefile in $(find /tmp/ -name '*.core' 2>/dev/null) ; do+ binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECFN | perl -pe "s/^.*\"(.*)\"\$/\$1/g")+ echo dumping $corefile for $binary+ gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile+ donediff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pmindex 7d017d4..8b9796c 100644--- a/src/test/perl/TestLib.pm+++ b/src/test/perl/TestLib.pm@@ -227,15 +227,29 @@ sub append_to_file
# that.
sub check_pg_config
{
- my ($regexp) = @_;+ my ($regexp, $value) = @_;
my ($stdout, $stderr);
+ my $match;
my $result = IPC::Run::run [ 'pg_config', '--includedir' ], '>',
\$stdout, '2>', \$stderr
or die "could not execute pg_config";
chomp($stdout);
open my $pg_config_h, '<', "$stdout/pg_config.h" or die "$!";
- my $match = (grep {/^$regexp/} <$pg_config_h>);+ if (defined $value)+ {+ while (<$pg_config_h>)+ {+ $_ =~ m/^$regexp/;+ next unless defined $1;+ $match = $1;+ last;+ }+ }+ else+ {+ $match = (grep {/^$regexp/} <$pg_config_h>);+ }
close $pg_config_h;
return $match;
}
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.plindex e1bd3c9..9551db1 100644--- a/src/test/recovery/t/002_archiving.pl+++ b/src/test/recovery/t/002_archiving.pl@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 1;+use Test::More tests => 2;
use File::Copy;
# Initialize master node, doing archives
@@ -49,3 +49,20 @@ $node_standby->poll_query_until('postgres', $caughtup_query)
my $result =
$node_standby->safe_psql('postgres', "SELECT count(*) FROM tab_int");
is($result, qq(1000), 'check content from archives');
++# Add some content, switch WAL file and read the last segment of the WAL+# file which should be zeroed out+my $current_file = $node_master->safe_psql('postgres',+ "SELECT pg_walfile_name(pg_current_wal_lsn())");+$node_master->safe_psql('postgres', "INSERT INTO tab_int VALUES (1)");+$node_master->safe_psql('postgres', "SELECT pg_switch_wal()");++my $xlog_blcksz = check_pg_config('#define XLOG_BLCKSZ (\d+)', 1);++open my $fh, '<:raw', $node_master->data_dir . '/pg_wal/' . $current_file;+# SEEK_END is defined as 2, but is not allowed when using strict mode+seek $fh, ($xlog_blcksz * -1), 2;+my $len = read($fh, my $bytes, $xlog_blcksz);+close $fh;++ok($bytes =~ /\A\x00*+\z/, 'make sure wal segment is zeroed');--
2.7.3
0001-Zero-headers-of-unused-pages-after-WAL-switch.patch (text/plain, 1.5K) download
From 35684bdaa1d27c3f4b7198541f3c92bb2b4cb2f4 Mon Sep 17 00:00:00 2001
From: Chapman Flack <chap@anastigmatix.net>
Date: Sun, 25 Feb 2018 11:44:47 -0500
Subject: [PATCH] Zero headers of unused pages after WAL switch.
When writing zeroed pages to the remainder of a WAL segment
after a WAL switch, ensure that the headers of those pages are
also zeroed, as their initialized values otherwise reduce the
compressibility of the WAL segment file by general tools.
---
src/backend/access/transam/xlog.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex 47a6c4d..a91ec7b 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -1556,7 +1556,16 @@ CopyXLogRecordToWAL(int write_len, bool isLogSwitch, XLogRecData *rdata,
{
/* initialize the next page (if not initialized already) */
WALInsertLockUpdateInsertingAt(CurrPos);
- AdvanceXLInsertBuffer(CurrPos, false);+ /*+ * Fields in the page header preinitialized by AdvanceXLInsertBuffer+ * to nonconstant values reduce the compressibility of WAL segments,+ * and aren't needed in the freespace following a switch record.+ * Re-zero that header area. This is not performance-critical, as+ * the more empty pages there are for this loop to touch, the less+ * busy the system is.+ */+ currpos = GetXLogBuffer(CurrPos);+ MemSet(currpos, 0, SizeOfXLogShortPHD);
CurrPos += XLOG_BLCKSZ;
}
}
--
2.7.3
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: chap@anastigmatix.net, daniel@yesql.se, tgl@sss.pgh.pa.us, michael.paquier@gmail.com, hlinnaka@iki.fi, robertmhaas@gmail.com, bruce@momjian.us
Subject: Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility
In-Reply-To: <5AAEFF11.3070903@anastigmatix.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox