From: Mark Dilger <mark.dilger@enterprisedb.com>
To: Noah Misch <noah@leadboat.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Fixing WAL instability in various TAP tests
Date: Mon, 27 Sep 2021 12:58:28 -0700
Message-ID: <C1D227C2-C271-4310-8C85-C5368C298622@enterprisedb.com> (raw)
In-Reply-To: <193AC968-C348-4D81-88BC-384560F87500@enterprisedb.com>
References: <32A1FDD1-9C7B-43B1-B3EE-49198DD3F887@enterprisedb.com>
<20210925052151.GA4082552@rfd.leadboat.com>
<2E9D28A3-1B54-446B-AC12-3BF86A2B522A@enterprisedb.com>
<2502590.1632579467@sss.pgh.pa.us>
<2B4C64CF-EE3F-474B-9685-6A927E5E49AE@enterprisedb.com>
<20210925160039.GA4134968@rfd.leadboat.com>
<193AC968-C348-4D81-88BC-384560F87500@enterprisedb.com>
> On Sep 25, 2021, at 11:04 AM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>
> I took Tom's response to be, "yeah, go ahead", and am mostly waiting for the weekend to be over to see if anybody else has anything to say about it.
Here is a patch set, one patch per test. The third patch enables its test in the Makefile, which is commented as having been disabled due to the test being unstable in the build farm. Re-enabling the test might be wrong, since the instability might not have been due to WAL being recycled early. I didn't find enough history about why that test was disabled, so trying it in the build farm again is the best I can suggest. Maybe somebody else on this list knows more?
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:download all (3 files, .tar.gz)v1-0001-Harden-test-againt-premature-discarding-of-WAL.patch (application/octet-stream, 1.3K) download
From 3ce8ff704de4468bba564aa3b3185b0bfdc12670 Mon Sep 17 00:00:00 2001
From: Mark Dilger <mark.dilger@enterprisedb.com>
Date: Mon, 27 Sep 2021 11:19:41 -0700
Subject: [PATCH v1 1/3] Harden test againt premature discarding of WAL
Not using a physical replication slot between the primary and
standby makes the test brittle should default GUC settings someday
be reduced, or WAL volume someday be increased, or some combination
of those two. We happen to get lucky now that the WAL generated by
the test won't be discarded before it is used. Harden the test.
---
src/test/recovery/t/015_promotion_pages.pl | 3 +++
1 file changed, 3 insertions(+)
diff --git a/src/test/recovery/t/015_promotion_pages.pl b/src/test/recovery/t/015_promotion_pages.plindex 562c4cd3e4..b73622c7df 100644--- a/src/test/recovery/t/015_promotion_pages.pl+++ b/src/test/recovery/t/015_promotion_pages.pl@@ -22,6 +22,8 @@ EOF
# Start the primary
$alpha->start;
+$alpha->safe_psql('postgres',+ "SELECT pg_create_physical_replication_slot('bravo')");
# setup/start a standby
$alpha->backup('bkp');
@@ -29,6 +31,7 @@ my $bravo = PostgresNode->new('bravo');
$bravo->init_from_backup($alpha, 'bkp', has_streaming => 1);
$bravo->append_conf('postgresql.conf', <<EOF);
checkpoint_timeout=1h
+primary_slot_name = 'bravo'
EOF
$bravo->start;
--
2.21.1 (Apple Git-122.3)
=
v1-0002-Harden-test-againt-premature-discarding-of-WAL.patch (application/octet-stream, 1.0K) download
From 8be19b4e7f48f40a82e8f308d443f1e1540e33d2 Mon Sep 17 00:00:00 2001
From: Mark Dilger <mark.dilger@enterprisedb.com>
Date: Mon, 27 Sep 2021 11:39:29 -0700
Subject: [PATCH v1 2/3] Harden test againt premature discarding of WAL
Not setting wal_keep_size when testing pg_basebackup makes the test
brittle. Set it large enough that in practice it should not need to
be increased even if future commits change how much data gets WAL
logged.
---
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.plindex a2cb2a7679..64668a8b28 100644--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl@@ -64,6 +64,7 @@ open my $conf, '>>', "$pgdata/postgresql.conf";
print $conf "max_replication_slots = 10\n";
print $conf "max_wal_senders = 10\n";
print $conf "wal_level = replica\n";
+print $conf "wal_keep_size = 10GB\n";
close $conf;
$node->restart;
--
2.21.1 (Apple Git-122.3)
=
v1-0003-Harden-and-enable-bloom-TAP-test.patch (application/octet-stream, 1.8K) download
From 6a6fdd8e215b579ed4404e85b3d43ce9fb4f526d Mon Sep 17 00:00:00 2001
From: Mark Dilger <mark.dilger@enterprisedb.com>
Date: Mon, 27 Sep 2021 12:25:47 -0700
Subject: [PATCH v1 3/3] Harden and enable bloom TAP test
Create a physical replication slot between primary and standby that
was missing in contrib/bloom's wal TAP test. This test was disabled
with a comment about some buildfarm animals failing. Enable the
test to see if the buildfarm will now be stable.
---
contrib/bloom/Makefile | 4 +---
contrib/bloom/t/001_wal.pl | 3 +++
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/contrib/bloom/Makefile b/contrib/bloom/Makefileindex 6d7a612fe4..8a781e4388 100644--- a/contrib/bloom/Makefile+++ b/contrib/bloom/Makefile@@ -16,9 +16,7 @@ PGFILEDESC = "bloom access method - signature file based index"
REGRESS = bloom
-# Disable TAP tests for this module for now, as these are unstable on several-# buildfarm environments.-# TAP_TESTS = 1+TAP_TESTS = 1
ifdef USE_PGXS
PG_CONFIG = pg_config
diff --git a/contrib/bloom/t/001_wal.pl b/contrib/bloom/t/001_wal.plindex 55ad35926f..ef5f8b5283 100644--- a/contrib/bloom/t/001_wal.pl+++ b/contrib/bloom/t/001_wal.pl@@ -46,6 +46,8 @@ SELECT * FROM tst WHERE i = 7 AND t = 'e';
$node_primary = PostgresNode->new('primary');
$node_primary->init(allows_streaming => 1);
$node_primary->start;
+$node_primary->safe_psql('postgres',+ "SELECT pg_create_physical_replication_slot('standby')");
my $backup_name = 'my_backup';
# Take backup
@@ -55,6 +57,7 @@ $node_primary->backup($backup_name);
$node_standby = PostgresNode->new('standby');
$node_standby->init_from_backup($node_primary, $backup_name,
has_streaming => 1);
+$node_standby->append_conf('postgresql.conf', "primary_slot_name = 'standby'");
$node_standby->start;
# Create some bloom index on primary
--
2.21.1 (Apple Git-122.3)
=
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: mark.dilger@enterprisedb.com, noah@leadboat.com, tgl@sss.pgh.pa.us
Subject: Re: Fixing WAL instability in various TAP tests
In-Reply-To: <C1D227C2-C271-4310-8C85-C5368C298622@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox