From: Tom Lane <tgl@sss.pgh.pa.us>
To: Mark Dilger <mark.dilger@enterprisedb.com>
cc: Michael Paquier <michael@paquier.xyz>,
        Andrew Dunstan <andrew@dunslane.net>, Noah Misch <noah@leadboat.com>,
        PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Fixing WAL instability in various TAP tests
In-reply-to: <2B700487-C43E-41AF-A8B3-444A12DFEC30@enterprisedb.com>
References: <32A1FDD1-9C7B-43B1-B3EE-49198DD3F887@enterprisedb.com>
 <20210925052151.GA4082552@rfd.leadboat.com>
 <2E9D28A3-1B54-446B-AC12-3BF86A2B522A@enterprisedb.com>
 <2502590.1632579467@sss.pgh.pa.us>
 <2B4C64CF-EE3F-474B-9685-6A927E5E49AE@enterprisedb.com>
 <20210925160039.GA4134968@rfd.leadboat.com>
 <193AC968-C348-4D81-88BC-384560F87500@enterprisedb.com>
 <C1D227C2-C271-4310-8C85-C5368C298622@enterprisedb.com>
 <2769443.1632773967@sss.pgh.pa.us> <YVJXEf119KrJDFdq@paquier.xyz>
 <2801531.1632795612@sss.pgh.pa.us> <2852500.1632850031@sss.pgh.pa.us>
 <0AE9B97A-EE55-41DD-A913-CDBCAA8E2214@enterprisedb.com>
 <2B700487-C43E-41AF-A8B3-444A12DFEC30@enterprisedb.com>
Comments: In-reply-to Mark Dilger <mark.dilger@enterprisedb.com>
	message dated "Tue, 28 Sep 2021 11:43:30 -0700"
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0"
Content-ID: <2870096.1632855526.0@sss.pgh.pa.us>
Date: Tue, 28 Sep 2021 15:00:13 -0400
Message-ID: <2870209.1632855613@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/2870209.1632855613%40sss.pgh.pa.us>
Precedence: bulk

------- =_aaaaaaaaaa0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <2870096.1632855526.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable

Mark Dilger <mark.dilger@enterprisedb.com> writes:
> Perhaps having the bloom index messed up answers that, though.  I think =
it should be easy enough to get the path to the heap main table fork and t=
he bloom main index fork for both the primary and standby and do a filesys=
tem comparison as part of the wal test.  That would tell us if they differ=
, and also if the differences are limited to just one or the other.

I think that's probably overkill, and definitely out-of-scope for
contrib/bloom.  If we fear that WAL replay is not reproducing the data
accurately, we should be testing for that in some more centralized place.

Anyway, I confirmed my diagnosis by adding a delay in WAL apply
(0001 below); that makes this test fall over spectacularly.
And 0002 fixes it.  So I propose to push 0002 as soon as the
v14 release freeze ends.

Should we back-patch 0002?  I'm inclined to think so.  Should
we then also back-patch enablement of the bloom test?  Less
sure about that, but I'd lean to doing so.  A test that appears
to be there but isn't actually invoked is pretty misleading.

			regards, tom lane


------- =_aaaaaaaaaa0
Content-Type: text/x-diff; name="0001-break-it.patch"; charset="us-ascii"
Content-ID: <2870096.1632855526.2@sss.pgh.pa.us>
Content-Description: 0001-break-it.patch
Content-Transfer-Encoding: quoted-printable

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transa=
m/xlog.c
index e51a7a749d..eecbe57aee 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -7370,6 +7370,9 @@ StartupXLOG(void)
 			{
 				bool		switchedTLI =3D false;
 =

+				if (random() < INT_MAX/100)
+					pg_usleep(100000);
+
 #ifdef WAL_DEBUG
 				if (XLOG_DEBUG ||
 					(rmid =3D=3D RM_XACT_ID && trace_recovery_messages <=3D DEBUG2) ||

------- =_aaaaaaaaaa0
Content-Type: text/x-diff; name="0002-fix-it.patch"; charset="us-ascii"
Content-ID: <2870096.1632855526.3@sss.pgh.pa.us>
Content-Description: 0002-fix-it.patch
Content-Transfer-Encoding: quoted-printable

diff --git a/contrib/bloom/t/001_wal.pl b/contrib/bloom/t/001_wal.pl
index 55ad35926f..be8916a8eb 100644
--- a/contrib/bloom/t/001_wal.pl
+++ b/contrib/bloom/t/001_wal.pl
@@ -16,12 +16,10 @@ sub test_index_replay
 {
 	my ($test_name) =3D @_;
 =

+	local $Test::Builder::Level =3D $Test::Builder::Level + 1;
+
 	# Wait for standby to catch up
-	my $applname =3D $node_standby->name;
-	my $caughtup_query =3D
-	  "SELECT pg_current_wal_lsn() <=3D write_lsn FROM pg_stat_replication W=
HERE application_name =3D '$applname';";
-	$node_primary->poll_query_until('postgres', $caughtup_query)
-	  or die "Timed out while waiting for standby 1 to catch up";
+	$node_primary->wait_for_catchup($node_standby);
 =

 	my $queries =3D qq(SET enable_seqscan=3Doff;
 SET enable_bitmapscan=3Don;

------- =_aaaaaaaaaa0--