Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\))
Subject: Re: Fixing WAL instability in various TAP tests
From: Mark Dilger <mark.dilger@enterprisedb.com>
In-Reply-To: <0AE9B97A-EE55-41DD-A913-CDBCAA8E2214@enterprisedb.com>
Date: Tue, 28 Sep 2021 11:43:30 -0700
Cc: Michael Paquier <michael@paquier.xyz>,
 Andrew Dunstan <andrew@dunslane.net>,
 Noah Misch <noah@leadboat.com>,
 PostgreSQL-development <pgsql-hackers@postgresql.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <2B700487-C43E-41AF-A8B3-444A12DFEC30@enterprisedb.com>
References: <32A1FDD1-9C7B-43B1-B3EE-49198DD3F887@enterprisedb.com>
 <20210925052151.GA4082552@rfd.leadboat.com>
 <2E9D28A3-1B54-446B-AC12-3BF86A2B522A@enterprisedb.com>
 <2502590.1632579467@sss.pgh.pa.us>
 <2B4C64CF-EE3F-474B-9685-6A927E5E49AE@enterprisedb.com>
 <20210925160039.GA4134968@rfd.leadboat.com>
 <193AC968-C348-4D81-88BC-384560F87500@enterprisedb.com>
 <C1D227C2-C271-4310-8C85-C5368C298622@enterprisedb.com>
 <2769443.1632773967@sss.pgh.pa.us> <YVJXEf119KrJDFdq@paquier.xyz>
 <2801531.1632795612@sss.pgh.pa.us> <2852500.1632850031@sss.pgh.pa.us>
 <0AE9B97A-EE55-41DD-A913-CDBCAA8E2214@enterprisedb.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/2B700487-C43E-41AF-A8B3-444A12DFEC30%40enterprisedb.com>
Precedence: bulk


> On Sep 28, 2021, at 11:07 AM, Mark Dilger =
<mark.dilger@enterprisedb.com> wrote:
>=20
> Looking closer at the TAP test, it's not ORDERing the result set from =
the SELECTs on either node, but it is comparing the sets for stringwise =
equality, which is certainly order dependent.

Taking the output from the buildfarm page, parsing out the first test's =
results and comparing got vs. expected for this test:

is($primary_result, $standby_result, "$test_name: query result =
matches");

the primary result had all the same rows as the standby, along with =
additional rows.  Comparing the results, they match other than rows =
missing from the standby that are present on the primary.  That seems =
consistent with the view that the query on the standby is running before =
all the data has replicated across.

However, the missing rows all have column i either 0 or 3, though the =
test round-robins i=3D0..9 as it performs the inserts.  I would expect =
the wal for the inserts to not cluster around any particular value of i. =
 The DELETE and VACUUM commands do operate on a single value of i, so =
that would make sense if the data failed to be deleted on the standby =
after successfully being deleted on the primary, but then I'd expect the =
standby to have more rows, not fewer.

Perhaps having the bloom index messed up answers that, though.  I think =
it should be easy enough to get the path to the heap main table fork and =
the bloom main index fork for both the primary and standby and do a =
filesystem comparison as part of the wal test.  That would tell us if =
they differ, and also if the differences are limited to just one or the =
other.

I'll go write that up....


=E2=80=94
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company