From: Tom Lane <tgl@sss.pgh.pa.us>
To: Noah Misch <noah@leadboat.com>
cc: pgbf@twiska.com, Michael Paquier <michael@paquier.xyz>,
        pgsql-hackers@postgresql.org
Subject: Re: XLogReadRecord() error in XlogReadTwoPhaseData()
In-reply-to: <3133373.1637385748@sss.pgh.pa.us>
References: <20211107013157.GB790288@rfd.leadboat.com>
 <YYiqxtEqhgb5gIWL@paquier.xyz> <20211112012234.GA1020314@rfd.leadboat.com>
 <2782601.1637189230@sss.pgh.pa.us>
 <20211118070506.GA1294133@rfd.leadboat.com>
 <20211120051823.GA1367142@rfd.leadboat.com>
 <3133373.1637385748@sss.pgh.pa.us>
Comments: In-reply-to Tom Lane <tgl@sss.pgh.pa.us>
	message dated "Sat, 20 Nov 2021 00:22:28 -0500"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3135409.1637387716.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Sat, 20 Nov 2021 00:55:16 -0500
Message-ID: <3135410.1637387716@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/3135410.1637387716%40sss.pgh.pa.us>
Precedence: bulk

I wrote:
> snapper just exhibited the same failure, too:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=3Dsnapper&dt=3D2=
021-11-18%2016%3A09%3A49

I grepped the buildfarm logs for all recent (last 3 months) occurrences of
'could not read two-phase state'.  Here's the results:

  sysname  |    branch     |      snapshot       |          stage         =
 |                                                                        =
                                         l                                =
                                                                          =
        =

-----------+---------------+---------------------+------------------------=
-+------------------------------------------------------------------------=
--------------------------------------------------------------------------=
--------------------------------------------------------------------------=
--------
 kittiwake | REL9_6_STABLE | 2021-10-24 12:01:10 | pgbenchCheck           =
 | #                   'client 1 aborted in state 3: ERROR:  could not rea=
d two-phase state from xlog at 0/158F4E0
 kittiwake | REL_13_STABLE | 2021-10-26 12:51:11 | ContribCheck-en_US.utf8=
 | #                   'pgbench: error: client 3 script 1 aborted in comma=
nd 4 query 0: ERROR:  could not read two-phase state from WAL at 0/168C8D8
 kittiwake | REL_14_STABLE | 2021-11-08 15:42:35 | ContribCheck-en_US.utf8=
 | #                   'pgbench: error: client 0 script 0 aborted in comma=
nd 3 query 0: ERROR:  could not read two-phase state from WAL at 0/17ABF48
 kittiwake | REL_13_STABLE | 2021-11-16 15:00:52 | ContribCheck-en_US.utf8=
 | #                   'pgbench: error: client 3 script 1 aborted in comma=
nd 4 query 0: ERROR:  could not read two-phase state from WAL at 0/1668020=
: incorrect resource manager data checksum in record at 0/1668020
 snapper   | REL_14_STABLE | 2021-11-18 16:09:49 | contrib-amcheckCheck   =
 | #                   'pgbench: error: client 3 script 1 aborted in comma=
nd 4 query 0: ERROR:  could not read two-phase state from WAL at 0/1770328=
: unexpected pageaddr 0/0 in log segment 000000010000000000000001, offset =
7798784
 tadarida  | REL_11_STABLE | 2021-11-11 13:29:58 | pgbenchCheck           =
 | #                   'client 3 aborted in command 3 (SQL) of script 0; E=
RROR:  could not read two-phase state from WAL at 0/1716C68
 tadarida  | REL_10_STABLE | 2021-11-12 13:01:15 | pgbenchCheck           =
 | #                   'client 4 aborted in command 3 of script 0; ERROR: =
 could not read two-phase state from WAL at 0/16F1850: invalid record leng=
th at 0/16F1850: wanted 24, got 0
 tadarida  | HEAD          | 2021-11-17 13:01:24 | contrib-amcheckCheck   =
 | #                   'pgbench: error: client 0 script 1 aborted in comma=
nd 4 query 0: ERROR:  could not read two-phase state from WAL at 0/159EF88=
: unexpected pageaddr 0/0 in log segment 000000010000000000000001, offset =
5890048

So not all are exactly 'unexpected pageaddr 0/0', but they do all
look like we read garbage data.

			regards, tom lane