Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mnTi4-0006IL-AK for pgsql-hackers@arkaria.postgresql.org; Wed, 17 Nov 2021 22:47:20 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mnTi2-0002Tg-W2 for pgsql-hackers@arkaria.postgresql.org; Wed, 17 Nov 2021 22:47:18 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mnTi2-0002TX-N0 for pgsql-hackers@lists.postgresql.org; Wed, 17 Nov 2021 22:47:18 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mnTi0-0007Za-0z for pgsql-hackers@postgresql.org; Wed, 17 Nov 2021 22:47:17 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.15.2/8.15.2) with ESMTP id 1AHMlASR2782602; Wed, 17 Nov 2021 17:47:10 -0500 From: Tom Lane To: Noah Misch cc: Michael Paquier , pgsql-hackers@postgresql.org Subject: Re: XLogReadRecord() error in XlogReadTwoPhaseData() In-reply-to: <20211112012234.GA1020314@rfd.leadboat.com> References: <20211107013157.GB790288@rfd.leadboat.com> <20211112012234.GA1020314@rfd.leadboat.com> Comments: In-reply-to Noah Misch message dated "Thu, 11 Nov 2021 17:22:34 -0800" MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <2782600.1637189230.1@sss.pgh.pa.us> Content-Transfer-Encoding: quoted-printable Date: Wed, 17 Nov 2021 17:47:10 -0500 Message-ID: <2782601.1637189230@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Noah Misch writes: > Tom Lane reported another instance today: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=3Dtadarida&dt=3D= 2021-11-11%2013%3A29%3A58 > Each of the three failures happened on a sparc64 Debian+gcc machine. I = had > tried ~8000 iterations on thorntail, another sparc64 Debian+gcc animal, > without reproducing this. >>> As a first step, let's report the actual XLogReadRecord() error messag= e. >>> Attached. >> Good catch! This looks good. > Pushed. Well, we didn't have to wait too long [1]: # at t/003_cic_2pc.pl line 143. # 'pgbench: error: client 0 script 1 aborted in command = 4 query 0: ERROR: could not read two-phase state from WAL at 0/159EF88: u= nexpected pageaddr 0/0 in log segment 000000010000000000000001, offset 589= 0048 # pgbench: error: client 2 script 3 aborted in command 2 query 0: ERROR: = canceling statement due to lock timeout # pgbench: fatal: Run was aborted; the above results are incomplete. I suppose "unexpected pageaddr 0/0" is most easily explained by supposing that XlogReadTwoPhaseData tried to read a WAL page that hadn't been written out yet. Have we got any synchronization around that? regards, tom lane [1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=3Dtadarida&dt=3D= 2021-11-17%2013%3A01%3A24