From: Tom Lane <tgl@sss.pgh.pa.us>
To: Thomas Munro <thomas.munro@gmail.com>
cc: Michael Paquier <michael@paquier.xyz>,
        Alexander Lakhin <exclusion@gmail.com>, Sergei Kornilov <sk@zsrv.org>,
        Noah Misch <noah@leadboat.com>,
        Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
        pgsql-bugs@lists.postgresql.org,
        Fujii Masao <masao.fujii@oss.nttdata.com>, pgbf@twiska.com
Subject: Re: BUG #17928: Standby fails to decode WAL on termination of primary
In-reply-to: 
 <CA+hUKGLF7NS-eUUuSaRoU7XvcTJDztf9wKXGo3GmjGnUrr8EzQ@mail.gmail.com>
References: <ZQpCG5QanJQXNVH-@paquier.xyz>
 <CA+hUKGKJx=S77ttAtJ_iA=oBGUf2VTnfUp-gtJEhdXCbmdkoAA@mail.gmail.com>
 <ZQ0R4kz3gN6bqS1N@paquier.xyz>
 <CA+hUKGK1+3rj3c7L9HPCc8HajTVmkTx1CEkU5=rqgV3b4OvWeQ@mail.gmail.com>
 <ZQ4r1vHcryBsSi_V@paquier.xyz>
 <CA+hUKGKs=yj55ejAwuUJQuURU=uKjYgRqEQcCi1GrP2+=jUgNQ@mail.gmail.com>
 <ZQ5tIoarrcwAlbVZ@paquier.xyz>
 <CA+hUKGJf3Hhb2MB88-rW2di2H9XT0xr6-hd6ZjGEwdJs3A=b+Q@mail.gmail.com>
 <ZQ9zf1QO8CP4TZRO@paquier.xyz>
 <CA+hUKGKo5O_f8aBDoEMkxHmuWdFfmPPo6wL4q85b_0PDuHwZEA@mail.gmail.com>
 <ZRDNNf8Etlvuo48a@paquier.xyz>
 <CA+hUKG+cXwfAk0dEbD5CZ76p1uADGywuvb6_N4Uhziek54FZHg@mail.gmail.com>
 <64446.1695611228@sss.pgh.pa.us>
 <CA+hUKGLF7NS-eUUuSaRoU7XvcTJDztf9wKXGo3GmjGnUrr8EzQ@mail.gmail.com>
Comments: In-reply-to Thomas Munro <thomas.munro@gmail.com>
	message dated "Mon, 25 Sep 2023 16:47:12 +1300"
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-ID: <86520.1695617949.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Mon, 25 Sep 2023 00:59:09 -0400
Message-ID: <86521.1695617949@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/86521.1695617949%40sss.pgh.pa.us>
Precedence: bulk

Thomas Munro <thomas.munro@gmail.com> writes:
> On Mon, Sep 25, 2023 at 4:07=E2=80=AFPM Tom Lane <tgl@sss.pgh.pa.us> wro=
te:
>> That one is mine, so let me know if there's something particular
>> you'd like me to check.

> Thanks!  But after spending the best part of a day installing and
> compiling stuff at snail's pace, I have just this moment managed to
> reproduce this reliably on an emulator running Debian armhf (=3D armv7).
> So hopefully I can track the issue down from here without bothering
> build farm owners.

FWIW, here's what I see on mamba:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  pg_comp_crc32c_sb8 (crc=3D3197562372, crc@entry=3D4294967295, =

    data=3Ddata@entry=3D0xfda6e030, len=3D<optimized out>) at pg_crc32c_sb=
8.c:56
56                      uint32          a =3D *p4++ ^ crc;
(gdb) bt
#0  pg_comp_crc32c_sb8 (crc=3D3197562372, crc@entry=3D4294967295, =

    data=3Ddata@entry=3D0xfda6e030, len=3D<optimized out>) at pg_crc32c_sb=
8.c:56
#1  0x0195ae5c in ValidXLogRecord (state=3D0xfde5cc88, record=3D0xfda6e018=
, =

    recptr=3D<optimized out>) at xlogreader.c:1195
#2  0x0195cf18 in XLogDecodeNextRecord (state=3Dstate@entry=3D0xfde5cc88, =

    nonblocking=3D<optimized out>) at xlogreader.c:842
#3  0x0195d0c8 in XLogReadAhead (state=3D0xfde5cc88, =

    nonblocking=3Dnonblocking@entry=3Dfalse) at xlogreader.c:969
#4  0x01959760 in XLogPrefetcherNextBlock (pgsr_private=3D4259700408, =

    lsn=3D0xfda6b8a0) at xlogprefetcher.c:496
#5  0x0195a72c in lrq_prefetch (lrq=3D<optimized out>) at xlogprefetcher.c=
:256
#6  lrq_complete_lsn (lsn=3D50520048, lrq=3D0xfda6b858) at xlogprefetcher.=
c:294
#7  XLogPrefetcherReadRecord (prefetcher=3Dprefetcher@entry=3D0xfde5deb8, =

    errmsg=3Derrmsg@entry=3D0xffffdd88) at xlogprefetcher.c:1041
#8  0x01960054 in ReadRecord (xlogprefetcher=3D0xfde5deb8, emode=3Demode@e=
ntry=3D15, =

    fetching_ckpt=3Dfetching_ckpt@entry=3Dfalse, replayTLI=3DreplayTLI@ent=
ry=3D3)
    at xlogrecovery.c:3067
#9  0x01963188 in PerformWalRecovery () at xlogrecovery.c:1756
#10 0x01952594 in StartupXLOG () at xlog.c:5470
#11 0x01c26e3c in StartupProcessMain () at startup.c:267
#12 0x01c1b504 in AuxiliaryProcessMain (auxtype=3Dauxtype@entry=3DStartupP=
rocess)
    at auxprocess.c:141
#13 0x01c2276c in StartChildProcess (type=3DStartupProcess) at postmaster.=
c:5347
#14 0x01c260ec in PostmasterMain (argc=3Dargc@entry=3D4, =

    argv=3Dargv@entry=3D0xffffe1e4) at postmaster.c:1457
#15 0x01ee39d8 in main (argc=3D4, argv=3D0xffffe1e4) at main.c:198

(gdb) f 1
#1  0x0195ae5c in ValidXLogRecord (state=3D0xfde5cc88, record=3D0xfda6e018=
, =

    recptr=3D<optimized out>) at xlogreader.c:1195
1195            COMP_CRC32C(crc, ((char *) record) + SizeOfXLogRecord, rec=
ord->xl_tot_len - SizeOfXLogRecord);
(gdb) p *record
$3 =3D {xl_tot_len =3D 0, xl_xid =3D 0, xl_prev =3D 0, xl_info =3D 0 '\000=
', =

  xl_rmid =3D 0 '\000', xl_crc =3D 0}

Sure looks like ValidXLogRecord is assuming that record->xl_tot_len
can be trusted without reservation.

			regards, tom lane