Feedback-ID: i0fe9450f:Fastmail
Date: Sat, 23 Sep 2023 13:44:18 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Alexander Lakhin <exclusion@gmail.com>, Sergei Kornilov <sk@zsrv.org>,
	Noah Misch <noah@leadboat.com>, Tom Lane <tgl@sss.pgh.pa.us>,
	Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
	pgsql-bugs@lists.postgresql.org,
	Fujii Masao <masao.fujii@oss.nttdata.com>
Subject: Re: BUG #17928: Standby fails to decode WAL on termination of primary
Message-ID: <ZQ5tIoarrcwAlbVZ@paquier.xyz>
References: <ZQlvihvHo5JspVpV@paquier.xyz>
 <CA+hUKGJVke-Dwy8-ehuCj_cF8d94C-L3gR8G5=wA967FgFimGA@mail.gmail.com>
 <ee604ee6-f464-1502-0289-f1046c85c182@gmail.com>
 <CA+hUKGKp_Go=EqA65r6naxrmXFfD-uH3DwPbm7tRCu9d2T=VyQ@mail.gmail.com>
 <ZQpCG5QanJQXNVH-@paquier.xyz>
 <CA+hUKGKJx=S77ttAtJ_iA=oBGUf2VTnfUp-gtJEhdXCbmdkoAA@mail.gmail.com>
 <ZQ0R4kz3gN6bqS1N@paquier.xyz>
 <CA+hUKGK1+3rj3c7L9HPCc8HajTVmkTx1CEkU5=rqgV3b4OvWeQ@mail.gmail.com>
 <ZQ4r1vHcryBsSi_V@paquier.xyz>
 <CA+hUKGKs=yj55ejAwuUJQuURU=uKjYgRqEQcCi1GrP2+=jUgNQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="1QoWGjMKRv5S/hdJ"
Content-Disposition: inline
In-Reply-To: 
 <CA+hUKGKs=yj55ejAwuUJQuURU=uKjYgRqEQcCi1GrP2+=jUgNQ@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/ZQ5tIoarrcwAlbVZ%40paquier.xyz>
Precedence: bulk


--1QoWGjMKRv5S/hdJ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Sat, Sep 23, 2023 at 02:02:02PM +1200, Thomas Munro wrote:
> Hmm, copperhead (riscv) showed an unusual failure, a segfault in
> suspiciously nearby code.  I don't immediately know what that's about,
> let's see if we get more clues...

Yep, something is going on with the prefetching code:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=copperhead&dt=2023-09-22%2023%3A16%3A33

Using host libthread_db library
"/lib/riscv64-linux-gnu/libthread_db.so.1". Core was generated by
`postgres: paris: startup recovering 000000030000000000000003
'. Program terminated with signal SIGSEGV, Segmentation fault.
#0  pg_comp_crc32c_sb8 (crc=1613114916, crc@entry=4294967295,
data=data@entry=0x2af9e00d48, len=<optimized out>) at
pg_crc32c_sb8.c:56 56 uint32 a = *p4++ ^ crc;
#0  pg_comp_crc32c_sb8 (crc=1613114916, crc@entry=4294967295,
data=data@entry=0x2af9e00d48, len=<optimized out>) at
pg_crc32c_sb8.c:56
#1  0x0000002ad59a1536 in ValidXLogRecord (state=0x2af9db1fc0,
record=0x2af9e00d30, recptr=50520048) at xlogreader.c:1195
#2  0x0000002ad59a285a in XLogDecodeNextRecord
(state=state@entry=0x2af9db1fc0, nonblocking=<optimized out>) at
xlogreader.c:842
#3  0x0000002ad59a28c0 in XLogReadAhead (state=0x2af9db1fc0,
nonblocking=nonblocking@entry=false) at xlogreader.c:969
#4  0x0000002ad59a0996 in XLogPrefetcherNextBlock
(pgsr_private=184580836680, lsn=0x2af9e14618) at xlogprefetcher.c:496
#5  0x0000002ad59a11c8 in lrq_prefetch (lrq=<optimized out>) at
xlogprefetcher.c:256
#6  lrq_complete_lsn (lsn=<optimized out>, lrq=0x2af9e145c8) at
xlogprefetcher.c:294
#7  XLogPrefetcherReadRecord
(prefetcher=prefetcher@entry=0x2af9e00d48,
errmsg=errmsg@entry=0x3fec9a2bf0) at xlogprefetcher.c:1041

The stack may point out at a different issue, but perhaps this is a
matter where we're returning now XLREAD_SUCCESS where previously we
had XLREAD_FAIL, causing this code to fail thinking that the block was
valid while it's not?
--
Michael

--1QoWGjMKRv5S/hdJ
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmUObSIACgkQnvQgOdby
QH3M9Q//eqX4Rc4sM/NJG/vT55R/qBWTSQrFl5ohI0dANzlT8TGAQecnIHPf2oag
QsYkzQ1mbxpsfesACd0aEim3gMoRjyeT0HTWv6q0FkEBNAspuBYOaWY2sDtfilqb
Oed4q6tJwXsDo1d0Fp8cdOWObxU0ujymUjZzGHdEXgdB0zOIBTzNPKioUNseCHV9
MNj8ggUZsPU1XT6S4bbflfyCdJXHD/0UAbPgY/q+CWn0SoKcsiUT4zg1bLtfVObx
oYiDZSBEfVClSRcJ3Es1A9GVMEVXnH/2jgLPL34ij1jF4IiKoVXeeCsZdO3qfc7b
ZeaRtiZf9QBNpyu9x2LVEL6KA+8T/n3T2YAGSIiFt1z38Ci+m8l+jaGcOlNIMqYg
+UJ9yulzJyjCl4rXF3cjxTjbm8/pU9Y4MU1sFlTjg2KD9VPNITctlbQ01jLzxV7M
Vs8k+duxAPKQyZFZVJe0jOEugkML/tSZoOrxV4BA79SUkfHQV60dEqkVxx+V8YFS
5gdTzOybp6PcN52bbiqSC6LLrQLhHQWcDpSHiDoGgU4fT0dOrbKJFiTS8+6hrkRl
/8Ziy7BKqXSwXBGF9pguOhz1Hm6AYYS9x3bXKIphInsYLCtYrGq8Cc+H936XFfaa
pBzskNCxPoQJhGOZok2bjaeQZym8+CpNV7v3fcFIwSvY6Zoth9g=
=aSkc
-----END PGP SIGNATURE-----

--1QoWGjMKRv5S/hdJ--