From: Tom Lane <tgl@sss.pgh.pa.us>
To: Andres Freund <andres@anarazel.de>
cc: Robert Haas <robertmhaas@gmail.com>, Daniel Gustafsson <daniel@yesql.se>,
        Kyotaro Horiguchi <horikyota.ntt@gmail.com>, deniel1495@mail.ru,
        Ibrar Ahmed <ibrar.ahmad@gmail.com>, tejeswarm@hotmail.com,
        hlinnaka <hlinnaka@iki.fi>,
        Masahiko Sawada <masahiko.sawada@2ndquadrant.com>,
        "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>,
        Daniel Wood <hexexpert@comcast.net>
Subject: Re: Corruption during WAL replay
In-reply-to: <20220325045438.enwakjqhrafzq5f2@alap3.anarazel.de>
References: 
 <CA+TgmoYG4-P8PUWPVZyCjR0_kUwfY0KFRJcO7WGO2fi7icGTpA@mail.gmail.com>
 <CA+TgmoYhFvsNedWw9eRJXPevy-RytpRMofbyETtU1SxUCG16-Q@mail.gmail.com>
 <3167047.1648169107@sss.pgh.pa.us>
 <CA+Tgmoa_TAJsUnFWzTevg=FzgXFyiHLJAY_L-jMDA=mH=ytysg@mail.gmail.com>
 <3170060.1648171358@sss.pgh.pa.us> <3173721.1648173548@sss.pgh.pa.us>
 <20220325022010.iqdj37tjqrqwxfrh@alap3.anarazel.de>
 <20220325024302.lhhi7jaapyetqviv@alap3.anarazel.de>
 <20220325034301.htu27xf54xjgyoca@alap3.anarazel.de>
 <3185871.1648181300@sss.pgh.pa.us>
 <20220325045438.enwakjqhrafzq5f2@alap3.anarazel.de>
Comments: In-reply-to Andres Freund <andres@anarazel.de>
	message dated "Thu, 24 Mar 2022 21:54:38 -0700"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3192025.1648185780.1@sss.pgh.pa.us>
Date: Fri, 25 Mar 2022 01:23:00 -0400
Message-ID: <3192026.1648185780@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/3192026.1648185780%40sss.pgh.pa.us>
Precedence: bulk

Andres Freund <andres@anarazel.de> writes:
> I do see that the LSN that ends up on the page is the same across a few runs
> of the test on serinus. Which presumably differs between different
> animals. Surprised that it's this predictable - but I guess the run is short
> enough that there's no variation due to autovacuum, checkpoints etc.

Uh-huh.  I'm not surprised that it's repeatable on a given animal.
What remains to be explained:

1. Why'd it start failing now?  I'm guessing that ce95c5437 *was* the
culprit after all, by slightly changing the amount of catalog data
written during initdb, and thus moving the initial LSN.

2. Why just these two animals?  If initial LSN is the critical thing,
then the results of "locale -a" would affect it, so platform
dependence is hardly surprising ... but I'd have thought that all
the animals on that host would use the same initial set of
collations.  OTOH, I see petalura and pogona just fell over too.
Do you have some of those animals --with-icu and others not?

> 16bit checksums for the win.

Yay :-(

As for a fix, would damaging more of the page help?  I guess
it'd just move around the one-in-64K chance of failure.
Maybe we have to intentionally corrupt (e.g. invert) the
checksum field specifically.

			regards, tom lane