Date: Tue, 29 Mar 2022 12:34:16 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Andres Freund <andres@anarazel.de>,
	Daniel Gustafsson <daniel@yesql.se>,
	Kyotaro Horiguchi <horikyota.ntt@gmail.com>, deniel1495@mail.ru,
	Ibrar Ahmed <ibrar.ahmad@gmail.com>, tejeswarm@hotmail.com,
	hlinnaka <hlinnaka@iki.fi>,
	Masahiko Sawada <masahiko.sawada@2ndquadrant.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>,
	Daniel Wood <hexexpert@comcast.net>
Subject: Re: Corruption during WAL replay
Message-ID: <20220329163416.GO10577@tamriel.snowman.net>
References: <20220325045438.enwakjqhrafzq5f2@alap3.anarazel.de>
 <20220325052654.3xpbmntatyofau2w@alap3.anarazel.de>
 <3193652.1648186725@sss.pgh.pa.us>
 <20220325060737.iayq5cs36jktqlag@alap3.anarazel.de>
 <3237467.1648216174@sss.pgh.pa.us>
 <CA+Tgmoa=1a+A_gK4gqBf7ypK9NyY5bo--Wqc7Wp3xm8fjsrUOQ@mail.gmail.com>
 <3238876.1648216924@sss.pgh.pa.us>
 <CA+TgmoaH=6MJtcqN44geA6uQqt1PCHg-2UvehScAxSmsUm4KVA@mail.gmail.com>
 <3241547.1648218889@sss.pgh.pa.us>
 <CA+TgmobXjkeptO-rNiOrq8zjUK2QusXKGgsRL-fqPCL0umc+3Q@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="n04fa9xHhxzeAdnV"
Content-Disposition: inline
In-Reply-To: 
 <CA+TgmobXjkeptO-rNiOrq8zjUK2QusXKGgsRL-fqPCL0umc+3Q@mail.gmail.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Archived-At: 
 <https://www.postgresql.org/message-id/20220329163416.GO10577%40tamriel.snowman.net>
Precedence: bulk


--n04fa9xHhxzeAdnV
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Robert Haas (robertmhaas@gmail.com) wrote:
> On Fri, Mar 25, 2022 at 10:34 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > I dunno.  Compatibility and speed concerns aside, that seems like an aw=
ful
> > lot of bits to be expending on every page compared to the value.
>=20
> I dunno either, but over on the TDE thread people seemed quite willing
> to expend like 16-32 *bytes* for page verifiers and nonces and things.

Absolutely.

> For compatibility and speed reasons, I doubt we could ever get by with
> doing that in every cluster, but I do have some hope of introducing
> something like that someday at least as an optional feature. It's not
> like a 16-bit checksum was state-of-the-art even when we introduced
> it. We just did it because we had 2 bytes that we could repurpose
> relatively painlessly, and not any larger number. And that's still the
> case today, so at least in the short term we will have to choose some
> other solution to this problem.

I agree that this would be great as an optional feature.  Those patches
to allow the system to be built with reserved space for $whatever would
allow us to have a larger checksum for those who want it and perhaps a
nonce with TDE for those who wish that in the future.  I mentioned
before that I thought it might be a good way to introduce page-level
epochs for 64bit xids too though it never seemed to get much traction.

Anyhow, this whole thread has struck me as a good reason to polish those
patches off and add on top of them an extended checksum ability, first,
independent of TDE, and remove the dependency of those patches from the
TDE effort and instead allow it to just leverage that ability.  I still
suspect we'll have some folks who will want TDE w/o a per-page nonce and
that could be an option but we'd be able to support TDE w/ integrity
pretty easily, which would be fantastic.

Thanks,

Stephen

--n04fa9xHhxzeAdnV
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJiQzUIAAoJEO1sijiDR2RVtI0P/jUOuOG6yQUYBMTpPCMqv1ry
TVou560g9ivTpdjzuAc8BFCutYZFDv0oqixmf0i/S6YPOiRKr1C3uB2R6T2Ay8kN
Oa3vuA/wbS5B2ecj095AgemyLSGHXOQVtp+j0zPyTebT3g06TF9gJdexKg1GrMHw
jj/TNqT5WoegOltUeppfpgbqZJjoVFOHmWXKQN5b2lfrpxEdxLR8vPTMb+04Rat/
djZu+//j1+2aWRc9+lNw0Lxwc/OwJiG0lQ5LchTFjOjmkDnq+vgIKW3tq3xpla+G
cEW5dbq1HcYs0JzluEoj+CZHfrpOW2LBb2CWujWNBLaEFHHU01n+ScQj70GC82Hb
BeWi441Lt1PCynco32/DbYelVeP/XpTyC4zZoZFKm2DyNCuET0FHRP4uPUNt1xZP
6snLkR7KGwfaelLKV18boCU6byXpCfyjrEIkFAUb72/Sv2erED68bW/cieNKD5Hi
Ml7gQlMaG3eL+k0Yj6YlzemgEygBdWKY6kSD0wLv17K37bk1EfK6rnm+YFS32M+V
Y8zEaH8TzoF95hsIqKqVR2kDvZI/B/4NxCa+8gaE1FihPtqQkKb3WQZ15GHK8wl+
JxPt3+krp3xUZhKS42caEz5zyiA+VxV2qH77LZjiYTjbZzlaBv/lE4shDUiiEHgm
TTnSccSBClD4taPG+xIJ
=s/Bf
-----END PGP SIGNATURE-----

--n04fa9xHhxzeAdnV--