Date: Sat, 5 Jan 2019 17:12:14 -0500
From: Stephen Frost <sfrost@snowman.net>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Magnus Hagander <magnus@hagander.net>,
	Michael Paquier <michael@paquier.xyz>,
	Michael Banck <michael.banck@credativ.de>,
	Postgres hackers <pgsql-hackers@postgresql.org>
Subject: Re: Offline enabling/disabling of data checksums
Message-ID: <20190105221214.GW2528@tamriel.snowman.net>
References: <20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de>
 <20181221232834.GB1948@paquier.xyz>
 <CABUevEyUaB=Ws=DW-XsBw4ibDXZhQZ8d3WR4485bkmm13kTGVw@mail.gmail.com>
 <348aef85-a967-80d2-7bef-7274909475cf@2ndquadrant.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="ruCS5aQMIBlP3nv7"
Content-Disposition: inline
In-Reply-To: <348aef85-a967-80d2-7bef-7274909475cf@2ndquadrant.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk


--ruCS5aQMIBlP3nv7
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Tomas Vondra (tomas.vondra@2ndquadrant.com) wrote:
> On 12/27/18 11:43 AM, Magnus Hagander wrote:
> > Plus, the majority of people *should* want them on :) We don't run with
> > say synchronous_commit=3Doff by default either to make it easier on tho=
se
> > that don't want to pay the overhead of full data safety :P (I know it's
> > not a direct match, but you get the idea)

+1 to having them on by default, we should have done that a long time
ago.

> I don't know, TBH. I agree making the on/off change cheaper makes moves
> us closer to 'on' by default, because they may disable it if needed. But
> it's not the whole story.
>=20
> If we enable checksums by default, 99% users will have them enabled.

Yes, and they'll then be able to catch data corruption much earlier.
Today, 99% of our users don't have them enabled and have no clue if
their data has been corrupted on disk, or not.  That's not good.

> That means more people will actually observe data corruption cases that
> went unnoticed so far. What shall we do with that? We don't have very
> good answers to that (tooling, docs) and I'd say "disable checksums" is
> not a particularly amazing response in this case :-(

Now that we've got a number of tools available which will check the
checksums in a running system and throw up warnings when found
(pg_basebackup, pgBackRest and I think other backup tools,
pg_checksums...), users will see corruption and have the option to
restore from a backup before those backups expire out and they're left
with a corrupt database and backups which also have that corruption.

This ongoing call for specific tooling to do "something" about checksums
is certainly good, but it's not right to say that we don't have existing
documentation- we do, quite a bit of it, and it's all under the heading
of "Backup and Recovery".

> FWIW I don't know what to do about that. We certainly can't prevent the
> data corruption, but maybe we could help with fixing it (although that's
> bound to be low-level work).

There's been some effort to try and automagically correct corrupted
pages but it's certainly not something I'm ready to trust beyond a
"well, this is what it might have been" review.  The answer today is to
find a backup which isn't corrupt and restore from it on a known-good
system.  If adding explicit documentation to that effect would reduce
your level of concern when it comes to enabling checksums by default,
then I'm happy to do that.

Thanks!

Stephen

--ruCS5aQMIBlP3nv7
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJcMSu+AAoJEO1sijiDR2RVXMEP/1mYn//2b8ik4jj5+maQ7KSG
IB+HGDP9RSNcigTJJdtQrOtB3g3YlofNyHdWJ0dylzLLR7NCxXId9xWW21g/R/49
h8tXBMcr/4vJ7XnWrc34qk6h4kkV6uIBHXgZ6db3xUfSFgfkmmMwGC3hhqi1GsFh
s54UUmpQktJplDQ1knTkcywYI5uowd5aPUdx3QIAK3hC2ethRSGymeuyNbBobuXe
xH0NRi1BrQRuhju8KJ1p19mJp8CRyqpzCW4ktlf5TuGK0WmcfP6j7xWDE+HF52NU
iHPBtc70/CFpHvV8DdhMq6kQYotSyOieY6CYBPGcuJhEvkRsQrk+rGBkelGrP0to
w1gcUXlD2VEVkVbK+0W5UYKK1JHm+oJjDxVm+/40Am7PKAdBK+o1FRITr2r6n7Jv
unUKLDFJ5L5GAq7obtibM/4jDOeLY5Op+ul4coR8ucpVYi4gio6Wk4SjKlWZHGS5
J79C13ZKLverXOJy/1j8j+49lBLJi9KmDyMG20Cv/Udx8m7XRtCQxGblmGhBaJeN
OPTDThI8+JSb4R/d6dj/E6zfyeNosdohqUQ4gka3RvnFsbKPGTBItslJRtHJHSs9
Rralu14Fyx8TeLJmxSn5XIWlrYHcziw+ywTuFHibS1etStbXoFaNlpL3OqslnLns
D+33d70Ld9OwAQnoOph1
=DlbW
-----END PGP SIGNATURE-----

--ruCS5aQMIBlP3nv7--