From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Magnus Hagander <magnus@hagander.net>
To: Michael Banck <michael.banck@credativ.de>
Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>
Cc: David Steele <david@pgmasters.net>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: pgsql: Validate page level checksums in base backups
Date: Tue, 10 Apr 2018 23:24:35 +0200
Message-ID: <f23e92ec-118d-e6ea-0c81-876ad62a588a@2ndquadrant.com> (raw)
In-Reply-To: <CABUevEzZ55qgLVtMxW=hXhzjHV3oU-gChpeKtufyf_siL-rBWg@mail.gmail.com>
References: <20180404121930.GE20852@nighthawk.caipicrew.dd-dns.de>
<20180404134732.tsxx2osvooolxy24@alvherre.pgsql>
<CABUevEwrCF7TzAyD=4WPfTyzH+5xaFYf+-=ATW58XLi7b4dURA@mail.gmail.com>
<20180404182247.GG20852@nighthawk.caipicrew.dd-dns.de>
<CABUevExp8BOo3ZjEXM5xkz5P0zrjZT4_eM7QM4DLyPorDVUQUQ@mail.gmail.com>
<20180405124113.GH20852@nighthawk.caipicrew.dd-dns.de>
<CABUevEzZ55qgLVtMxW=hXhzjHV3oU-gChpeKtufyf_siL-rBWg@mail.gmail.com>
Hi,
I think there's a bug in sendFile(). We do check checksums on all pages
that pass this LSN check:
/*
* Only check pages which have not been modified since the
* start of the base backup. Otherwise, they might have been
* written only halfway and the checksum would not be valid.
* However, replaying WAL would reinstate the correct page in
* this case.
*/
if (PageGetLSN(page) < startptr)
{
...
}
Now, imagine the page is new, i.e. all-zeroes. That means the LSN is 0/0
too, and we'll try to verify the checksum - but we actually do not set
checksums on empty pages.
So I think it should be something like this:
if ((!PageIsNew(page)) && (PageGetLSN(page) < startptr))
{
...
}
It might be worth verifying that the page is actually all-zeroes (and
not just with corrupted pd_upper value. Not sure it's worth it.
I've found this by fairly trivial stress testing - running pgbench and
pg_basebackup in a loop. It was failing pretty reliably (~75% of runs).
With the proposed change I see no further failures.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@2ndquadrant.com, magnus@hagander.net, michael.banck@credativ.de, alvherre@alvh.no-ip.org, tgl@sss.pgh.pa.us, andres@anarazel.de, david@pgmasters.net
Subject: Re: pgsql: Validate page level checksums in base backups
In-Reply-To: <f23e92ec-118d-e6ea-0c81-876ad62a588a@2ndquadrant.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox