From: Andres Freund <andres@anarazel.de>
To: Robert Haas <robertmhaas@gmail.com>
Cc: David Steele <david@pgmasters.net>
Cc: Noah Misch <noah@leadboat.com>
Cc: Stephen Frost <sfrost@snowman.net>
Cc: Amit Kapila <amit.kapila16@gmail.com>
Cc: Suraj Kharage <suraj.kharage@enterprisedb.com>
Cc: tushar <tushar.ahuja@enterprisedb.com>
Cc: Rajkumar Raghuwanshi <rajkumar.raghuwanshi@enterprisedb.com>
Cc: Rushabh Lathia <rushabh.lathia@gmail.com>
Cc: Tels <nospam-pg-abuse@bloodgate.com>
Cc: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
Cc: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Cc: Jeevan Chalke <jeevan.chalke@enterprisedb.com>
Cc: vignesh C <vignesh21@gmail.com>
Subject: Re: backup manifests
Date: Thu, 2 Apr 2020 11:23:46 -0700
Message-ID: <20200402182346.6iffoadxu2hsbi2s@alap3.anarazel.de> (raw)
In-Reply-To: <CA+Tgmoa2_8Ti9xez8wEH7Z_RJQv=cwPC=MeqUXcdVW=5-uMCig@mail.gmail.com>
References: <ddbb128b-86e5-7ffb-b5dc-15a414dfde0e@pgmasters.net>
<20200330020814.nspra4mvby42yoa4@alap3.anarazel.de>
<CA+TgmoZj9tuVRDw=wj1p2-TTSJ+qUEWvh6updZJsw-5gxWUW1w@mail.gmail.com>
<20200330185944.42bxysvem6b757ew@alap3.anarazel.de>
<CA+TgmoawEeE5qpFgj5Vy2zZGKzd3ZSEhGrD_JdPqPd2GB8u1Cw@mail.gmail.com>
<20200331225034.odt3l2w4h3dr6ggk@alap3.anarazel.de>
<CA+TgmobV+kW5cBCa8_LGvytdOH71gnpqDsmKBR7bghEFgR7Y1w@mail.gmail.com>
<CA+Tgmob+xDcvEUTznkvianyJzKK8nUM6bWfkrnZQkL-XEa3NEA@mail.gmail.com>
<20200402172318.3kvfrrewxmemzpia@alap3.anarazel.de>
<CA+Tgmoa2_8Ti9xez8wEH7Z_RJQv=cwPC=MeqUXcdVW=5-uMCig@mail.gmail.com>
Hi,
On 2020-04-02 14:16:27 -0400, Robert Haas wrote:
> On Thu, Apr 2, 2020 at 1:23 PM Andres Freund <andres@anarazel.de> wrote:
> > I suspect its possible to control the timing by preventing the
> > checkpoint at the end of recovery from completing within a relevant
> > timeframe. I think configuring a large checkpoint_timeout and using a
> > non-fast base backup ought to do the trick. The state can be advanced by
> > separately triggering an immediate checkpoint? Or by changing the
> > checkpoint_timeout?
>
> That might make the window fairly wide on normal systems, but I'm not
> sure about Raspberry Pi BF members or things running
> CLOBBER_CACHE_ALWAYS/RECURSIVELY. I guess I could try it.
You can set checkpoint_timeout to be a day. If that's not enough, well,
then I think we have other problems.
> > FWIW, the only check I'd really like to see in this release is the
> > crosscheck with the files length and the actually read data (to be able
> > to disagnose FS issues).
>
> Not sure I understand this comment. Isn't that a subset of what the
> patch already does? Are you asking for something to be changed?
Yes, I am asking for something to be changed: I'd like the code that
read()s the file when computing the checksum to add up how many bytes
were read, and compare that to the size in the manifest. And if there's
a difference report an error about that, instead of a checksum failure.
I've repeatedly seen filesystem issues lead to to earlier EOFs when
read()ing than what stat() returns. It'll be pretty annoying to have to
debug a general "checksum failure", rather than just knowing that
reading stopped after 100MB of 1GB.
Greetings,
Andres Freund
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: andres@anarazel.de, robertmhaas@gmail.com, david@pgmasters.net, noah@leadboat.com, sfrost@snowman.net, amit.kapila16@gmail.com, suraj.kharage@enterprisedb.com, tushar.ahuja@enterprisedb.com, rajkumar.raghuwanshi@enterprisedb.com, rushabh.lathia@gmail.com, nospam-pg-abuse@bloodgate.com, andrew.dunstan@2ndquadrant.com, jeevan.chalke@enterprisedb.com, vignesh21@gmail.com
Subject: Re: backup manifests
In-Reply-To: <20200402182346.6iffoadxu2hsbi2s@alap3.anarazel.de>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox