Date: Thu, 2 Apr 2020 11:23:46 -0700
From: Andres Freund <andres@anarazel.de>
To: Robert Haas <robertmhaas@gmail.com>
Cc: David Steele <david@pgmasters.net>, Noah Misch <noah@leadboat.com>,
	Stephen Frost <sfrost@snowman.net>,
	Amit Kapila <amit.kapila16@gmail.com>,
	Suraj Kharage <suraj.kharage@enterprisedb.com>,
	tushar <tushar.ahuja@enterprisedb.com>,
	Rajkumar Raghuwanshi <rajkumar.raghuwanshi@enterprisedb.com>,
	Rushabh Lathia <rushabh.lathia@gmail.com>,
	Tels <nospam-pg-abuse@bloodgate.com>,
	Andrew Dunstan <andrew.dunstan@2ndquadrant.com>,
	PostgreSQL Hackers <pgsql-hackers@postgresql.org>,
	Jeevan Chalke <jeevan.chalke@enterprisedb.com>,
	vignesh C <vignesh21@gmail.com>
Subject: Re: backup manifests
Message-ID: <20200402182346.6iffoadxu2hsbi2s@alap3.anarazel.de>
References: <ddbb128b-86e5-7ffb-b5dc-15a414dfde0e@pgmasters.net>
 <20200330020814.nspra4mvby42yoa4@alap3.anarazel.de>
 <CA+TgmoZj9tuVRDw=wj1p2-TTSJ+qUEWvh6updZJsw-5gxWUW1w@mail.gmail.com>
 <20200330185944.42bxysvem6b757ew@alap3.anarazel.de>
 <CA+TgmoawEeE5qpFgj5Vy2zZGKzd3ZSEhGrD_JdPqPd2GB8u1Cw@mail.gmail.com>
 <20200331225034.odt3l2w4h3dr6ggk@alap3.anarazel.de>
 <CA+TgmobV+kW5cBCa8_LGvytdOH71gnpqDsmKBR7bghEFgR7Y1w@mail.gmail.com>
 <CA+Tgmob+xDcvEUTznkvianyJzKK8nUM6bWfkrnZQkL-XEa3NEA@mail.gmail.com>
 <20200402172318.3kvfrrewxmemzpia@alap3.anarazel.de>
 <CA+Tgmoa2_8Ti9xez8wEH7Z_RJQv=cwPC=MeqUXcdVW=5-uMCig@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CA+Tgmoa2_8Ti9xez8wEH7Z_RJQv=cwPC=MeqUXcdVW=5-uMCig@mail.gmail.com>
Precedence: bulk

Hi,

On 2020-04-02 14:16:27 -0400, Robert Haas wrote:
> On Thu, Apr 2, 2020 at 1:23 PM Andres Freund <andres@anarazel.de> wrote:
> > I suspect its possible to control the timing by preventing the
> > checkpoint at the end of recovery from completing within a relevant
> > timeframe. I think configuring a large checkpoint_timeout and using a
> > non-fast base backup ought to do the trick. The state can be advanced by
> > separately triggering an immediate checkpoint? Or by changing the
> > checkpoint_timeout?
> 
> That might make the window fairly wide on normal systems, but I'm not
> sure about Raspberry Pi BF members or things running
> CLOBBER_CACHE_ALWAYS/RECURSIVELY. I guess I could try it.

You can set checkpoint_timeout to be a day. If that's not enough, well,
then I think we have other problems.


> > FWIW, the only check I'd really like to see in this release is the
> > crosscheck with the files length and the actually read data (to be able
> > to disagnose FS issues).
> 
> Not sure I understand this comment. Isn't that a subset of what the
> patch already does? Are you asking for something to be changed?

Yes, I am asking for something to be changed: I'd like the code that
read()s the file when computing the checksum to add up how many bytes
were read, and compare that to the size in the manifest. And if there's
a difference report an error about that, instead of a checksum failure.

I've repeatedly seen filesystem issues lead to to earlier EOFs when
read()ing than what stat() returns. It'll be pretty annoying to have to
debug a general "checksum failure", rather than just knowing that
reading stopped after 100MB of 1GB.

Greetings,

Andres Freund