Date: Sun, 29 Mar 2020 22:58:54 -0700
From: Noah Misch <noah@leadboat.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: David Steele <david@pgmasters.net>, Stephen Frost <sfrost@snowman.net>,
	Amit Kapila <amit.kapila16@gmail.com>,
	Suraj Kharage <suraj.kharage@enterprisedb.com>,
	tushar <tushar.ahuja@enterprisedb.com>,
	Rajkumar Raghuwanshi <rajkumar.raghuwanshi@enterprisedb.com>,
	Rushabh Lathia <rushabh.lathia@gmail.com>,
	Tels <nospam-pg-abuse@bloodgate.com>,
	Andrew Dunstan <andrew.dunstan@2ndquadrant.com>,
	PostgreSQL Hackers <pgsql-hackers@postgresql.org>,
	Jeevan Chalke <jeevan.chalke@enterprisedb.com>,
	vignesh C <vignesh21@gmail.com>
Subject: Re: backup manifests
Message-ID: <20200330055854.GC2324620@rfd.leadboat.com>
References: <20200323224217.GN13712@tamriel.snowman.net>
 <CA+TgmoYdFx_Hm4TUmc2qxLvaUo1KXH2yuxBk1RrgT5Oj+PwvNw@mail.gmail.com>
 <20200325133106.GS13712@tamriel.snowman.net>
 <CA+TgmoYOKC_8o-AR1jTQs0mOrFx=_Rcy5udor1m-LjyJNiSWPQ@mail.gmail.com>
 <20200325205433.GV13712@tamriel.snowman.net>
 <CA+TgmoZcEXq0SUveFRbqrso1EsDKCn8YXPSyWJBhC-c3fDbO=Q@mail.gmail.com>
 <16538d02-fd4b-6c4f-81a5-132c8fe8c3e9@pgmasters.net>
 <CA+TgmoZfj3wqcgoaXp+4a7ZSk3coARcxvghT7m=7Hf=8r6st6w@mail.gmail.com>
 <20200329034010.GB2247658@rfd.leadboat.com>
 <CA+TgmobSzurSGzy4uLXq__h3pKFnb4RFHepa6yxs3h4d3gwGEg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CA+TgmobSzurSGzy4uLXq__h3pKFnb4RFHepa6yxs3h4d3gwGEg@mail.gmail.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk

On Sun, Mar 29, 2020 at 08:42:35PM -0400, Robert Haas wrote:
> On Sat, Mar 28, 2020 at 11:40 PM Noah Misch <noah@leadboat.com> wrote:
> > I think this functionality doesn't belong in its own program.  If you suspect
> > pg_basebackup or pg_restore will eventually gain the ability to merge
> > incremental backups into a recovery-ready base backup, I would put the
> > functionality in that program.  Otherwise, I would put it in pg_checksums.
> > For me, part of the friction here is that the program description indicates
> > general verification, but the actual functionality merely checks hashes on a
> > directory tree that happens to represent a PostgreSQL base backup.
> 
> Suraj's original patch made this part of pg_basebackup, but I didn't
> really like that, because I wanted it to have its own set of options.
> I still think all the options I've added are pretty useful ones, and I
> can think of other things somebody might want to do. It feels very
> uncomfortable to make pg_basebackup, or pg_checksums, take either
> options from set A and do thing X, or options from set B and do thing
> Y.

pg_checksums does already have that property, for what it's worth.  (More
specifically, certain options dictate the mode, and it reports an error if
another option is incompatible with the mode.)

> But it feels clear that the name pg_validatebackup is not going
> over very well with anyone. I think I should rename it to
> pg_validatemanifest.

Between those two, I would use "pg_validatebackup" if there's a fair chance it
will end up doing the pg_waldump check.  Otherwise, I would use
"pg_validatemanifest".  I still most prefer delivering this as a mode of an
existing program.

> > > +             parse->pathname = palloc(raw_length + 1);
> >
> > I don't see this freed anywhere; is it?  (It's useful to make peak memory
> > consumption not grow in proportion to the number of files backed up.)
> 
> We need the hash table to remain populated for the whole run time of
> the tool, because we're essentially doing a full join of the actual
> directory contents against the manifest contents. That's a bit
> unfortunate but it doesn't seem simple to improve. I think the only
> people who are really going to suffer are people who have an enormous
> pile of empty or nearly-empty relations. People who have large
> databases for the normal reason - i.e. a reasonable number of tables
> that hold a lot of data - will have manifests of very manageable size.

Okay.