Subject: Re: backup manifests
To: Robert Haas <robertmhaas@gmail.com>, Noah Misch <noah@leadboat.com>
Cc: Stephen Frost <sfrost@snowman.net>, Amit Kapila
 <amit.kapila16@gmail.com>, Suraj Kharage <suraj.kharage@enterprisedb.com>,
 tushar <tushar.ahuja@enterprisedb.com>,
 Rajkumar Raghuwanshi <rajkumar.raghuwanshi@enterprisedb.com>,
 Rushabh Lathia <rushabh.lathia@gmail.com>,
 Tels <nospam-pg-abuse@bloodgate.com>,
 Andrew Dunstan <andrew.dunstan@2ndquadrant.com>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>,
 Jeevan Chalke <jeevan.chalke@enterprisedb.com>,
 vignesh C <vignesh21@gmail.com>
References: 
 <CAA4eK1+WyCVLZ03ZPH+POAY_NE80p+QxXDUnF10iryHj3vFjTA@mail.gmail.com>
 <CA+TgmoZBrwYvfQ=ymXk669+hgdQxpoypQVYdWiGwpOE_yuMW4A@mail.gmail.com>
 <20200323224217.GN13712@tamriel.snowman.net>
 <CA+TgmoYdFx_Hm4TUmc2qxLvaUo1KXH2yuxBk1RrgT5Oj+PwvNw@mail.gmail.com>
 <20200325133106.GS13712@tamriel.snowman.net>
 <CA+TgmoYOKC_8o-AR1jTQs0mOrFx=_Rcy5udor1m-LjyJNiSWPQ@mail.gmail.com>
 <20200325205433.GV13712@tamriel.snowman.net>
 <CA+TgmoZcEXq0SUveFRbqrso1EsDKCn8YXPSyWJBhC-c3fDbO=Q@mail.gmail.com>
 <16538d02-fd4b-6c4f-81a5-132c8fe8c3e9@pgmasters.net>
 <CA+TgmoZfj3wqcgoaXp+4a7ZSk3coARcxvghT7m=7Hf=8r6st6w@mail.gmail.com>
 <20200329034010.GB2247658@rfd.leadboat.com>
 <CA+TgmobSzurSGzy4uLXq__h3pKFnb4RFHepa6yxs3h4d3gwGEg@mail.gmail.com>
From: David Steele <david@pgmasters.net>
Message-ID: <fab407bb-04a1-b00c-e285-47fe3c103be2@pgmasters.net>
Date: Sun, 29 Mar 2020 20:54:41 -0400
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0)
 Gecko/20100101 Thunderbird/68.6.0
MIME-Version: 1.0
In-Reply-To: 
 <CA+TgmobSzurSGzy4uLXq__h3pKFnb4RFHepa6yxs3h4d3gwGEg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 3/29/20 8:42 PM, Robert Haas wrote:
> On Sat, Mar 28, 2020 at 11:40 PM Noah Misch <noah@leadboat.com> wrote:
>> I don't see this freed anywhere; is it?  (It's useful to make peak memory
>> consumption not grow in proportion to the number of files backed up.)
> 
> We need the hash table to remain populated for the whole run time of
> the tool, because we're essentially doing a full join of the actual
> directory contents against the manifest contents. That's a bit
> unfortunate but it doesn't seem simple to improve. I think the only
> people who are really going to suffer are people who have an enormous
> pile of empty or nearly-empty relations. People who have large
> databases for the normal reason - i.e. a reasonable number of tables
> that hold a lot of data - will have manifests of very manageable size.

+1

-- 
-David
david@pgmasters.net