From: David Steele <david@pgmasters.net>
To: Robert Haas <robertmhaas@gmail.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: backup manifests
Date: Fri, 20 Sep 2019 11:21:24 -0400
Message-ID: <2cec49b2-eeab-574f-2b5e-1f437b2cbec5@pgmasters.net> (raw)
In-Reply-To: <CA+TgmoY+hFkOiyTOar0AynbbO5NN87ScKcGH8c0ajb=+55n6og@mail.gmail.com>
References: <CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com>
<4b6244ea-4094-9055-080b-7afb274bf270@pgmasters.net>
<CA+TgmobANXUCZ1XrLJQbX95e3fJa0dRqV09Scjb2kmqt+gdkvg@mail.gmail.com>
<b2b696b4-b11f-e954-c86a-8252a62a2e40@pgmasters.net>
<CA+TgmobZrNYR-ATtfZiZ_k-W7tSPgvmYZmyiqumQig4R4fkzHw@mail.gmail.com>
<CA+TgmoY+hFkOiyTOar0AynbbO5NN87ScKcGH8c0ajb=+55n6og@mail.gmail.com>
On 9/20/19 10:59 AM, Robert Haas wrote:
> On Fri, Sep 20, 2019 at 9:46 AM Robert Haas <robertmhaas@gmail.com> wrote:
>> - appendStringInfo et. al. I don't think it would be that hard to move
>> this to src/common, but I'm also not sure it really solves the
>> problem, because StringInfo has a 1GB limit, and there's no rule at
>> all that a backup manifest has got to be less than 1GB.
>
> Hmm. That's actually going to be a problem on the server side, no
> matter what we do on the client side. We have to send the manifest
> after we send everything else, so that we know what we sent. But if we
> sent a lot of files, the manifest might be really huge. I had been
> thinking that we would generate the manifest on the server and send it
> to the client after everything else, but maybe this is an argument for
> generating the manifest on the client side and writing it
> incrementally. That would require the client to peek at the contents
> of every tar file it receives all the time, which it currently doesn't
> need to do, but it does peek inside them a little bit, so maybe it's
> OK.
>
> Another alternative would be to have the server spill the manifest in
> progress to a temp file and then stream it from there to the client.
This seems reasonable to me.
We keep an in-memory representation which is just an array of structs
and is fairly compact -- 1 million files uses ~150MB of memory. We just
format and stream this to storage when saving. Saving is easier than
loading, of course.
--
-David
david@pgmasters.net
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: david@pgmasters.net, robertmhaas@gmail.com
Subject: Re: backup manifests
In-Reply-To: <2cec49b2-eeab-574f-2b5e-1f437b2cbec5@pgmasters.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox