From: David Steele <david@pgmasters.net>
To: Andres Freund <andres@anarazel.de>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Postgres hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Add recovery to pg_control and remove backup_label
Date: Tue, 21 Nov 2023 14:48:59 -0400
Message-ID: <5cd0edb1-57eb-4814-8236-4f2ccc65f369@pgmasters.net> (raw)
In-Reply-To: <20231121175918.3s7k6iijg2zm6m4v@awork3.anarazel.de>
References: <a1d77886-97ac-4ec3-8f5d-7f0d95cbdc1e@pgmasters.net>
<ZVqzRSbi2feDhmf7@paquier.xyz>
<CA+TgmoZqyarS2ud8PXywh_+r6tsss4_D3=F+8PVSaa2MO+HPkA@mail.gmail.com>
<38b212b7-b8c0-46ea-940b-f27ec3a24deb@pgmasters.net>
<CA+TgmoYo1Gu5+0xy7-ivmjsdDfCPDPyyV+yL66XjMeOwTCEAFA@mail.gmail.com>
<c4adaf22-b3f4-4902-acd5-53ec07e88c3c@pgmasters.net>
<CA+Tgmoae7=XFBMfBEW7YOCASq7_sv4ohT9T8PyQnz70VVco42Q@mail.gmail.com>
<75071637-6b5c-483b-ac55-9c15b69eaca0@pgmasters.net>
<20231120204110.ofcmzdjbjlas4kuy@awork3.anarazel.de>
<62013cd7-b2a6-477e-b8dd-9a27be3073e0@pgmasters.net>
<20231121175918.3s7k6iijg2zm6m4v@awork3.anarazel.de>
On 11/21/23 13:59, Andres Freund wrote:
>
> On 2023-11-21 13:41:15 -0400, David Steele wrote:
>> On 11/20/23 16:41, Andres Freund wrote:
>>>
>>> On 2023-11-20 15:56:19 -0400, David Steele wrote:
>>>> I understand this is an option -- but does it need to be? What is the
>>>> benefit of excluding the manifest?
>>>
>>> It's not free to create the manifest, particularly if checksums are enabled.
>>
>> It's virtually free, even with the basic CRCs.
>
> Huh?
<snip>
> I'd not call 7.06->4.77 or 6.76->4.77 "virtually free".
OK, but how does that look with compression -- to a remote location?
Uncompressed backup to local storage doesn't seem very realistic. With
gzip compression we measure SHA1 checksums at about 5% of total CPU.
Obviously that goes up with zstd or lz4. but parallelism helps offset
that cost, at least in clock time.
I can't understate how valuable checksums are in finding corruption,
especially in long-lived backups.
>> Anyway, would you really want a backup without a manifest? How would you
>> know something is missing? In particular, for page incremental how do you
>> know something is new (but not WAL logged) if there is no manifest? Is the
>> plan to just recopy anything not WAL logged with each incremental?
>
> Shrug. If you just want to create a new standby by copying the primary, I
> don't think creating and then validating the manifest buys you much. Long term
> backups are a different story, particularly if data files are stored
> individually, rather than in a single checksummed file.
Fine, but you are probably not using page incremental if just using
pg_basebackup to create a standby. With page incremental, at least one
of the backups will already exist, which argues for a manifest.
>>> Also, for external backups, there's no manifest...
>>
>> There certainly is a manifest for many external backup solutions. Not having
>> a manifest is just running with scissors, backup-wise.
>
> You mean that you have an external solution gin up a backup manifest? I fail
> to see how that's relevant here?
Just saying that for external backups there *is* often a manifest and it
is a good thing to have.
Regards,
-David
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: david@pgmasters.net, andres@anarazel.de, robertmhaas@gmail.com, michael@paquier.xyz, pgsql-hackers@lists.postgresql.org
Subject: Re: Add recovery to pg_control and remove backup_label
In-Reply-To: <5cd0edb1-57eb-4814-8236-4f2ccc65f369@pgmasters.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox