Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps  (TLS1.3) tls
 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <pgsql-hackers-owner+archive@lists.postgresql.org>)
	id 1r5Vo9-00Dmjw-6t
	for pgsql-hackers@arkaria.postgresql.org; Tue, 21 Nov 2023 18:49:13 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.94.2)
	(envelope-from <pgsql-hackers-owner+archive@lists.postgresql.org>)
	id 1r5Vo7-001fbF-IW
	for pgsql-hackers@arkaria.postgresql.org; Tue, 21 Nov 2023 18:49:11 +0000
Received: from makus.postgresql.org ([2001:4800:3e1:1::229])
	by malur.postgresql.org with esmtps  (TLS1.3) tls
 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <david@pgmasters.net>)
	id 1r5Vo7-001fb7-8v
	for pgsql-hackers@lists.postgresql.org; Tue, 21 Nov 2023 18:49:11 +0000
Received: from mail.thelabyrinth.net ([45.56.70.56])
	by makus.postgresql.org with esmtps  (TLS1.2) tls
 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <david@pgmasters.net>)
	id 1r5Vo4-006gUg-Pl
	for pgsql-hackers@lists.postgresql.org; Tue, 21 Nov 2023 18:49:10 +0000
Received: from [10.5.0.2] (unknown [45.14.195.101])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: dsteele)
	by mail.thelabyrinth.net (Postfix) with ESMTPSA id E31E754625;
	Tue, 21 Nov 2023 18:49:06 +0000 (UTC)
Message-ID: <5cd0edb1-57eb-4814-8236-4f2ccc65f369@pgmasters.net>
Date: Tue, 21 Nov 2023 14:48:59 -0400
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Add recovery to pg_control and remove backup_label
Content-Language: en-US
To: Andres Freund <andres@anarazel.de>
Cc: Robert Haas <robertmhaas@gmail.com>, Michael Paquier
 <michael@paquier.xyz>, Postgres hackers <pgsql-hackers@lists.postgresql.org>
References: <a1d77886-97ac-4ec3-8f5d-7f0d95cbdc1e@pgmasters.net>
 <ZVqzRSbi2feDhmf7@paquier.xyz>
 <CA+TgmoZqyarS2ud8PXywh_+r6tsss4_D3=F+8PVSaa2MO+HPkA@mail.gmail.com>
 <38b212b7-b8c0-46ea-940b-f27ec3a24deb@pgmasters.net>
 <CA+TgmoYo1Gu5+0xy7-ivmjsdDfCPDPyyV+yL66XjMeOwTCEAFA@mail.gmail.com>
 <c4adaf22-b3f4-4902-acd5-53ec07e88c3c@pgmasters.net>
 <CA+Tgmoae7=XFBMfBEW7YOCASq7_sv4ohT9T8PyQnz70VVco42Q@mail.gmail.com>
 <75071637-6b5c-483b-ac55-9c15b69eaca0@pgmasters.net>
 <20231120204110.ofcmzdjbjlas4kuy@awork3.anarazel.de>
 <62013cd7-b2a6-477e-b8dd-9a27be3073e0@pgmasters.net>
 <20231121175918.3s7k6iijg2zm6m4v@awork3.anarazel.de>
From: David Steele <david@pgmasters.net>
In-Reply-To: <20231121175918.3s7k6iijg2zm6m4v@awork3.anarazel.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
List-Id: <pgsql-hackers.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-hackers@lists.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-hackers>
Archived-At: 
 <https://www.postgresql.org/message-id/5cd0edb1-57eb-4814-8236-4f2ccc65f369%40pgmasters.net>
Precedence: bulk

On 11/21/23 13:59, Andres Freund wrote:
> 
> On 2023-11-21 13:41:15 -0400, David Steele wrote:
>> On 11/20/23 16:41, Andres Freund wrote:
>>>
>>> On 2023-11-20 15:56:19 -0400, David Steele wrote:
>>>> I understand this is an option -- but does it need to be? What is the
>>>> benefit of excluding the manifest?
>>>
>>> It's not free to create the manifest, particularly if checksums are enabled.
>>
>> It's virtually free, even with the basic CRCs.
> 
> Huh?

<snip>

> I'd not call 7.06->4.77 or 6.76->4.77 "virtually free".

OK, but how does that look with compression -- to a remote location? 
Uncompressed backup to local storage doesn't seem very realistic. With 
gzip compression we measure SHA1 checksums at about 5% of total CPU. 
Obviously that goes up with zstd or lz4. but parallelism helps offset 
that cost, at least in clock time.

I can't understate how valuable checksums are in finding corruption, 
especially in long-lived backups.

>> Anyway, would you really want a backup without a manifest? How would you
>> know something is missing? In particular, for page incremental how do you
>> know something is new (but not WAL logged) if there is no manifest? Is the
>> plan to just recopy anything not WAL logged with each incremental?
> 
> Shrug. If you just want to create a new standby by copying the primary, I
> don't think creating and then validating the manifest buys you much. Long term
> backups are a different story, particularly if data files are stored
> individually, rather than in a single checksummed file.

Fine, but you are probably not using page incremental if just using 
pg_basebackup to create a standby. With page incremental, at least one 
of the backups will already exist, which argues for a manifest.

>>> Also, for external backups, there's no manifest...
>>
>> There certainly is a manifest for many external backup solutions. Not having
>> a manifest is just running with scissors, backup-wise.
> 
> You mean that you have an external solution gin up a backup manifest? I fail
> to see how that's relevant here?

Just saying that for external backups there *is* often a manifest and it 
is a good thing to have.

Regards,
-David