Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps  (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <pgsql-hackers-owner+archive@lists.postgresql.org>)
	id 1rxEPU-007AVS-1e
	for pgsql-hackers@arkaria.postgresql.org; Wed, 17 Apr 2024 23:09:48 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.94.2)
	(envelope-from <pgsql-hackers-owner+archive@lists.postgresql.org>)
	id 1rxEPS-00Fs9w-Ej
	for pgsql-hackers@arkaria.postgresql.org; Wed, 17 Apr 2024 23:09:46 +0000
Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29])
	by malur.postgresql.org with esmtps  (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <david@pgmasters.net>)
	id 1rxEPS-00Fs9o-5B
	for pgsql-hackers@lists.postgresql.org; Wed, 17 Apr 2024 23:09:46 +0000
Received: from mail.thelabyrinth.net ([45.56.70.56])
	by magus.postgresql.org with esmtps  (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <david@pgmasters.net>)
	id 1rxEPP-001X2f-Ld
	for pgsql-hackers@postgresql.org; Wed, 17 Apr 2024 23:09:45 +0000
Received: from [10.5.0.2] (unknown [45.134.224.55])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: dsteele)
	by mail.thelabyrinth.net (Postfix) with ESMTPSA id C58B25466D;
	Wed, 17 Apr 2024 23:09:40 +0000 (UTC)
Message-ID: <e6f930c3-590c-47b9-b094-217bb2a3e22e@pgmasters.net>
Date: Thu, 18 Apr 2024 09:09:37 +1000
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: pg_combinebackup does not detect missing files
Content-Language: en-US
To: Robert Haas <robertmhaas@gmail.com>
Cc: Pg Hackers <pgsql-hackers@postgresql.org>
References: <9badd24d-5bd9-4c35-ba85-4c38a2feb73e@pgmasters.net>
 <CA+TgmoaVxr_o3mrDBrBcXm3gowr9Qc4ABW-c73NR_201KkDavw@mail.gmail.com>
 <f1a4b02e-f6cd-412d-8ea7-9f3fb3fdcc8b@pgmasters.net>
 <CA+TgmoZmuhkjGx=97kEniMRNC9Qaifjh0zMFZnHne72JGFNdKA@mail.gmail.com>
From: David Steele <david@pgmasters.net>
In-Reply-To: <CA+TgmoZmuhkjGx=97kEniMRNC9Qaifjh0zMFZnHne72JGFNdKA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
List-Id: <pgsql-hackers.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-hackers@lists.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-hackers>
Archived-At: <https://www.postgresql.org/message-id/e6f930c3-590c-47b9-b094-217bb2a3e22e%40pgmasters.net>
Precedence: bulk

On 4/18/24 01:03, Robert Haas wrote:
> On Tue, Apr 16, 2024 at 7:25 PM David Steele <david@pgmasters.net> wrote:
> 
> But it will not go out of its way to perform checks that are unrelated
> to its documented purpose. And I don't think it should, especially if
> we have another tool that already does that.
> 
>>> I'm not averse to having some kind of statement in the documentation
>>> along the lines of "Note that pg_combinebackup does not attempt to
>>> verify that the individual backups are intact; for that, use
>>> pg_verifybackup."
>>
>> I think we should do this at a minimum.
> 
> Here is a patch to do that.

I think here:

+   <application>pg_basebackup</application> only attempts to verify

you mean:

+   <application>pg_combinebackup</application> only attempts to verify

Otherwise this looks good to me.

>> Especially given how pg_combinebackup works, backups are going to
>> undergo a lot of user manipulation (pushing to and pull from storage,
>> decompressing, untaring, etc.) and I think that means we should take
>> extra care.
> 
> We are in agreement on that point, if nothing else. I am terrified of
> users having problems with pg_combinebackup and me not being able to
> tell whether those problems are due to user error, Robert error, or
> something else. I put a lot of effort into detecting dumb things that
> I thought a user might do, and a lot of effort into debugging output
> so that when things do go wrong anyway, we have a reasonable chance of
> figuring out exactly where they went wrong. We do seem to have a
> philosophical difference about what the scope of those checks ought to
> be, and I don't really expect what I wrote above to convince you that
> my position is correct, but perhaps it will convince you that I have a
> thoughtful position, as opposed to just having done stuff at random.

Fair enough. I accept that your reasoning is not random, but I'm still 
not very satisfied that the user needs to run a separate and rather 
expensive process to do the verification when pg_combinebackup already 
has the necessary information at hand. My guess is that most users will 
elect to skip verification.

At least now they'll have the information they need to make an informed 
choice.

Regards,
-David