Message-ID: <477f38d6-df05-60a2-2266-e54a4f232ac6@pgmasters.net>
Date: Thu, 9 Jun 2022 18:34:11 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.10.0
Subject: Re: pg_basebackup and pg_receivewal timing, missing WAL files
Content-Language: en-US
To: Stephen Frost <sfrost@snowman.net>, Tim <timfosho@gmail.com>
Cc: posgres support <pgsql-admin@postgresql.org>
References: 
 <CAKhLO5itE_xp7KX7Vxnho8Uzczgwt8iUD9bVtKNPamBOst=hPA@mail.gmail.com>
 <20220609183641.GN9030@tamriel.snowman.net>
 <CAKhLO5gY4rUS65RUD_OGyHQTxoOa1-akLBWjh-CDQiVDJknecw@mail.gmail.com>
 <20220609212409.GR9030@tamriel.snowman.net>
From: David Steele <david@pgmasters.net>
In-Reply-To: <20220609212409.GR9030@tamriel.snowman.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/477f38d6-df05-60a2-2266-e54a4f232ac6%40pgmasters.net>
Precedence: bulk

On 6/9/22 5:24 PM, Stephen Frost wrote:
> 
> * Tim (timfosho@gmail.com) wrote:
>> We currently use pgbackrest as our primary backup tool actually, and it
>> works great! The issue I ran into with trying to set it up that way with
>> pgbackrest, is when configuring the second repo, pgbackrest will have to
>> archive all WAL files to it, going back to the oldest backup. And we have
> 
> Eh?  Why is that?  I think there's some confusion here regarding how
> pgbackrest multi-repo works.  We won't archive back to the 'oldest
> backup' when you add a second repo, we'll just archive *new* WAL to it
> after it's been configured.

By default pgbackrest will keep all WAL since the start of the oldest 
backup in a repo. Each repo can have its own retention settings so this 
might be a *different* oldest WAL per repo.

However, you can use repo-retention-archive and 
repo-retention-archive-type to change this behavior, see 
https://pgbackrest.org/configuration.html#section-repository/option-repo-retention-archive.

Basically, if you need to keep a number of full backups for compliance 
you can set repo-retention-archive-type=full and 
repo-retention-archive-type=1 in that repo and WAL in between the 
backups will only be kept after the last backup. WAL required to make 
backups consistent is always kept until the backup expires.

We've had requests for repo-retention-archive-type=0, i.e. only archive 
WAL during a backup, but we have not had time to implement that. Some 
users have implemented a work-around based on checking the backup lock. 
There is an issue that describes the method but I was not able to find 
it after a brief search. This is safer than it sounds because pgbackrest 
will check that all WAL required to make the backup consistent reached 
the archive. So, if the wrapper is broken you'll get errors.

Regards,
-- 
-David
david@pgmasters.net