Message-ID: <dee5201a-cf77-ea09-ec9b-0d22afb91039@pgmasters.net>
Date: Wed, 3 Aug 2022 08:08:59 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.11.0
Subject: Re: pgbackrest archive.info got deleted unexpected
Content-Language: en-US
To: "James Pang (chaolpan)" <chaolpan@cisco.com>,
 "pgsql-admin@lists.postgresql.org" <pgsql-admin@lists.postgresql.org>
References: 
 <PH0PR11MB51911C27148F788333BBC782D69C9@PH0PR11MB5191.namprd11.prod.outlook.com>
From: David Steele <david@pgmasters.net>
In-Reply-To: 
 <PH0PR11MB51911C27148F788333BBC782D69C9@PH0PR11MB5191.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: 
 <https://www.postgresql.org/message-id/dee5201a-cf77-ea09-ec9b-0d22afb91039%40pgmasters.net>
Precedence: bulk

On 8/3/22 05:23, James Pang (chaolpan) wrote:
> 
>     We use pgbackrest 2.34 on RHEL8.4 to backup PGv13 database to AWS S3 
> buckets , we found after some time successful backup , suddenly got 
> below error:
> 
> 2022-07-23 10:30:04.743 P00   INFO: archive-push:async command begin 
> 2.38: [/pgdata/pg_wal] --archive-async --archive-push-queue-max=10240MB 
> --compress-type=none --exec-id=813450-3a783f75 --log-level-console=off 
> --log-level-stderr=off --pg1-path=/pgdata --process-max=2 
> --repo1-path=/pgbackup --repo1-s3-bucket=xxxxxxxxx 
> --repo1-s3-endpoint=s3-fips.us-west-2.amazonaws.com 
> --repo1-s3-key-type=auto --repo1-s3-region=us-west-2 --repo1-type=s3 
> --spool-path=/pgarchive/pgbackrest --stanza=xxxxxxx
> 
> 2022-07-23 10:30:04.744 P00   INFO: push 1 WAL file(s) to archive: 
> 00000001000000DB000000A4
> 
> 2022-07-23 10:30:04.907 P00  ERROR: [103]: unable to find a valid 
> repository:
> 
>                                      repo1: [FileMissingError] unable to 
> load info file '/pgbackup/archive/xxxxxx/archive.info' or 
> '/pgbackup/archive/xxxxxx/archive.info.copy':
> 
>                                      FileMissingError: unable to open 
> missing file '/pgbackup/archive/xxxxxx/archive.info' for read
> 
>                                      FileMissingError: unable to open 
> missing file '/pgbackup/archive/xxxxxx/archive.info.copy' for read
> 
>                                      HINT: archive.info cannot be opened 
> but is required to push/get WAL segments.
> 
>                                      HINT: is archive_command configured 
> correctly in postgresql.conf?
> 
>                                      HINT: has a stanza-create been 
> performed?
> 
>                                      HINT: use --no-archive-check to 
> disable archive checks during backup if you have an alternate archiving 
> scheme.
> 
> Then we login to check both archive.info and archive.info.copy got 
> disappeared in S3 bucket , we do not have job or command to manually 
> delete that file, and we do not recreate standza either, not sure why 
> suddenly missed the archive.info. without the archive.info, we have to 
> recreate pgbackrest stanza repo again, but all old backup pieces got 
> invalid too.
> 
> Could you help figure out how that happened ? we faced this issue 
> several times .

These files do not get updated after stanza-create. Do you have any kind 
of auto-expiration of old files in the bucket? e.g., 
https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration.

Regards,
-David