pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedpgbackrest archive.info got deleted unexpected
4+ messages / 4 participants
[nested] [flat]
* pgbackrest archive.info got deleted unexpected
@ 2022-08-03 09:23 James Pang (chaolpan) <chaolpan@cisco.com>
0 siblings, 3 replies; 4+ messages in thread
From: James Pang (chaolpan) @ 2022-08-03 09:23 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
Hi,
We use pgbackrest 2.34 on RHEL8.4 to backup PGv13 database to AWS S3 buckets , we found after some time successful backup , suddenly got below error:
2022-07-23 10:30:04.743 P00 INFO: archive-push:async command begin 2.38: [/pgdata/pg_wal] --archive-async --archive-push-queue-max=10240MB --compress-type=none --exec-id=813450-3a783f75 --log-level-console=off --log-level-stderr=off --pg1-path=/pgdata --process-max=2 --repo1-path=/pgbackup --repo1-s3-bucket=xxxxxxxxx --repo1-s3-endpoint=s3-fips.us-west-2.amazonaws.com --repo1-s3-key-type=auto --repo1-s3-region=us-west-2 --repo1-type=s3 --spool-path=/pgarchive/pgbackrest --stanza=xxxxxxx
2022-07-23 10:30:04.744 P00 INFO: push 1 WAL file(s) to archive: 00000001000000DB000000A4
2022-07-23 10:30:04.907 P00 ERROR: [103]: unable to find a valid repository:
repo1: [FileMissingError] unable to load info file '/pgbackup/archive/xxxxxx/archive.info' or '/pgbackup/archive/xxxxxx/archive.info.copy':
FileMissingError: unable to open missing file '/pgbackup/archive/xxxxxx/archive.info' for read
FileMissingError: unable to open missing file '/pgbackup/archive/xxxxxx/archive.info.copy' for read
HINT: archive.info cannot be opened but is required to push/get WAL segments.
HINT: is archive_command configured correctly in postgresql.conf?
HINT: has a stanza-create been performed?
HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving scheme.
Then we login to check both archive.info and archive.info.copy got disappeared in S3 bucket , we do not have job or command to manually delete that file, and we do not recreate standza either, not sure why suddenly missed the archive.info. without the archive.info, we have to recreate pgbackrest stanza repo again, but all old backup pieces got invalid too.
Could you help figure out how that happened ? we faced this issue several times .
Thanks,
James
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pgbackrest archive.info got deleted unexpected
@ 2022-08-03 12:02 Ron <ronljohnsonjr@gmail.com>
parent: James Pang (chaolpan) <chaolpan@cisco.com>
2 siblings, 0 replies; 4+ messages in thread
From: Ron @ 2022-08-03 12:02 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
On 8/3/22 04:23, James Pang (chaolpan) wrote:
>
> Hi,
>
> We use pgbackrest 2.34 on RHEL8.4 to backup PGv13 database to AWS S3
> buckets , we found after some time successful backup , suddenly got below
> error:
>
> 2022-07-23 10:30:04.743 P00 INFO: archive-push:async command begin 2.38:
> [/pgdata/pg_wal] --archive-async --archive-push-queue-max=10240MB
> --compress-type=none --exec-id=813450-3a783f75 --log-level-console=off
> --log-level-stderr=off --pg1-path=/pgdata --process-max=2
> --repo1-path=/pgbackup --repo1-s3-bucket=xxxxxxxxx
> --repo1-s3-endpoint=s3-fips.us-west-2.amazonaws.com
> --repo1-s3-key-type=auto --repo1-s3-region=us-west-2 --repo1-type=s3
> --spool-path=/pgarchive/pgbackrest --stanza=xxxxxxx
>
> 2022-07-23 10:30:04.744 P00 INFO: push 1 WAL file(s) to archive:
> 00000001000000DB000000A4
>
> 2022-07-23 10:30:04.907 P00 ERROR: [103]: unable to find a valid repository:
>
> repo1: [FileMissingError] unable to
> load info file '/pgbackup/archive/xxxxxx/archive.info' or
> '/pgbackup/archive/xxxxxx/archive.info.copy':
>
> FileMissingError: unable to open missing file
> '/pgbackup/archive/xxxxxx/archive.info' for read
>
> FileMissingError: unable to open missing file
> '/pgbackup/archive/xxxxxx/archive.info.copy' for read
>
> HINT: archive.info cannot be opened
> but is required to push/get WAL segments.
>
> HINT: is archive_command configured
> correctly in postgresql.conf?
>
> HINT: has a stanza-create been performed?
>
> HINT: use --no-archive-check to
> disable archive checks during backup if you have an alternate archiving
> scheme.
>
> Then we login to check both archive.info and archive.info.copy got
> disappeared in S3 bucket , we do not have job or command to manually
> delete that file, and we do not recreate standza either, not sure why
> suddenly missed the archive.info. without the archive.info, we have to
> recreate pgbackrest stanza repo again, but all old backup pieces got
> invalid too.
>
> Could you help figure out how that happened ? we faced this issue several
> times .
>
This is certainly a question for AWS ("how do I audit bucket deletes?") or
the pgbackrest github site, not postgresql.
--
Angular momentum makes the world go 'round.
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pgbackrest archive.info got deleted unexpected
@ 2022-08-03 12:08 David Steele <david@pgmasters.net>
parent: James Pang (chaolpan) <chaolpan@cisco.com>
2 siblings, 0 replies; 4+ messages in thread
From: David Steele @ 2022-08-03 12:08 UTC (permalink / raw)
To: James Pang (chaolpan) <chaolpan@cisco.com>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
On 8/3/22 05:23, James Pang (chaolpan) wrote:
>
> We use pgbackrest 2.34 on RHEL8.4 to backup PGv13 database to AWS S3
> buckets , we found after some time successful backup , suddenly got
> below error:
>
> 2022-07-23 10:30:04.743 P00 INFO: archive-push:async command begin
> 2.38: [/pgdata/pg_wal] --archive-async --archive-push-queue-max=10240MB
> --compress-type=none --exec-id=813450-3a783f75 --log-level-console=off
> --log-level-stderr=off --pg1-path=/pgdata --process-max=2
> --repo1-path=/pgbackup --repo1-s3-bucket=xxxxxxxxx
> --repo1-s3-endpoint=s3-fips.us-west-2.amazonaws.com
> --repo1-s3-key-type=auto --repo1-s3-region=us-west-2 --repo1-type=s3
> --spool-path=/pgarchive/pgbackrest --stanza=xxxxxxx
>
> 2022-07-23 10:30:04.744 P00 INFO: push 1 WAL file(s) to archive:
> 00000001000000DB000000A4
>
> 2022-07-23 10:30:04.907 P00 ERROR: [103]: unable to find a valid
> repository:
>
> repo1: [FileMissingError] unable to
> load info file '/pgbackup/archive/xxxxxx/archive.info' or
> '/pgbackup/archive/xxxxxx/archive.info.copy':
>
> FileMissingError: unable to open
> missing file '/pgbackup/archive/xxxxxx/archive.info' for read
>
> FileMissingError: unable to open
> missing file '/pgbackup/archive/xxxxxx/archive.info.copy' for read
>
> HINT: archive.info cannot be opened
> but is required to push/get WAL segments.
>
> HINT: is archive_command configured
> correctly in postgresql.conf?
>
> HINT: has a stanza-create been
> performed?
>
> HINT: use --no-archive-check to
> disable archive checks during backup if you have an alternate archiving
> scheme.
>
> Then we login to check both archive.info and archive.info.copy got
> disappeared in S3 bucket , we do not have job or command to manually
> delete that file, and we do not recreate standza either, not sure why
> suddenly missed the archive.info. without the archive.info, we have to
> recreate pgbackrest stanza repo again, but all old backup pieces got
> invalid too.
>
> Could you help figure out how that happened ? we faced this issue
> several times .
These files do not get updated after stanza-create. Do you have any kind
of auto-expiration of old files in the bucket? e.g.,
https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration.
Regards,
-David
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pgbackrest archive.info got deleted unexpected
@ 2022-08-03 12:09 Dan Smith <j.daniel.smith1@gmail.com>
parent: James Pang (chaolpan) <chaolpan@cisco.com>
2 siblings, 0 replies; 4+ messages in thread
From: Dan Smith @ 2022-08-03 12:09 UTC (permalink / raw)
To: pgsql-admin <pgsql-admin@lists.postgresql.org>
This doesn't have anything directly to do with Postgres server or client.
Questions which might help you with troubleshooting the issue:
- Are you certain the file is gone and not just owned by a specific user?
- Was the file a symbolic link or a mount?
- Have you checked logs including history for users that have access to the
server?
- Have you checked the S3 bucket? Does it still exist and have the data?
- How was this installed/setup the first time? Do you have a ticket or
documentation that might help you with troubleshooting?
Again, this is the wrong forum / list for the question, but, given the
information, getting answers to some of these would be my approach. REL
support or https://pgbackrest.org/#support (which points to Crunchy Data)
may be able to help further.
Best regards,
Dan Smith
On Wed, Aug 3, 2022, 05:23 James Pang (chaolpan) <chaolpan@cisco.com> wrote:
> Hi,
>
> We use pgbackrest 2.34 on RHEL8.4 to backup PGv13 database to AWS S3
> buckets , we found after some time successful backup , suddenly got below
> error:
>
> 2022-07-23 10:30:04.743 P00 INFO: archive-push:async command begin 2.38:
> [/pgdata/pg_wal] --archive-async --archive-push-queue-max=10240MB
> --compress-type=none --exec-id=813450-3a783f75 --log-level-console=off
> --log-level-stderr=off --pg1-path=/pgdata --process-max=2
> --repo1-path=/pgbackup --repo1-s3-bucket=xxxxxxxxx --repo1-s3-endpoint=
> s3-fips.us-west-2.amazonaws.com --repo1-s3-key-type=auto
> --repo1-s3-region=us-west-2 --repo1-type=s3
> --spool-path=/pgarchive/pgbackrest --stanza=xxxxxxx
>
> 2022-07-23 10:30:04.744 P00 INFO: push 1 WAL file(s) to archive:
> 00000001000000DB000000A4
>
> 2022-07-23 10:30:04.907 P00 ERROR: [103]: unable to find a valid
> repository:
>
> repo1: [FileMissingError] unable to
> load info file '/pgbackup/archive/xxxxxx/archive.info' or
> '/pgbackup/archive/xxxxxx/archive.info.copy':
>
> FileMissingError: unable to open
> missing file '/pgbackup/archive/xxxxxx/archive.info' for read
>
> FileMissingError: unable to open
> missing file '/pgbackup/archive/xxxxxx/archive.info.copy' for read
>
> HINT: archive.info cannot be opened
> but is required to push/get WAL segments.
>
> HINT: is archive_command configured
> correctly in postgresql.conf?
>
> HINT: has a stanza-create been
> performed?
>
> HINT: use --no-archive-check to
> disable archive checks during backup if you have an alternate archiving
> scheme.
>
>
>
> Then we login to check both archive.info and archive.info.copy got
> disappeared in S3 bucket , we do not have job or command to manually delete
> that file, and we do not recreate standza either, not sure why suddenly
> missed the archive.info. without the archive.info, we have to recreate
> pgbackrest stanza repo again, but all old backup pieces got invalid too.
>
>
>
> Could you help figure out how that happened ? we faced this issue several
> times .
>
>
>
> Thanks,
>
>
>
> James
>
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2022-08-03 12:09 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-08-03 09:23 pgbackrest archive.info got deleted unexpected James Pang (chaolpan) <chaolpan@cisco.com>
2022-08-03 12:02 ` Ron <ronljohnsonjr@gmail.com>
2022-08-03 12:08 ` David Steele <david@pgmasters.net>
2022-08-03 12:09 ` Dan Smith <j.daniel.smith1@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox