Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oJD4o-00072y-W0 for pgsql-admin@arkaria.postgresql.org; Wed, 03 Aug 2022 12:02:15 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1oJD4n-0001sv-Ob for pgsql-admin@arkaria.postgresql.org; Wed, 03 Aug 2022 12:02:13 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oJD4n-0001sl-AD for pgsql-admin@lists.postgresql.org; Wed, 03 Aug 2022 12:02:13 +0000 Received: from mail-oi1-x233.google.com ([2607:f8b0:4864:20::233]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1oJD4l-0004TQ-Os for pgsql-admin@lists.postgresql.org; Wed, 03 Aug 2022 12:02:12 +0000 Received: by mail-oi1-x233.google.com with SMTP id q184so711650oif.1 for ; Wed, 03 Aug 2022 05:02:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :references:from:in-reply-to; bh=ExXEeymQOV1PjvPsSAtTRCB5HtpRoDOvuyGnkiYrfIU=; b=O1u+UHd7kJL3SSIEGWSbD+eI1h3917CR7PUhiP0k0CllbgB5XHBHFybY0W3jAAX04z QF6uMGU0tqUeM5WexUmHVtmQQ9L/SB2Oe/kF7yZLrOcmhxaS+Wi/2n+OW04VIMQLiwqQ T4e7vq5ApPsde8DV8l7aG3HQLMjRYawkfl6DRCCB8JY7krwcRiLoeJNfc9r+sJjChGkh bRJW7KC05P8vrE6+OsASeelAOXSvG8EYISa51wIOTgJSVRr5E+dnui63/oMyrZQLtqDC 6vRx2oppXi4u4LpHD7tsJVKUmtRhXXDXAiyz6njllXbBQ5THglQitSMFIAAMLkyFhcq0 6jKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:references:from:in-reply-to; bh=ExXEeymQOV1PjvPsSAtTRCB5HtpRoDOvuyGnkiYrfIU=; b=aTNb3CBKdee7bYyoG6sqcr9nikG4FeNnkdEmoJDx9OESWN6KqO9KX9DC85XpuDpCjw /Gtdc8BV0cS5+5H36363DJUuYMe5fLTdfMdWtadnJHNalIMFuE8Ks+D8V6pc1WxiMXm4 SPF5LEBLJf28g+a51L8WGOPoX1hcuI9qRvChZw99x/rlpKoAtGqX01WBAB885lsIpqsT Nt0er91Pyyg12FqKOauNNmc/ydz78gptAObS9npYWudKmlgfV+M1MpctzAV+DYGz/3E/ zGvbecF1rQljhrArwnF+sMhusm2TRNzA2R34Dyouxv4+fLJ6x8I/uLb+TV6ZG0K010Px oiNw== X-Gm-Message-State: ACgBeo2qSTEaAkLxD+yWTwXCSDjD5Xs801obtParItWDTQkxYUc7Bx9t SQ+isdBOIEkFkIcAOPdy0l0ltX+BXBU= X-Google-Smtp-Source: AA6agR4HQi+qa8c8spHR02sFj1N1wGci60jAO8fZPEZXYUNRFKpnD/Ko9gHChE+fXdKUJBjV1HIaLg== X-Received: by 2002:a05:6808:169e:b0:331:522a:4521 with SMTP id bb30-20020a056808169e00b00331522a4521mr1543536oib.293.1659528129277; Wed, 03 Aug 2022 05:02:09 -0700 (PDT) Received: from [192.168.88.10] (ip68-11-68-85.no.no.cox.net. [68.11.68.85]) by smtp.googlemail.com with ESMTPSA id i18-20020a056808031200b00326cb6225f8sm3426611oie.44.2022.08.03.05.02.08 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 03 Aug 2022 05:02:08 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------g0s9WsRq3RUz93r0M2dUd0az" Message-ID: Date: Wed, 3 Aug 2022 07:02:08 -0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.1 Subject: Re: pgbackrest archive.info got deleted unexpected Content-Language: en-US To: pgsql-admin@lists.postgresql.org References: From: Ron In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------g0s9WsRq3RUz93r0M2dUd0az Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 8/3/22 04:23, James Pang (chaolpan) wrote: > > Hi, > >    We use pgbackrest 2.34 on RHEL8.4 to backup PGv13 database to AWS S3 > buckets , we found after some time successful backup , suddenly got below > error: > > 2022-07-23 10:30:04.743 P00   INFO: archive-push:async command begin 2.38: > [/pgdata/pg_wal] --archive-async --archive-push-queue-max=10240MB > --compress-type=none --exec-id=813450-3a783f75 --log-level-console=off > --log-level-stderr=off --pg1-path=/pgdata --process-max=2 > --repo1-path=/pgbackup --repo1-s3-bucket=xxxxxxxxx > --repo1-s3-endpoint=s3-fips.us-west-2.amazonaws.com > --repo1-s3-key-type=auto --repo1-s3-region=us-west-2 --repo1-type=s3 > --spool-path=/pgarchive/pgbackrest --stanza=xxxxxxx > > 2022-07-23 10:30:04.744 P00   INFO: push 1 WAL file(s) to archive: > 00000001000000DB000000A4 > > 2022-07-23 10:30:04.907 P00  ERROR: [103]: unable to find a valid repository: > >                                     repo1: [FileMissingError] unable to > load info file '/pgbackup/archive/xxxxxx/archive.info' or > '/pgbackup/archive/xxxxxx/archive.info.copy': > > FileMissingError: unable to open missing file > '/pgbackup/archive/xxxxxx/archive.info' for read > > FileMissingError: unable to open missing file > '/pgbackup/archive/xxxxxx/archive.info.copy' for read > >                                     HINT: archive.info cannot be opened > but is required to push/get WAL segments. > >                                     HINT: is archive_command configured > correctly in postgresql.conf? > >                                     HINT: has a stanza-create been performed? > >                                     HINT: use --no-archive-check to > disable archive checks during backup if you have an alternate archiving > scheme. > > Then we login to check both archive.info and archive.info.copy got > disappeared in S3 bucket , we do not have job or command to manually > delete that file, and we do not recreate standza either, not sure why > suddenly missed the archive.info.  without the archive.info, we have to > recreate pgbackrest stanza repo again, but all old backup pieces got > invalid too. > > Could you help figure out how that happened ? we faced this issue several > times . > This is certainly a question for AWS ("how do I audit bucket deletes?") or the pgbackrest github site, not postgresql. -- Angular momentum makes the world go 'round. --------------g0s9WsRq3RUz93r0M2dUd0az Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit On 8/3/22 04:23, James Pang (chaolpan) wrote:

Hi,

   We use pgbackrest 2.34 on RHEL8.4 to backup PGv13 database to AWS S3 buckets , we found after some time successful backup , suddenly got below error:

2022-07-23 10:30:04.743 P00   INFO: archive-push:async command begin 2.38: [/pgdata/pg_wal] --archive-async --archive-push-queue-max=10240MB --compress-type=none --exec-id=813450-3a783f75 --log-level-console=off --log-level-stderr=off --pg1-path=/pgdata --process-max=2 --repo1-path=/pgbackup --repo1-s3-bucket=xxxxxxxxx --repo1-s3-endpoint=s3-fips.us-west-2.amazonaws.com --repo1-s3-key-type=auto --repo1-s3-region=us-west-2 --repo1-type=s3 --spool-path=/pgarchive/pgbackrest --stanza=xxxxxxx

2022-07-23 10:30:04.744 P00   INFO: push 1 WAL file(s) to archive: 00000001000000DB000000A4

2022-07-23 10:30:04.907 P00  ERROR: [103]: unable to find a valid repository:

                                    repo1: [FileMissingError] unable to load info file '/pgbackup/archive/xxxxxx/archive.info' or '/pgbackup/archive/xxxxxx/archive.info.copy':

                                    FileMissingError: unable to open missing file '/pgbackup/archive/xxxxxx/archive.info' for read

                                    FileMissingError: unable to open missing file '/pgbackup/archive/xxxxxx/archive.info.copy' for read

                                    HINT: archive.info cannot be opened but is required to push/get WAL segments.

                                    HINT: is archive_command configured correctly in postgresql.conf?

                                    HINT: has a stanza-create been performed?

                                    HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving scheme.

 

Then we login to check both archive.info and archive.info.copy got disappeared in S3 bucket , we do not have job or command to manually delete that file, and we do not recreate standza either, not sure why suddenly missed the archive.info.  without the archive.info, we have to recreate pgbackrest stanza repo again, but all old backup pieces got invalid too.

 

Could you help figure out how that happened ? we faced this issue several times .


This is certainly a question for AWS ("how do I audit bucket deletes?") or the pgbackrest github site, not postgresql.

--
Angular momentum makes the world go 'round.
--------------g0s9WsRq3RUz93r0M2dUd0az--