pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedFull backup failed
6+ messages / 5 participants
[nested] [flat]
* Full backup failed
@ 2024-02-19 04:05 Rajesh Kumar <rajeshkumar.dba09@gmail.com>
0 siblings, 2 replies; 6+ messages in thread
From: Rajesh Kumar @ 2024-02-19 04:05 UTC (permalink / raw)
To: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Hi,
Backup policy is weekly full backup using pgbackrest. Yesterday backups
failed and got below error. Please help
time="2024-02-18T18:23:17Z" level=info msg="crunchy-pgbackrest starts"
2time="2024-02-18T18:23:17Z" level=info msg="debug flag set to false"
3time="2024-02-18T18:23:17Z" level=info msg="backrest backup command
requested"
4time="2024-02-18T18:23:17Z" level=info msg="command to execute is
[pgbackrest backup --stanza=db --repo=1 --type=full]"
5time="2024-02-18T18:26:59Z" level=info msg="output=[]"
6time="2024-02-18T18:26:59Z" level=info msg="stderr=[ERROR: [082]: WAL
segment 0000003F0000050A0000000C was not archived before the 60000ms
timeout\n HINT: check the archive_command to ensure that all options are
correct (especially --stanza).\n HINT: check the PostgreSQL server log for
errors.\n HINT: run the 'start' command if the stanza was previously
stopped.\n]"
7time="2024-02-18T18:26:59Z" level=fatal msg="command terminated with exit
code 82"
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Full backup failed
@ 2024-02-19 05:05 Don Seiler <don@seiler.us>
parent: Rajesh Kumar <rajeshkumar.dba09@gmail.com>
1 sibling, 0 replies; 6+ messages in thread
From: Don Seiler @ 2024-02-19 05:05 UTC (permalink / raw)
To: Rajesh Kumar <rajeshkumar.dba09@gmail.com>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
On Sun, Feb 18, 2024 at 10:05 PM Rajesh Kumar <rajeshkumar.dba09@gmail.com>
wrote:
> 6time="2024-02-18T18:26:59Z" level=info msg="stderr=[ERROR: [082]: WAL
> segment 0000003F0000050A0000000C was not archived before the 60000ms
> timeout\n HINT: check the archive_command to ensure that all options are
> correct (especially --stanza).\n HINT: check the PostgreSQL server log for
> errors.\n HINT: run the 'start' command if the stanza was previously
> stopped.\n]"
>
> 7time="2024-02-18T18:26:59Z" level=fatal msg="command terminated with exit
> code 82"
>
What is your database instance's archive_command set to? Is that normally
succeeding?
--
Don Seiler
www.seiler.us
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Full backup failed
@ 2024-02-19 07:23 Laurenz Albe <laurenz.albe@cybertec.at>
parent: Rajesh Kumar <rajeshkumar.dba09@gmail.com>
1 sibling, 1 reply; 6+ messages in thread
From: Laurenz Albe @ 2024-02-19 07:23 UTC (permalink / raw)
To: Rajesh Kumar <rajeshkumar.dba09@gmail.com>; Pgsql-admin <pgsql-admin@lists.postgresql.org>
On Mon, 2024-02-19 at 09:35 +0530, Rajesh Kumar wrote:
> 6time="2024-02-18T18:26:59Z" level=info msg="stderr=[ERROR: [082]: WAL segment 0000003F0000050A0000000C was not archived before the 60000ms timeout\n HINT: check the archive_command to ensure that all options are correct (especially --stanza).\n HINT: check the PostgreSQL server log for errors.\n HINT: run the 'start' command if the stanza was previously stopped.\n]"
Look into the PostgreSQL log for messages from the "archive_comand".
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Full backup failed
@ 2024-02-19 07:43 Rajesh Kumar <rajeshkumar.dba09@gmail.com>
parent: Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 2 replies; 6+ messages in thread
From: Rajesh Kumar @ 2024-02-19 07:43 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Archive command is set to pgbackrest --stanza=db archive-push "%p"
There were no such issue since last year.
In postgres logs, don't see anything related to archive. But I see,
WARNING,0100, aborting backup due to backend exiting before pg_backup_stop
was called....pgbackrest[backup] client backend, 58568621736954596
On Mon, 19 Feb 2024, 12:53 Laurenz Albe, <laurenz.albe@cybertec.at> wrote:
> On Mon, 2024-02-19 at 09:35 +0530, Rajesh Kumar wrote:
> > 6time="2024-02-18T18:26:59Z" level=info msg="stderr=[ERROR: [082]: WAL
> segment 0000003F0000050A0000000C was not archived before the 60000ms
> timeout\n HINT: check the archive_command to ensure that all options are
> correct (especially --stanza).\n HINT: check the PostgreSQL server log for
> errors.\n HINT: run the 'start' command if the stanza was previously
> stopped.\n]"
>
> Look into the PostgreSQL log for messages from the "archive_comand".
>
> Yours,
> Laurenz Albe
>
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Full backup failed
@ 2024-02-19 07:50 Michael Banck <mbanck@gmx.net>
parent: Rajesh Kumar <rajeshkumar.dba09@gmail.com>
1 sibling, 0 replies; 6+ messages in thread
From: Michael Banck @ 2024-02-19 07:50 UTC (permalink / raw)
To: Rajesh Kumar <rajeshkumar.dba09@gmail.com>; +Cc: Laurenz Albe <laurenz.albe@cybertec.at>; Pgsql-admin <pgsql-admin@lists.postgresql.org>
Hi,
On Mon, Feb 19, 2024 at 01:13:36PM +0530, Rajesh Kumar wrote:
> Archive command is set to pgbackrest --stanza=db archive-push "%p"
That looks right.
> There were no such issue since last year.
>
> In postgres logs, don't see anything related to archive. But I see,
> WARNING,0100, aborting backup due to backend exiting before pg_backup_stop
> was called....pgbackrest[backup] client backend, 58568621736954596
This error is due to the timeout that pgbackrest backend suffered.
I think one reason pgbackrest backup could time out after 60s is WAL
pileup, i.e. it tries to check whether the current WAL archive was
successfully archived in 60s but if there is a huge lag in archiving
(for whatever reason) the current WAL segment might not get archived for
another 60 seconds and the backup fails.
So if you have that data in monitoring, check whether archiving was
maybe lagging behind, or otherwise, check whether there are a lot of
.ready files in the pg_wal/archive_status sub-directory right now and/or
possibly just try the backup again.
Michael
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Full backup failed
@ 2024-02-19 14:27 Amitpal Singh <palshekhawat@gmail.com>
parent: Rajesh Kumar <rajeshkumar.dba09@gmail.com>
1 sibling, 0 replies; 6+ messages in thread
From: Amitpal Singh @ 2024-02-19 14:27 UTC (permalink / raw)
To: Rajesh Kumar <rajeshkumar.dba09@gmail.com>; +Cc: Laurenz Albe <laurenz.albe@cybertec.at>; Pgsql-admin <pgsql-admin@lists.postgresql.org>
You can do one thing delete the stanza and re create it , it will work .
On Mon, Feb 19, 2024, 1:14 PM Rajesh Kumar <rajeshkumar.dba09@gmail.com>
wrote:
> Archive command is set to pgbackrest --stanza=db archive-push "%p"
>
> There were no such issue since last year.
>
> In postgres logs, don't see anything related to archive. But I see,
> WARNING,0100, aborting backup due to backend exiting before pg_backup_stop
> was called....pgbackrest[backup] client backend, 58568621736954596
>
> On Mon, 19 Feb 2024, 12:53 Laurenz Albe, <laurenz.albe@cybertec.at> wrote:
>
>> On Mon, 2024-02-19 at 09:35 +0530, Rajesh Kumar wrote:
>> > 6time="2024-02-18T18:26:59Z" level=info msg="stderr=[ERROR: [082]: WAL
>> segment 0000003F0000050A0000000C was not archived before the 60000ms
>> timeout\n HINT: check the archive_command to ensure that all options are
>> correct (especially --stanza).\n HINT: check the PostgreSQL server log for
>> errors.\n HINT: run the 'start' command if the stanza was previously
>> stopped.\n]"
>>
>> Look into the PostgreSQL log for messages from the "archive_comand".
>>
>> Yours,
>> Laurenz Albe
>>
>
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2024-02-19 14:27 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-02-19 04:05 Full backup failed Rajesh Kumar <rajeshkumar.dba09@gmail.com>
2024-02-19 05:05 ` Don Seiler <don@seiler.us>
2024-02-19 07:23 ` Laurenz Albe <laurenz.albe@cybertec.at>
2024-02-19 07:43 ` Rajesh Kumar <rajeshkumar.dba09@gmail.com>
2024-02-19 07:50 ` Michael Banck <mbanck@gmx.net>
2024-02-19 14:27 ` Amitpal Singh <palshekhawat@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox