pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedpg_backrest no longer backing up
4+ messages / 2 participants
[nested] [flat]
* pg_backrest no longer backing up
@ 2017-03-08 19:48 Aldo Sarmiento <aldo@bigpurpledot.com>
0 siblings, 1 reply; 4+ messages in thread
From: Aldo Sarmiento @ 2017-03-08 19:48 UTC (permalink / raw)
To: pgsql-admin
Hello,
Sometime in December in 2016 we ran out of space for our backup directory,
which we increased promptly. However, it appears that backrest stopped
taking backups since the incident.
Versions of things:
pg_backrest 0.75
postgresql 9.4
ubuntu 14.04 (This is the backup server, not master)
Couple things I noticed:
*postgres@pgdedicated:/backup/backrest/log$ cat
prod-9.4-backup-20170308.log*
-------------------PROCESS START-------------------
2017-03-08 01:30:02.034 T00 INFO: backup start: type = diff
2017-03-08 01:30:02.835 T00 INFO: last backup label = 20161204-025729F,
version = 0.75
2017-03-08 01:30:02.911 T00 INFO: executing pg_start_backup() with label
"pg_backrest backup started 2017-03-08 01:30:02": backup will begin after
the requested immediate checkpoint completes
2017-03-08 01:30:02.976 T00 ERROR: process terminated on signal or
exception, 0 threads stopped
*postgres@pgdedicated:/backup/backrest/log$ ls -lah
/backup/backrest/backup/prod-9.4/*
total 44K
drwxrwxr-x 11 postgres postgres 4.0K Dec 22 14:02 .
drwxrwxr-x 3 postgres postgres 4.0K Mar 15 2016 ..
drwxrwxr-x 3 postgres postgres 4.0K Nov 20 03:03 20161120-030305F
drwxrwxr-x 3 postgres postgres 4.0K Nov 27 02:53 20161127-025355F
drwxrwxr-x 3 postgres postgres 4.0K Dec 3 02:24
20161127-025355F_20161203-022437D
drwxrwxr-x 3 postgres postgres 4.0K Dec 4 02:57 20161204-025729F
drwxrwxr-x 3 postgres postgres 4.0K Dec 5 02:24
20161204-025729F_20161205-022427D
drwxrwxr-x 3 postgres postgres 4.0K Dec 6 02:25
20161204-025729F_20161206-022512D
drwxrwxr-x 3 postgres postgres 4.0K Dec 7 02:28
20161204-025729F_20161207-022814D
drwxrwxr-x 3 postgres postgres 4.0K Dec 8 02:27
20161204-025729F_20161208-022730D
drwxrwxr-x 3 postgres postgres 4.0K Dec 9 02:28
20161204-025729F_20161209-022855D
lrwxrwxrwx 1 postgres postgres 33 Dec 9 02:28 latest ->
20161204-025729F_20161209-022855D
*postgres@pgdedicated:/backup/backrest/log$ cat /etc/cron.d/backrest*
# Vacuum every Sunday before Full backup
#05 00 * * 0 postgres vacuumdb -qaz
# Full backup every Sunday
#30 01 * * 0 postgres /usr/lib/backrest/bin/pg_backrest.pl
--stanza=prod-9.4 --type=full backup
# Differential backup all other days
#30 01 * * 1,2,3,4,5,6 postgres /usr/lib/backrest/bin/pg_backrest.pl
--stanza=prod-9.4 --type=diff backup
*postgres@pgdedicated:/backup/backrest/log$ cat /etc/pg_backrest.conf*
[global:general]
repo-path=/backup/backrest
[global:backup]
start-fast=y
[global:expire]
retention-full=6 *# This was originally 8, dialed it down to 6 when ran out
of space*
retention-diff=6
retention-archive-type=full
retention-archive=6 *# This was originally 8, dialed it down to 6 when ran
out of space*
[prod-9.4]
db-path=/var/lib/postgresql/9.4/replica
Question: Not sure when the backrest cron was commented out, but how is it
still attempting to do backups without this setup? I'm assuming it's being
invoked some other way?
Also, any idea what the issue is with the error?
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pg_backrest no longer backing up
@ 2017-03-08 22:18 David Steele <david@pgmasters.net>
parent: Aldo Sarmiento <aldo@bigpurpledot.com>
0 siblings, 1 reply; 4+ messages in thread
From: David Steele @ 2017-03-08 22:18 UTC (permalink / raw)
To: Aldo Sarmiento <aldo@bigpurpledot.com>; pgsql-admin
Hi Aldo,
On 3/8/17 11:48 AM, Aldo Sarmiento wrote:
> Sometime in December in 2016 we ran out of space for our backup
> directory, which we increased promptly. However, it appears that
> backrest stopped taking backups since the incident.
>
> Versions of things:
>
> pg_backrest 0.75
This is an incredibly old, pre-1.0 version of pgBackRest. I suggest you
upgrade to a current version. 1.X will not be compatible with your
current repo so it will need to be a fresh install.
> *postgres@pgdedicated:/backup/backrest/log$ cat
> prod-9.4-backup-20170308.log*
> -------------------PROCESS START-------------------
> 2017-03-08 01:30:02.034 T00 INFO: backup start: type = diff
> 2017-03-08 01:30:02.835 T00 INFO: last backup label =
> 20161204-025729F, version = 0.75
> 2017-03-08 01:30:02.911 T00 INFO: executing pg_start_backup() with
> label "pg_backrest backup started 2017-03-08 01:30:02": backup will
> begin after the requested immediate checkpoint completes
> 2017-03-08 01:30:02.976 T00 ERROR: process terminated on signal or
> exception, 0 threads stopped
This could be any number of things and there's not enough information
given here to even begin to guess.
> Question: Not sure when the backrest cron was commented out, but how is
> it still attempting to do backups without this setup? I'm assuming it's
> being invoked some other way?
I'm guessing it's being invoked in some other way, perhaps the postgres
user's crontab?
> Also, any idea what the issue is with the error?
Again, there's not enough information given for me to form an opinion.
Regards,
--
-David
david@pgmasters.net
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pg_backrest no longer backing up
@ 2017-03-08 22:53 Aldo Sarmiento <aldo@bigpurpledot.com>
parent: David Steele <david@pgmasters.net>
0 siblings, 1 reply; 4+ messages in thread
From: Aldo Sarmiento @ 2017-03-08 22:53 UTC (permalink / raw)
To: David Steele <david@pgmasters.net>; pgsql-admin
Hi David,
Actually got it figured out. Main thing is that I should clarify that the
aforementioned crontab was irrelevant since I was looking at the replica.
Once logged into master, the crontab is intact and uncommented. The problem
was that a backup failed and pg_backrest was failing to start.
I tried adding auto-stop=true to conf, but it didn't help. In the end I
just did pg_stop_backup() and things are running smoothly now.
As for the upgrade, yes that's a need we have. Just trying to see if we can
migrate our current backups to work with the new version so we don't lose
ability for PITR.
Thanks!
On Wed, Mar 8, 2017 at 2:18 PM David Steele <david@pgmasters.net> wrote:
> Hi Aldo,
>
> On 3/8/17 11:48 AM, Aldo Sarmiento wrote:
>
> > Sometime in December in 2016 we ran out of space for our backup
> > directory, which we increased promptly. However, it appears that
> > backrest stopped taking backups since the incident.
> >
> > Versions of things:
> >
> > pg_backrest 0.75
>
> This is an incredibly old, pre-1.0 version of pgBackRest. I suggest you
> upgrade to a current version. 1.X will not be compatible with your
> current repo so it will need to be a fresh install.
>
> > *postgres@pgdedicated:/backup/backrest/log$ cat
> > prod-9.4-backup-20170308.log*
> > -------------------PROCESS START-------------------
> > 2017-03-08 01:30:02.034 T00 INFO: backup start: type = diff
> > 2017-03-08 01:30:02.835 T00 INFO: last backup label =
> > 20161204-025729F, version = 0.75
> > 2017-03-08 01:30:02.911 T00 INFO: executing pg_start_backup() with
> > label "pg_backrest backup started 2017-03-08 01:30:02": backup will
> > begin after the requested immediate checkpoint completes
> > 2017-03-08 01:30:02.976 T00 ERROR: process terminated on signal or
> > exception, 0 threads stopped
>
> This could be any number of things and there's not enough information
> given here to even begin to guess.
>
> > Question: Not sure when the backrest cron was commented out, but how is
> > it still attempting to do backups without this setup? I'm assuming it's
> > being invoked some other way?
>
> I'm guessing it's being invoked in some other way, perhaps the postgres
> user's crontab?
>
> > Also, any idea what the issue is with the error?
>
> Again, there's not enough information given for me to form an opinion.
>
> Regards,
> --
> -David
> david@pgmasters.net
>
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pg_backrest no longer backing up
@ 2017-03-13 20:34 David Steele <david@pgmasters.net>
parent: Aldo Sarmiento <aldo@bigpurpledot.com>
0 siblings, 0 replies; 4+ messages in thread
From: David Steele @ 2017-03-13 20:34 UTC (permalink / raw)
To: Aldo Sarmiento <aldo@bigpurpledot.com>; pgsql-admin
Hi Aldo,
On 3/8/17 5:53 PM, Aldo Sarmiento wrote:
>
> Actually got it figured out. Main thing is that I should clarify that
> the aforementioned crontab was irrelevant since I was looking at the
> replica.
>
> Once logged into master, the crontab is intact and uncommented.
That makes sense.
> The
> problem was that a backup failed and pg_backrest was failing to start.
>
> I tried adding auto-stop=true to conf, but it didn't help. In the end I
> just did pg_stop_backup() and things are running smoothly now.
The option is actually stop-auto:
http://www.pgbackrest.org/user-guide.html#backup/option-stop-auto
But it wasn't added until 0.80 so wouldn't work for you anyway.
> As for the upgrade, yes that's a need we have. Just trying to see if we
> can migrate our current backups to work with the new version so we don't
> lose ability for PITR.
It might be possible but would require specialized knowledge and custom
code. We decided to make a clean break at 1.0 and require users to
create a new repository due to the number of changes in the format.
Regards,
--
-David
david@pgmasters.net
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2017-03-13 20:34 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-03-08 19:48 pg_backrest no longer backing up Aldo Sarmiento <aldo@bigpurpledot.com>
2017-03-08 22:18 ` David Steele <david@pgmasters.net>
2017-03-08 22:53 ` Aldo Sarmiento <aldo@bigpurpledot.com>
2017-03-13 20:34 ` David Steele <david@pgmasters.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox