public inbox for [email protected]
help / color / mirror / Atom feedpgbackrest creating new directories (messing up cron jobs).
6+ messages / 2 participants
[nested] [flat]
* pgbackrest creating new directories (messing up cron jobs).
@ 2020-03-10 10:35 [email protected]
2020-03-10 13:44 ` Re: pgbackrest creating new directories (messing up cron jobs). Stephen Frost <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: [email protected] @ 2020-03-10 10:35 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
Hello,
My pgbackrest creates the files in
/var/lib/pgbackrest/archive/main/10/0000000100000000
Each file looks something like:
000000010000000100000022-033bbe600913c068e89c3d063afb1f0527bf513d.gz.
However, when the last two digits of the section before the hyphen become
FF, pgbackrest creates a new directory:
/var/lib/pgbackrest/archive/main/10/0000000100000001.
This is annoying as I have a cron job set up to focus on files in the
0000000100000000 directory.
Is there a way to force bgbackrest to only use one directory (e.g.
0000000100000000)?
Thanks.
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbackrest creating new directories (messing up cron jobs).
2020-03-10 10:35 pgbackrest creating new directories (messing up cron jobs). [email protected]
@ 2020-03-10 13:44 ` Stephen Frost <[email protected]>
2020-03-10 22:23 ` Re: pgbackrest creating new directories (messing up cron jobs). [email protected]
0 siblings, 1 reply; 6+ messages in thread
From: Stephen Frost @ 2020-03-10 13:44 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
Greetings,
* [email protected] ([email protected]) wrote:
> My pgbackrest creates the files in
> /var/lib/pgbackrest/archive/main/10/0000000100000000
Yes, those are files which are created in the repo.
> Each file looks something like:
> 000000010000000100000022-033bbe600913c068e89c3d063afb1f0527bf513d.gz.
That's currently the naming, but we anticipate changing it in the future
and so I strongly recommend that you do not make assumptions or depend
on that exact naming.
> However, when the last two digits of the section before the hyphen become
> FF, pgbackrest creates a new directory:
> /var/lib/pgbackrest/archive/main/10/0000000100000001.
Yes.
> This is annoying as I have a cron job set up to focus on files in the
> 0000000100000000 directory.
What is the cronjob doing..? You really shouldn't be hacking around
with things in the repo- there's commands available like archive-get to
extract out WAL files from the repo.
> Is there a way to force bgbackrest to only use one directory (e.g.
> 0000000100000000)?
No.
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbackrest creating new directories (messing up cron jobs).
2020-03-10 10:35 pgbackrest creating new directories (messing up cron jobs). [email protected]
2020-03-10 13:44 ` Re: pgbackrest creating new directories (messing up cron jobs). Stephen Frost <[email protected]>
@ 2020-03-10 22:23 ` [email protected]
2020-03-10 22:37 ` Re: pgbackrest creating new directories (messing up cron jobs). Stephen Frost <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: [email protected] @ 2020-03-10 22:23 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: [email protected]
Thank you for your answer!
>
>> This is annoying as I have a cron job set up to focus on files in the
>> 0000000100000000 directory.
>>
>
> What is the cronjob doing..? You really shouldn't be hacking around
> with things in the repo- there's commands available like archive-get to
> extract out WAL files from the repo.
>
The cronjob is using 'mutt' to mail me the .gz files at a set period.
It says:
cd /var/lib/pgbackrest/archive/main/10/0000000100000000/
echo "" | mutt -s "Ttile" [email protected] -a *.gz
Of course, once 0000000100000000 becomes 0000000100000001, this no longer
works.
Is there, perhaps, a better way to achieve my goal?
Thanks again - I do appreciate all your help.
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbackrest creating new directories (messing up cron jobs).
2020-03-10 10:35 pgbackrest creating new directories (messing up cron jobs). [email protected]
2020-03-10 13:44 ` Re: pgbackrest creating new directories (messing up cron jobs). Stephen Frost <[email protected]>
2020-03-10 22:23 ` Re: pgbackrest creating new directories (messing up cron jobs). [email protected]
@ 2020-03-10 22:37 ` Stephen Frost <[email protected]>
2020-03-11 00:08 ` Re: pgbackrest creating new directories (messing up cron jobs). [email protected]
0 siblings, 1 reply; 6+ messages in thread
From: Stephen Frost @ 2020-03-10 22:37 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
Greetings,
* [email protected] ([email protected]) wrote:
> >> This is annoying as I have a cron job set up to focus on files in the
> >> 0000000100000000 directory.
> >
> > What is the cronjob doing..? You really shouldn't be hacking around
> > with things in the repo- there's commands available like archive-get to
> > extract out WAL files from the repo.
>
> The cronjob is using 'mutt' to mail me the .gz files at a set period.
>
> It says:
>
> cd /var/lib/pgbackrest/archive/main/10/0000000100000000/
> echo "" | mutt -s "Ttile" [email protected] -a *.gz
Isn't that going to mail you the same WAL over and over again if you set
it up as a cronjob..? Is that really what you want?
> Of course, once 0000000100000000 becomes 0000000100000001, this no longer
> works.
That doesn't seem like the worst of the problems here.
> Is there, perhaps, a better way to achieve my goal?
Well, you could use archive-get with pgbackrest to pull out the files by
requesting each segment number, but I'm not sure what the idea here is
exactly- *why* are you email'ing them?
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbackrest creating new directories (messing up cron jobs).
2020-03-10 10:35 pgbackrest creating new directories (messing up cron jobs). [email protected]
2020-03-10 13:44 ` Re: pgbackrest creating new directories (messing up cron jobs). Stephen Frost <[email protected]>
2020-03-10 22:23 ` Re: pgbackrest creating new directories (messing up cron jobs). [email protected]
2020-03-10 22:37 ` Re: pgbackrest creating new directories (messing up cron jobs). Stephen Frost <[email protected]>
@ 2020-03-11 00:08 ` [email protected]
2020-03-11 00:37 ` Re: pgbackrest creating new directories (messing up cron jobs). Stephen Frost <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: [email protected] @ 2020-03-11 00:08 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: [email protected]
I think I need to take a step backwards and ask something else before
moving forward.
>
> Isn't that going to mail you the same WAL over and over again if you set
> it up as a cronjob..? Is that really what you want?
>
I took a full backup when I started. Then I used type=diff. This creates
two archives every x hours (depending on the time period set in cron).
For example:
-rw-r----- 1 postgres postgres 27145 Mar 11 00:00
0000000100000001000000B9-6f3902fe5c3bdebc3c1c124ec6821c7206e350da.gz
-rw-r----- 1 postgres postgres 27126 Mar 11 00:00
0000000100000001000000BA-214e7142c6eda0a350577f6bd624c3db203e184f.gz
Only one of these relates to the database I setup in PostgreSQL. I don't
know what the other one is but it might be related to the "default"
database 'postgres'. (This is just a guess).
Irrespective, I would only want a new archive file if something has
changed in the database. Yet I seem to always receive two new files every
x hours even if nothing has changed.
So I tried type-incr but that seems to produce the same results as type=diff.
I wonder what I am doing wrong?
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbackrest creating new directories (messing up cron jobs).
2020-03-10 10:35 pgbackrest creating new directories (messing up cron jobs). [email protected]
2020-03-10 13:44 ` Re: pgbackrest creating new directories (messing up cron jobs). Stephen Frost <[email protected]>
2020-03-10 22:23 ` Re: pgbackrest creating new directories (messing up cron jobs). [email protected]
2020-03-10 22:37 ` Re: pgbackrest creating new directories (messing up cron jobs). Stephen Frost <[email protected]>
2020-03-11 00:08 ` Re: pgbackrest creating new directories (messing up cron jobs). [email protected]
@ 2020-03-11 00:37 ` Stephen Frost <[email protected]>
0 siblings, 0 replies; 6+ messages in thread
From: Stephen Frost @ 2020-03-11 00:37 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
Greetings,
* [email protected] ([email protected]) wrote:
> I think I need to take a step backwards and ask something else before
> moving forward.
Good idea.
> > Isn't that going to mail you the same WAL over and over again if you set
> > it up as a cronjob..? Is that really what you want?
>
> I took a full backup when I started. Then I used type=diff. This creates
> two archives every x hours (depending on the time period set in cron).
WAL files are not "archives" in the sense that they are a complete
database backup- they're just the write-ahead-logs from the ongoing
running of PG. You need the actual data files as well as the WAL to
perform a restore.
> For example:
>
> -rw-r----- 1 postgres postgres 27145 Mar 11 00:00
> 0000000100000001000000B9-6f3902fe5c3bdebc3c1c124ec6821c7206e350da.gz
>
> -rw-r----- 1 postgres postgres 27126 Mar 11 00:00
> 0000000100000001000000BA-214e7142c6eda0a350577f6bd624c3db203e184f.gz
>
> Only one of these relates to the database I setup in PostgreSQL. I don't
> know what the other one is but it might be related to the "default"
> database 'postgres'. (This is just a guess).
No, that's not how WAL works. Those are two WAL files and they're both
generated as part of running the PG system- they are not specific to one
database or another in PG.
> Irrespective, I would only want a new archive file if something has
> changed in the database. Yet I seem to always receive two new files every
> x hours even if nothing has changed.
These individuals files are not archives of the entire PG system or
anything you can use to reconstruct a running PG system with just one
file.
> So I tried type-incr but that seems to produce the same results as type=diff.
The difference between incremental and differential backups is
documented at https://pgbackrest.org
> I wonder what I am doing wrong?
I'm afraid there are some pretty fundamental misunderstandings that you
have about the PG WAL, PG backups, and what you can do with a PG cluster
andf file-level backup/restore.
Documentation about the WAL can be found here:
https://www.postgresql.org/docs/current/wal-intro.html
If what you're actually looking for is a single file that has a backup
of your entire database, you can get that by running:
pgbackrest backup --archive-copy
and then do:
pgbackrest restore --pg-path=/path/to/somewhere
tar -czf pgdata.tar.gz /path/to/somewhere
or so. Check the pgbackrest docs for the specific command syntax and
such.
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2020-03-11 00:37 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-03-10 10:35 pgbackrest creating new directories (messing up cron jobs). [email protected]
2020-03-10 13:44 ` Stephen Frost <[email protected]>
2020-03-10 22:23 ` [email protected]
2020-03-10 22:37 ` Stephen Frost <[email protected]>
2020-03-11 00:08 ` [email protected]
2020-03-11 00:37 ` Stephen Frost <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox