public inbox for [email protected]help / color / mirror / Atom feed
pgbackrest creating new directories (messing up cron jobs). 6+ messages / 2 participants [nested] [flat]
* pgbackrest creating new directories (messing up cron jobs). @ 2020-03-10 10:35 [email protected] 0 siblings, 1 reply; 6+ messages in thread From: [email protected] @ 2020-03-10 10:35 UTC (permalink / raw) To: [email protected]; +Cc: [email protected] Hello, My pgbackrest creates the files in /var/lib/pgbackrest/archive/main/10/0000000100000000 Each file looks something like: 000000010000000100000022-033bbe600913c068e89c3d063afb1f0527bf513d.gz. However, when the last two digits of the section before the hyphen become FF, pgbackrest creates a new directory: /var/lib/pgbackrest/archive/main/10/0000000100000001. This is annoying as I have a cron job set up to focus on files in the 0000000100000000 directory. Is there a way to force bgbackrest to only use one directory (e.g. 0000000100000000)? Thanks. ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbackrest creating new directories (messing up cron jobs). @ 2020-03-10 13:44 Stephen Frost <[email protected]> parent: [email protected] 0 siblings, 1 reply; 6+ messages in thread From: Stephen Frost @ 2020-03-10 13:44 UTC (permalink / raw) To: [email protected]; +Cc: [email protected] Greetings, * [email protected] ([email protected]) wrote: > My pgbackrest creates the files in > /var/lib/pgbackrest/archive/main/10/0000000100000000 Yes, those are files which are created in the repo. > Each file looks something like: > 000000010000000100000022-033bbe600913c068e89c3d063afb1f0527bf513d.gz. That's currently the naming, but we anticipate changing it in the future and so I strongly recommend that you do not make assumptions or depend on that exact naming. > However, when the last two digits of the section before the hyphen become > FF, pgbackrest creates a new directory: > /var/lib/pgbackrest/archive/main/10/0000000100000001. Yes. > This is annoying as I have a cron job set up to focus on files in the > 0000000100000000 directory. What is the cronjob doing..? You really shouldn't be hacking around with things in the repo- there's commands available like archive-get to extract out WAL files from the repo. > Is there a way to force bgbackrest to only use one directory (e.g. > 0000000100000000)? No. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbackrest creating new directories (messing up cron jobs). @ 2020-03-10 22:23 [email protected] parent: Stephen Frost <[email protected]> 0 siblings, 1 reply; 6+ messages in thread From: [email protected] @ 2020-03-10 22:23 UTC (permalink / raw) To: Stephen Frost <[email protected]>; +Cc: [email protected] Thank you for your answer! > >> This is annoying as I have a cron job set up to focus on files in the >> 0000000100000000 directory. >> > > What is the cronjob doing..? You really shouldn't be hacking around > with things in the repo- there's commands available like archive-get to > extract out WAL files from the repo. > The cronjob is using 'mutt' to mail me the .gz files at a set period. It says: cd /var/lib/pgbackrest/archive/main/10/0000000100000000/ echo "" | mutt -s "Ttile" [email protected] -a *.gz Of course, once 0000000100000000 becomes 0000000100000001, this no longer works. Is there, perhaps, a better way to achieve my goal? Thanks again - I do appreciate all your help. ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbackrest creating new directories (messing up cron jobs). @ 2020-03-10 22:37 Stephen Frost <[email protected]> parent: [email protected] 0 siblings, 1 reply; 6+ messages in thread From: Stephen Frost @ 2020-03-10 22:37 UTC (permalink / raw) To: [email protected]; +Cc: [email protected] Greetings, * [email protected] ([email protected]) wrote: > >> This is annoying as I have a cron job set up to focus on files in the > >> 0000000100000000 directory. > > > > What is the cronjob doing..? You really shouldn't be hacking around > > with things in the repo- there's commands available like archive-get to > > extract out WAL files from the repo. > > The cronjob is using 'mutt' to mail me the .gz files at a set period. > > It says: > > cd /var/lib/pgbackrest/archive/main/10/0000000100000000/ > echo "" | mutt -s "Ttile" [email protected] -a *.gz Isn't that going to mail you the same WAL over and over again if you set it up as a cronjob..? Is that really what you want? > Of course, once 0000000100000000 becomes 0000000100000001, this no longer > works. That doesn't seem like the worst of the problems here. > Is there, perhaps, a better way to achieve my goal? Well, you could use archive-get with pgbackrest to pull out the files by requesting each segment number, but I'm not sure what the idea here is exactly- *why* are you email'ing them? Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbackrest creating new directories (messing up cron jobs). @ 2020-03-11 00:08 [email protected] parent: Stephen Frost <[email protected]> 0 siblings, 1 reply; 6+ messages in thread From: [email protected] @ 2020-03-11 00:08 UTC (permalink / raw) To: Stephen Frost <[email protected]>; +Cc: [email protected] I think I need to take a step backwards and ask something else before moving forward. > > Isn't that going to mail you the same WAL over and over again if you set > it up as a cronjob..? Is that really what you want? > I took a full backup when I started. Then I used type=diff. This creates two archives every x hours (depending on the time period set in cron). For example: -rw-r----- 1 postgres postgres 27145 Mar 11 00:00 0000000100000001000000B9-6f3902fe5c3bdebc3c1c124ec6821c7206e350da.gz -rw-r----- 1 postgres postgres 27126 Mar 11 00:00 0000000100000001000000BA-214e7142c6eda0a350577f6bd624c3db203e184f.gz Only one of these relates to the database I setup in PostgreSQL. I don't know what the other one is but it might be related to the "default" database 'postgres'. (This is just a guess). Irrespective, I would only want a new archive file if something has changed in the database. Yet I seem to always receive two new files every x hours even if nothing has changed. So I tried type-incr but that seems to produce the same results as type=diff. I wonder what I am doing wrong? ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbackrest creating new directories (messing up cron jobs). @ 2020-03-11 00:37 Stephen Frost <[email protected]> parent: [email protected] 0 siblings, 0 replies; 6+ messages in thread From: Stephen Frost @ 2020-03-11 00:37 UTC (permalink / raw) To: [email protected]; +Cc: [email protected] Greetings, * [email protected] ([email protected]) wrote: > I think I need to take a step backwards and ask something else before > moving forward. Good idea. > > Isn't that going to mail you the same WAL over and over again if you set > > it up as a cronjob..? Is that really what you want? > > I took a full backup when I started. Then I used type=diff. This creates > two archives every x hours (depending on the time period set in cron). WAL files are not "archives" in the sense that they are a complete database backup- they're just the write-ahead-logs from the ongoing running of PG. You need the actual data files as well as the WAL to perform a restore. > For example: > > -rw-r----- 1 postgres postgres 27145 Mar 11 00:00 > 0000000100000001000000B9-6f3902fe5c3bdebc3c1c124ec6821c7206e350da.gz > > -rw-r----- 1 postgres postgres 27126 Mar 11 00:00 > 0000000100000001000000BA-214e7142c6eda0a350577f6bd624c3db203e184f.gz > > Only one of these relates to the database I setup in PostgreSQL. I don't > know what the other one is but it might be related to the "default" > database 'postgres'. (This is just a guess). No, that's not how WAL works. Those are two WAL files and they're both generated as part of running the PG system- they are not specific to one database or another in PG. > Irrespective, I would only want a new archive file if something has > changed in the database. Yet I seem to always receive two new files every > x hours even if nothing has changed. These individuals files are not archives of the entire PG system or anything you can use to reconstruct a running PG system with just one file. > So I tried type-incr but that seems to produce the same results as type=diff. The difference between incremental and differential backups is documented at https://pgbackrest.org > I wonder what I am doing wrong? I'm afraid there are some pretty fundamental misunderstandings that you have about the PG WAL, PG backups, and what you can do with a PG cluster andf file-level backup/restore. Documentation about the WAL can be found here: https://www.postgresql.org/docs/current/wal-intro.html If what you're actually looking for is a single file that has a backup of your entire database, you can get that by running: pgbackrest backup --archive-copy and then do: pgbackrest restore --pg-path=/path/to/somewhere tar -czf pgdata.tar.gz /path/to/somewhere or so. Check the pgbackrest docs for the specific command syntax and such. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2020-03-11 00:37 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-03-10 10:35 pgbackrest creating new directories (messing up cron jobs). [email protected] 2020-03-10 13:44 ` Stephen Frost <[email protected]> 2020-03-10 22:23 ` [email protected] 2020-03-10 22:37 ` Stephen Frost <[email protected]> 2020-03-11 00:08 ` [email protected] 2020-03-11 00:37 ` Stephen Frost <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox