pg.ddx.io pgsql-docs@postgresql.org mailing list archive
help / color / mirror / Atom feedincoorect restore_command
9+ messages / 5 participants
[nested] [flat]
* incoorect restore_command
@ 2021-02-22 07:36 PG Doc comments form <noreply@postgresql.org>
2021-02-24 05:49 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 9+ messages in thread
From: PG Doc comments form @ 2021-02-22 07:36 UTC (permalink / raw)
To: pgsql-docs@lists.postgresql.org; +Cc: phil@philphonic.de
The following documentation comment has been logged on the website:
Page: https://www.postgresql.org/docs/12/continuous-archiving.html
Description:
Hi everyone,
I've been trying out saving and restoring compressed archive logs. The
restore_command stated in the docs at "25.3.6.2. Compressed Archive Logs"
('gunzip < /mnt/server/archivedir/%f > %p') did not work for me, because the
archive_command ('gzip < %p > /var/lib/pgsql/archive/%f') alters the
filename to %f.gz
I had to change the restore_command to 'gunzip <
/mnt/server/archivedir/%f.gz > %p'.
Phil
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: incoorect restore_command
2021-02-22 07:36 incoorect restore_command PG Doc comments form <noreply@postgresql.org>
@ 2021-02-24 05:49 ` Michael Paquier <michael@paquier.xyz>
2021-02-24 06:24 ` Re: incoorect restore_command Philipp Gramzow <phil@philphonic.de>
0 siblings, 1 reply; 9+ messages in thread
From: Michael Paquier @ 2021-02-24 05:49 UTC (permalink / raw)
To: phil@philphonic.de; pgsql-docs@lists.postgresql.org
On Mon, Feb 22, 2021 at 07:36:28AM +0000, PG Doc comments form wrote:
> I've been trying out saving and restoring compressed archive logs. The
> restore_command stated in the docs at "25.3.6.2. Compressed Archive Logs"
> ('gunzip < /mnt/server/archivedir/%f > %p') did not work for me, because the
> archive_command ('gzip < %p > /var/lib/pgsql/archive/%f') alters the
> filename to %f.gz
On which platform please? Using a pipe with gzip does not alter the
output file name where the data is pushed to.
> I had to change the restore_command to 'gunzip <
> /mnt/server/archivedir/%f.gz > %p'.
Now, I kind of agree that compressing a file and not using a proper
.gz extension for its name can be confusing. So what about the
attached to tweak both archive_command and restore_command in this
section of the docs?
--
Michael
Attachments:
doc-archive-gz.patch (text/x-diff, 827B) download
diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml
index 3c8aaed0b6..089102a599 100644
--- a/doc/src/sgml/backup.sgml
+++ b/doc/src/sgml/backup.sgml
@@ -1520,11 +1520,11 @@ tar -rf /var/lib/pgsql/backup.tar /var/lib/pgsql/archive/
If archive storage size is a concern, you can use
<application>gzip</application> to compress the archive files:
<programlisting>
-archive_command = 'gzip < %p > /var/lib/pgsql/archive/%f'
+archive_command = 'gzip < %p > /var/lib/pgsql/archive/%f.gz'
</programlisting>
You will then need to use <application>gunzip</application> during recovery:
<programlisting>
-restore_command = 'gunzip < /mnt/server/archivedir/%f > %p'
+restore_command = 'gunzip < /mnt/server/archivedir/%f.gz > %p'
</programlisting>
</para>
</sect3>
* Re: incoorect restore_command
2021-02-22 07:36 incoorect restore_command PG Doc comments form <noreply@postgresql.org>
2021-02-24 05:49 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
@ 2021-02-24 06:24 ` Philipp Gramzow <phil@philphonic.de>
2021-02-24 07:52 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
2021-02-26 16:54 ` Re: incoorect restore_command Stephen Frost <sfrost@snowman.net>
0 siblings, 2 replies; 9+ messages in thread
From: Philipp Gramzow @ 2021-02-24 06:24 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; pgsql-docs@lists.postgresql.org
Am 24.02.2021 um 06:49 schrieb Michael Paquier:
> On Mon, Feb 22, 2021 at 07:36:28AM +0000, PG Doc comments form wrote:
>> I've been trying out saving and restoring compressed archive logs. The
>> restore_command stated in the docs at "25.3.6.2. Compressed Archive Logs"
>> ('gunzip < /mnt/server/archivedir/%f > %p') did not work for me, because the
>> archive_command ('gzip < %p > /var/lib/pgsql/archive/%f') alters the
>> filename to %f.gz
> On which platform please? Using a pipe with gzip does not alter the
> output file name where the data is pushed to.
We're on RHEL 7. Thanks for pointing out my mistake, someone altered our
archive_command to use .gz extensions without me noticing it. Please
excuse the confusion.
>> I had to change the restore_command to 'gunzip <
>> /mnt/server/archivedir/%f.gz > %p'.
> Now, I kind of agree that compressing a file and not using a proper
> .gz extension for its name can be confusing. So what about the
> attached to tweak both archive_command and restore_command in this
> section of the docs?
> --
> Michael
I agree, using a proper extension would be more straightforward. I'm
sure that's the reason why someone changed our archive_command.
Phil
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: incoorect restore_command
2021-02-22 07:36 incoorect restore_command PG Doc comments form <noreply@postgresql.org>
2021-02-24 05:49 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
2021-02-24 06:24 ` Re: incoorect restore_command Philipp Gramzow <phil@philphonic.de>
@ 2021-02-24 07:52 ` Michael Paquier <michael@paquier.xyz>
2021-02-24 11:15 ` Re: incoorect restore_command Fujii Masao <masao.fujii@oss.nttdata.com>
1 sibling, 1 reply; 9+ messages in thread
From: Michael Paquier @ 2021-02-24 07:52 UTC (permalink / raw)
To: Philipp Gramzow <phil@philphonic.de>; +Cc: pgsql-docs@lists.postgresql.org
On Wed, Feb 24, 2021 at 07:24:17AM +0100, Philipp Gramzow wrote:
> I agree, using a proper extension would be more straightforward. I'm sure
> that's the reason why someone changed our archive_command.
Okay, let's do so in the docs then. Others may have comments to
offer, so I'll first wait a bit before applying my suggestion from
upthread.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: incoorect restore_command
2021-02-22 07:36 incoorect restore_command PG Doc comments form <noreply@postgresql.org>
2021-02-24 05:49 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
2021-02-24 06:24 ` Re: incoorect restore_command Philipp Gramzow <phil@philphonic.de>
2021-02-24 07:52 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
@ 2021-02-24 11:15 ` Fujii Masao <masao.fujii@oss.nttdata.com>
2021-02-25 06:54 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 9+ messages in thread
From: Fujii Masao @ 2021-02-24 11:15 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; Philipp Gramzow <phil@philphonic.de>; +Cc: pgsql-docs@lists.postgresql.org
On 2021/02/24 16:52, Michael Paquier wrote:
> On Wed, Feb 24, 2021 at 07:24:17AM +0100, Philipp Gramzow wrote:
>> I agree, using a proper extension would be more straightforward. I'm sure
>> that's the reason why someone changed our archive_command.
>
> Okay, let's do so in the docs then. Others may have comments to
> offer, so I'll first wait a bit before applying my suggestion from
> upthread.
I agree with this change.
But I have one question; why do those commands use different
archive directories? Isn't it better to use the same one?
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: incoorect restore_command
2021-02-22 07:36 incoorect restore_command PG Doc comments form <noreply@postgresql.org>
2021-02-24 05:49 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
2021-02-24 06:24 ` Re: incoorect restore_command Philipp Gramzow <phil@philphonic.de>
2021-02-24 07:52 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
2021-02-24 11:15 ` Re: incoorect restore_command Fujii Masao <masao.fujii@oss.nttdata.com>
@ 2021-02-25 06:54 ` Michael Paquier <michael@paquier.xyz>
2021-02-25 09:03 ` Re: incoorect restore_command Fujii Masao <masao.fujii@oss.nttdata.com>
0 siblings, 1 reply; 9+ messages in thread
From: Michael Paquier @ 2021-02-25 06:54 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@oss.nttdata.com>; +Cc: Philipp Gramzow <phil@philphonic.de>; pgsql-docs@lists.postgresql.org
On Wed, Feb 24, 2021 at 08:15:59PM +0900, Fujii Masao wrote:
> But I have one question; why do those commands use different
> archive directories? Isn't it better to use the same one?
storage.sgml uses /var/lib/pgsql/data for the location of the data,
and the archive path is a mix between /mnt/server/archivedir/ and
/var/lib/pgsql/archive/. However, the former is used for
pg_archivecleanup and in postgresql.conf.sample, so why not just using
/mnt/server/archivedir/ in backup.sgml? Please see the attached.
--
Michael
Attachments:
doc-archive-gz-v2.patch (text/x-diff, 2.0K) download
diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml
index 3c8aaed0b6..d8a60f7529 100644
--- a/doc/src/sgml/backup.sgml
+++ b/doc/src/sgml/backup.sgml
@@ -1484,7 +1484,7 @@ restore_command = 'cp /mnt/server/archivedir/%f %p'
<literal>on</literal>, and set up an <varname>archive_command</varname> that performs
archiving only when a <emphasis>switch file</emphasis> exists. For example:
<programlisting>
-archive_command = 'test ! -f /var/lib/pgsql/backup_in_progress || (test ! -f /var/lib/pgsql/archive/%f && cp %p /var/lib/pgsql/archive/%f)'
+archive_command = 'test ! -f /var/lib/pgsql/backup_in_progress || (test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f)'
</programlisting>
This command will perform archiving when
<filename>/var/lib/pgsql/backup_in_progress</filename> exists, and otherwise
@@ -1501,7 +1501,7 @@ psql -c "select pg_start_backup('hot_backup');"
tar -cf /var/lib/pgsql/backup.tar /var/lib/pgsql/data/
psql -c "select pg_stop_backup();"
rm /var/lib/pgsql/backup_in_progress
-tar -rf /var/lib/pgsql/backup.tar /var/lib/pgsql/archive/
+tar -rf /var/lib/pgsql/backup.tar /mnt/server/archivedir/
</programlisting>
The switch file <filename>/var/lib/pgsql/backup_in_progress</filename> is
created first, enabling archiving of completed WAL files to occur.
@@ -1520,11 +1520,11 @@ tar -rf /var/lib/pgsql/backup.tar /var/lib/pgsql/archive/
If archive storage size is a concern, you can use
<application>gzip</application> to compress the archive files:
<programlisting>
-archive_command = 'gzip < %p > /var/lib/pgsql/archive/%f'
+archive_command = 'gzip < %p > /mnt/server/archivedir/%f.gz'
</programlisting>
You will then need to use <application>gunzip</application> during recovery:
<programlisting>
-restore_command = 'gunzip < /mnt/server/archivedir/%f > %p'
+restore_command = 'gunzip < /mnt/server/archivedir/%f.gz > %p'
</programlisting>
</para>
</sect3>
* Re: incoorect restore_command
2021-02-22 07:36 incoorect restore_command PG Doc comments form <noreply@postgresql.org>
2021-02-24 05:49 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
2021-02-24 06:24 ` Re: incoorect restore_command Philipp Gramzow <phil@philphonic.de>
2021-02-24 07:52 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
2021-02-24 11:15 ` Re: incoorect restore_command Fujii Masao <masao.fujii@oss.nttdata.com>
2021-02-25 06:54 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
@ 2021-02-25 09:03 ` Fujii Masao <masao.fujii@oss.nttdata.com>
2021-02-26 05:42 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 9+ messages in thread
From: Fujii Masao @ 2021-02-25 09:03 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Philipp Gramzow <phil@philphonic.de>; pgsql-docs@lists.postgresql.org
On 2021/02/25 15:54, Michael Paquier wrote:
> On Wed, Feb 24, 2021 at 08:15:59PM +0900, Fujii Masao wrote:
>> But I have one question; why do those commands use different
>> archive directories? Isn't it better to use the same one?
>
> storage.sgml uses /var/lib/pgsql/data for the location of the data,
> and the archive path is a mix between /mnt/server/archivedir/ and
> /var/lib/pgsql/archive/. However, the former is used for
> pg_archivecleanup and in postgresql.conf.sample, so why not just using
> /mnt/server/archivedir/ in backup.sgml? Please see the attached.
-archive_command = 'test ! -f /var/lib/pgsql/backup_in_progress || (test ! -f /var/lib/pgsql/archive/%f && cp %p /var/lib/pgsql/archive/%f)'
+archive_command = 'test ! -f /var/lib/pgsql/backup_in_progress || (test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f)'
Regarding the section "Standalone Hot Backups", all the directories and
file seem to be placed under /var/lib/pgsql, so at least for me it looks a bit
strange to change only the path of archive directory. So I don't think that
we need to do this change.
-tar -rf /var/lib/pgsql/backup.tar /var/lib/pgsql/archive/
+tar -rf /var/lib/pgsql/backup.tar /mnt/server/archivedir/
Same as above.
-archive_command = 'gzip < %p > /var/lib/pgsql/archive/%f'
+archive_command = 'gzip < %p > /mnt/server/archivedir/%f.gz'
</programlisting>
You will then need to use <application>gunzip</application> during recovery:
<programlisting>
-restore_command = 'gunzip < /mnt/server/archivedir/%f > %p'
+restore_command = 'gunzip < /mnt/server/archivedir/%f.gz > %p'
LGTM. Thanks for the patch!
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: incoorect restore_command
2021-02-22 07:36 incoorect restore_command PG Doc comments form <noreply@postgresql.org>
2021-02-24 05:49 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
2021-02-24 06:24 ` Re: incoorect restore_command Philipp Gramzow <phil@philphonic.de>
2021-02-24 07:52 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
2021-02-24 11:15 ` Re: incoorect restore_command Fujii Masao <masao.fujii@oss.nttdata.com>
2021-02-25 06:54 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
2021-02-25 09:03 ` Re: incoorect restore_command Fujii Masao <masao.fujii@oss.nttdata.com>
@ 2021-02-26 05:42 ` Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 9+ messages in thread
From: Michael Paquier @ 2021-02-26 05:42 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@oss.nttdata.com>; +Cc: Philipp Gramzow <phil@philphonic.de>; pgsql-docs@lists.postgresql.org
On Thu, Feb 25, 2021 at 06:03:57PM +0900, Fujii Masao wrote:
> Regarding the section "Standalone Hot Backups", all the directories and
> file seem to be placed under /var/lib/pgsql, so at least for me it looks a bit
> strange to change only the path of archive directory. So I don't think that
> we need to do this change.
>
> -tar -rf /var/lib/pgsql/backup.tar /var/lib/pgsql/archive/
> +tar -rf /var/lib/pgsql/backup.tar /mnt/server/archivedir/
>
> Same as above.
Okay. I found the thing a bit inconsistent while looking at the whole
picture. Anyway, dropped those two parts.
> -archive_command = 'gzip < %p > /var/lib/pgsql/archive/%f'
> +archive_command = 'gzip < %p > /mnt/server/archivedir/%f.gz'
> </programlisting>
> You will then need to use <application>gunzip</application> during recovery:
> <programlisting>
> -restore_command = 'gunzip < /mnt/server/archivedir/%f > %p'
> +restore_command = 'gunzip < /mnt/server/archivedir/%f.gz > %p'
>
> LGTM. Thanks for the patch!
Thanks for the review. I have applied only this part, then.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: incoorect restore_command
2021-02-22 07:36 incoorect restore_command PG Doc comments form <noreply@postgresql.org>
2021-02-24 05:49 ` Re: incoorect restore_command Michael Paquier <michael@paquier.xyz>
2021-02-24 06:24 ` Re: incoorect restore_command Philipp Gramzow <phil@philphonic.de>
@ 2021-02-26 16:54 ` Stephen Frost <sfrost@snowman.net>
1 sibling, 0 replies; 9+ messages in thread
From: Stephen Frost @ 2021-02-26 16:54 UTC (permalink / raw)
To: Philipp Gramzow <phil@philphonic.de>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-docs@lists.postgresql.org
Greetings,
* Philipp Gramzow (phil@philphonic.de) wrote:
> Am 24.02.2021 um 06:49 schrieb Michael Paquier:
> >On Mon, Feb 22, 2021 at 07:36:28AM +0000, PG Doc comments form wrote:
> >>I've been trying out saving and restoring compressed archive logs. The
> >>restore_command stated in the docs at "25.3.6.2. Compressed Archive Logs"
> >>('gunzip < /mnt/server/archivedir/%f > %p') did not work for me, because the
> >>archive_command ('gzip < %p > /var/lib/pgsql/archive/%f') alters the
> >>filename to %f.gz
> >On which platform please? Using a pipe with gzip does not alter the
> >output file name where the data is pushed to.
> We're on RHEL 7. Thanks for pointing out my mistake, someone altered our
> archive_command to use .gz extensions without me noticing it. Please excuse
> the confusion.
Note that the simple commands in the documentation are really just to
give you an example of what an archive command would look like- they
shouldn't be used verbatim since they don't provide any guarantees that
the resulting compressed file has actually been written out to disk
(something that an archive command really should provide the guarantee
of- otherwise a crash will result in WAL files likely going missing and
therefore you'll lose the ability to do PITR).
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2021-02-26 16:54 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-02-22 07:36 incoorect restore_command PG Doc comments form <noreply@postgresql.org>
2021-02-24 05:49 ` Michael Paquier <michael@paquier.xyz>
2021-02-24 06:24 ` Philipp Gramzow <phil@philphonic.de>
2021-02-24 07:52 ` Michael Paquier <michael@paquier.xyz>
2021-02-24 11:15 ` Fujii Masao <masao.fujii@oss.nttdata.com>
2021-02-25 06:54 ` Michael Paquier <michael@paquier.xyz>
2021-02-25 09:03 ` Fujii Masao <masao.fujii@oss.nttdata.com>
2021-02-26 05:42 ` Michael Paquier <michael@paquier.xyz>
2021-02-26 16:54 ` Stephen Frost <sfrost@snowman.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox