From: Justin Pryzby <pryzby@telsasoft.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: David Steele <david@pgmasters.net>
Cc: Andres Freund <andres@anarazel.de>
Cc: Noah Misch <noah@leadboat.com>
Cc: Stephen Frost <sfrost@snowman.net>
Cc: Amit Kapila <amit.kapila16@gmail.com>
Cc: Suraj Kharage <suraj.kharage@enterprisedb.com>
Cc: tushar <tushar.ahuja@enterprisedb.com>
Cc: Rajkumar Raghuwanshi <rajkumar.raghuwanshi@enterprisedb.com>
Cc: Rushabh Lathia <rushabh.lathia@gmail.com>
Cc: Tels <nospam-pg-abuse@bloodgate.com>
Cc: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
Cc: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Cc: Jeevan Chalke <jeevan.chalke@enterprisedb.com>
Cc: vignesh C <vignesh21@gmail.com>
Subject: Re: backup manifests
Date: Fri, 3 Apr 2020 16:24:45 -0500
Message-ID: <20200403212445.GB12283@telsasoft.com> (raw)
In-Reply-To: <CA+TgmoaKhxPr0FWsEiZoXpMER44pMJ28L-Z7wcRrOKoGFwjiUQ@mail.gmail.com>
References: <CA+TgmobV+kW5cBCa8_LGvytdOH71gnpqDsmKBR7bghEFgR7Y1w@mail.gmail.com>
<CA+Tgmob+xDcvEUTznkvianyJzKK8nUM6bWfkrnZQkL-XEa3NEA@mail.gmail.com>
<20200402172318.3kvfrrewxmemzpia@alap3.anarazel.de>
<CA+Tgmoa2_8Ti9xez8wEH7Z_RJQv=cwPC=MeqUXcdVW=5-uMCig@mail.gmail.com>
<20200402182346.6iffoadxu2hsbi2s@alap3.anarazel.de>
<CA+TgmoboTL8cBt_P5Jx6RjL=Hk3Y8PbpGsh9KcH_MZuswVqdYA@mail.gmail.com>
<CA+TgmoaGoQM74CMJhqg3_M7osWgkT2xSxo9aGVu9E4Nt+8Z4sg@mail.gmail.com>
<20200402194750.7ze74b4t6b6o4cui@alap3.anarazel.de>
<e3101c23-26b7-6fa2-14a8-af24c96602b5@pgmasters.net>
<CA+TgmoaKhxPr0FWsEiZoXpMER44pMJ28L-Z7wcRrOKoGFwjiUQ@mail.gmail.com>
On Fri, Apr 03, 2020 at 03:22:23PM -0400, Robert Haas wrote:
> I've pushed all the patches.
I didn't manage to look at this in advance but have some doc fixes.
word-diff:
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgmlindex 536de9a698..d84afb7b18 100644--- a/doc/src/sgml/protocol.sgml+++ b/doc/src/sgml/protocol.sgml@@ -2586,7 +2586,7 @@ The commands accepted in replication mode are:
and sent along with the backup. The manifest is a list of every
file present in the backup with the exception of any WAL files that
may be included. It also stores the size, last modification time, and
[-an optional-]{+optionally a+} checksum for each file.
A value of <literal>force-escape</literal> forces all filenames
to be hex-encoded; otherwise, this type of encoding is performed only
for files whose names are non-UTF8 octet sequences.
@@ -2602,7 +2602,7 @@ The commands accepted in replication mode are:
<term><literal>MANIFEST_CHECKSUMS</literal></term>
<listitem>
<para>
Specifies the {+checksum+} algorithm that should be applied to each file included
in the backup manifest. Currently, the available
algorithms are <literal>NONE</literal>, <literal>CRC32C</literal>,
<literal>SHA224</literal>, <literal>SHA256</literal>,
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgmlindex c778e061f3..922688e227 100644--- a/doc/src/sgml/ref/pg_basebackup.sgml+++ b/doc/src/sgml/ref/pg_basebackup.sgml@@ -604,7 +604,7 @@ PostgreSQL documentation
not contain any checksums. Otherwise, it will contain a checksum
of each file in the backup using the specified algorithm. In addition,
the manifest will always contain a <literal>SHA256</literal>
checksum of its own [-contents.-]{+content.+} The <literal>SHA</literal> algorithms
are significantly more CPU-intensive than <literal>CRC32C</literal>,
so selecting one of them may increase the time required to complete
the backup.
@@ -614,7 +614,7 @@ PostgreSQL documentation
of each file for users who wish to verify that the backup has not been
tampered with, while the CRC32C algorithm provides a checksum which is
much faster to calculate and good at catching errors due to accidental
changes but is not resistant to [-targeted-]{+malicious+} modifications. Note that, to
be useful against an adversary who has access to the backup, the backup
manifest would need to be stored securely elsewhere or otherwise
verified not to have been modified since the backup was taken.
diff --git a/doc/src/sgml/ref/pg_validatebackup.sgml b/doc/src/sgml/ref/pg_validatebackup.sgmlindex 19888dc196..748ac439a6 100644--- a/doc/src/sgml/ref/pg_validatebackup.sgml+++ b/doc/src/sgml/ref/pg_validatebackup.sgml@@ -41,12 +41,12 @@ PostgreSQL documentation
</para>
<para>
It is important to note that[-that-] the validation which is performed by
<application>pg_validatebackup</application> does not and [-can not-]{+cannot+} include
every check which will be performed by a running server when attempting
to make use of the backup. Even if you use this tool, you should still
perform test restores and verify that the resulting databases work as
expected and that they[-appear to-] contain the correct data. However,
<application>pg_validatebackup</application> can detect many problems
that commonly occur due to storage problems or user error.
</para>
@@ -73,7 +73,7 @@ PostgreSQL documentation
a <literal>backup_manifest</literal> file in the target directory or
about anything inside <literal>pg_wal</literal>, even though these
files won't be listed in the backup manifest. Only files are checked;
the presence or absence [-or-]{+of+} directories is not verified, except
indirectly: if a directory is missing, any files it should have contained
will necessarily also be missing.
</para>
@@ -84,7 +84,7 @@ PostgreSQL documentation
for any files for which the computed checksum does not match the
checksum stored in the manifest. This step is not performed for any files
which produced errors in the previous step, since they are already known
to have problems. [-Also, files-]{+Files+} which were ignored in the previous step are
also ignored in this step.
</para>
@@ -123,7 +123,7 @@ PostgreSQL documentation
<title>Options</title>
<para>
The following command-line options control the [-behavior.-]{+behavior of this program.+}
<variablelist>
<varlistentry>
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.cindex 3b18e733cd..aa72a6ff10 100644--- a/src/backend/replication/basebackup.c+++ b/src/backend/replication/basebackup.c@@ -1148,7 +1148,7 @@ AddFileToManifest(manifest_info *manifest, const char *spcoid,
}
/*
* Each file's entry [-need-]{+needs+} to be separated from any entry that follows by a
* comma, but there's no comma before the first one or after the last one.
* To make that work, adding a file to the manifest starts by terminating
* the most recently added line, with a comma if appropriate, but does not
--
Justin
Attachments:
[text/x-diff] 0001-docs-backup-manifests.patch (6.3K, ../20200403212445.GB12283@telsasoft.com/2-0001-docs-backup-manifests.patch)
download | inline diff:
From 0bc8211dd325e8fec55eab4a3089ed1768563502 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 3 Apr 2020 16:17:28 -0500
Subject: [PATCH] docs: backup manifests
commit 0d8c9c1210c44b36ec2efcb223a1dfbe897a3661
Author: Robert Haas <rhaas@postgresql.org>
---
doc/src/sgml/protocol.sgml | 4 ++--
doc/src/sgml/ref/pg_basebackup.sgml | 4 ++--
doc/src/sgml/ref/pg_validatebackup.sgml | 12 ++++++------
src/backend/replication/basebackup.c | 2 +-
4 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgmlindex 536de9a698..d84afb7b18 100644--- a/doc/src/sgml/protocol.sgml+++ b/doc/src/sgml/protocol.sgml@@ -2586,7 +2586,7 @@ The commands accepted in replication mode are:
and sent along with the backup. The manifest is a list of every
file present in the backup with the exception of any WAL files that
may be included. It also stores the size, last modification time, and
- an optional checksum for each file.+ optionally a checksum for each file.
A value of <literal>force-escape</literal> forces all filenames
to be hex-encoded; otherwise, this type of encoding is performed only
for files whose names are non-UTF8 octet sequences.
@@ -2602,7 +2602,7 @@ The commands accepted in replication mode are:
<term><literal>MANIFEST_CHECKSUMS</literal></term>
<listitem>
<para>
- Specifies the algorithm that should be applied to each file included+ Specifies the checksum algorithm that should be applied to each file included
in the backup manifest. Currently, the available
algorithms are <literal>NONE</literal>, <literal>CRC32C</literal>,
<literal>SHA224</literal>, <literal>SHA256</literal>,
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgmlindex c778e061f3..922688e227 100644--- a/doc/src/sgml/ref/pg_basebackup.sgml+++ b/doc/src/sgml/ref/pg_basebackup.sgml@@ -604,7 +604,7 @@ PostgreSQL documentation
not contain any checksums. Otherwise, it will contain a checksum
of each file in the backup using the specified algorithm. In addition,
the manifest will always contain a <literal>SHA256</literal>
- checksum of its own contents. The <literal>SHA</literal> algorithms+ checksum of its own content. The <literal>SHA</literal> algorithms
are significantly more CPU-intensive than <literal>CRC32C</literal>,
so selecting one of them may increase the time required to complete
the backup.
@@ -614,7 +614,7 @@ PostgreSQL documentation
of each file for users who wish to verify that the backup has not been
tampered with, while the CRC32C algorithm provides a checksum which is
much faster to calculate and good at catching errors due to accidental
- changes but is not resistant to targeted modifications. Note that, to+ changes but is not resistant to malicious modifications. Note that, to
be useful against an adversary who has access to the backup, the backup
manifest would need to be stored securely elsewhere or otherwise
verified not to have been modified since the backup was taken.
diff --git a/doc/src/sgml/ref/pg_validatebackup.sgml b/doc/src/sgml/ref/pg_validatebackup.sgmlindex 19888dc196..748ac439a6 100644--- a/doc/src/sgml/ref/pg_validatebackup.sgml+++ b/doc/src/sgml/ref/pg_validatebackup.sgml@@ -41,12 +41,12 @@ PostgreSQL documentation
</para>
<para>
- It is important to note that that the validation which is performed by- <application>pg_validatebackup</application> does not and can not include+ It is important to note that the validation which is performed by+ <application>pg_validatebackup</application> does not and cannot include
every check which will be performed by a running server when attempting
to make use of the backup. Even if you use this tool, you should still
perform test restores and verify that the resulting databases work as
- expected and that they appear to contain the correct data. However,+ expected and that they contain the correct data. However,
<application>pg_validatebackup</application> can detect many problems
that commonly occur due to storage problems or user error.
</para>
@@ -73,7 +73,7 @@ PostgreSQL documentation
a <literal>backup_manifest</literal> file in the target directory or
about anything inside <literal>pg_wal</literal>, even though these
files won't be listed in the backup manifest. Only files are checked;
- the presence or absence or directories is not verified, except+ the presence or absence of directories is not verified, except
indirectly: if a directory is missing, any files it should have contained
will necessarily also be missing.
</para>
@@ -84,7 +84,7 @@ PostgreSQL documentation
for any files for which the computed checksum does not match the
checksum stored in the manifest. This step is not performed for any files
which produced errors in the previous step, since they are already known
- to have problems. Also, files which were ignored in the previous step are+ to have problems. Files which were ignored in the previous step are
also ignored in this step.
</para>
@@ -123,7 +123,7 @@ PostgreSQL documentation
<title>Options</title>
<para>
- The following command-line options control the behavior.+ The following command-line options control the behavior of this program.
<variablelist>
<varlistentry>
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.cindex 3b18e733cd..aa72a6ff10 100644--- a/src/backend/replication/basebackup.c+++ b/src/backend/replication/basebackup.c@@ -1148,7 +1148,7 @@ AddFileToManifest(manifest_info *manifest, const char *spcoid,
}
/*
- * Each file's entry need to be separated from any entry that follows by a+ * Each file's entry needs to be separated from any entry that follows by a
* comma, but there's no comma before the first one or after the last one.
* To make that work, adding a file to the manifest starts by terminating
* the most recently added line, with a comma if appropriate, but does not
--
2.17.0
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: pryzby@telsasoft.com, robertmhaas@gmail.com, david@pgmasters.net, andres@anarazel.de, noah@leadboat.com, sfrost@snowman.net, amit.kapila16@gmail.com, suraj.kharage@enterprisedb.com, tushar.ahuja@enterprisedb.com, rajkumar.raghuwanshi@enterprisedb.com, rushabh.lathia@gmail.com, nospam-pg-abuse@bloodgate.com, andrew.dunstan@2ndquadrant.com, jeevan.chalke@enterprisedb.com, vignesh21@gmail.com
Subject: Re: backup manifests
In-Reply-To: <20200403212445.GB12283@telsasoft.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox