Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jKTo6-0005le-DT for pgsql-hackers@arkaria.postgresql.org; Fri, 03 Apr 2020 21:24:54 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jKTo4-0005G2-Px for pgsql-hackers@arkaria.postgresql.org; Fri, 03 Apr 2020 21:24:52 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jKTo4-0005FA-EC for pgsql-hackers@lists.postgresql.org; Fri, 03 Apr 2020 21:24:52 +0000 Received: from mail-qk1-x741.google.com ([2607:f8b0:4864:20::741]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1jKTo1-0006wZ-5s for pgsql-hackers@postgresql.org; Fri, 03 Apr 2020 21:24:51 +0000 Received: by mail-qk1-x741.google.com with SMTP id l25so1642362qkk.3 for ; Fri, 03 Apr 2020 14:24:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=IJXjQrQEfxuG17KYlxNI/qoWxPsgIMJSJusv43kUcsU=; b=WdoBE02I1E68tGV8y/8BM/EcvcdRFkeqP27O5ObOXUWmhcZAmuoZuDQwZuzg54U+I+ mZGrFuV4wMFRlgUHECmBVQJnQMP8WtZvj/JxBpBGRqPp+ZFIXZPCb7XkK8hpDMaD7CKK iQR8yaAXWvDKl3Ton9qGmyoWG6EW58GSsAgM3HFBqbOEiadSiI4AUfBCAm+pulPTI/Wf KflMbSALjGYaTPoR8/QhZukyZF1Nz+kDCdAXPNSu6erJM1Nl1OeT72QC0HuK6h3q9v2O IZ3bcwIZUKp7/2EUYF97WY7+N1y5DsH6f5bee5A+147eHdLQQaWW/92GpxjCZzeGZNHI KGMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=IJXjQrQEfxuG17KYlxNI/qoWxPsgIMJSJusv43kUcsU=; b=p724NTl3kl3FBw0hlvJYr4ZBnAqGKd+svJqKpDefySDfUDgaOrHOBIPa2lvUnyZ/47 cxUPre+SnMQGrwOVRmC/xYcusSI69gzFcbs+KEB3J10HXYtm5yVIpQpQFG6bF9Q/fasb +i0yYHnGF0oeuTP6FyoPx9N3JcBv6l6ILXXRazuv7IoHm+MubG/Lub3hb3BKPJ94wkjU mDIsPtSKNkgz0+44Bwvypm8nNVbM+8y4i7wPWizWCcONSiN12fslU5UCQMJxn5pPZpxQ +xrsnBiVheGOcW0IZSINGLuiwb+HPca6I/CW1IneIKk3Vg8qdD64jHJLeKzvrj+Gz72u C5Kg== X-Gm-Message-State: AGi0PuaiO6w0nkw8hDisTIG7SZ76A6VAT0pnQ9cXUa7GCaAjetalnoal Jmt0xquxX22ReflHIesKYiOn8w== X-Google-Smtp-Source: APiQypJdvfEBivbXPtmqtQlpDBC0bg9wGlM4AynLH8C2JCyhl6me4H7cMyDixC0KaNURb//yxegP5g== X-Received: by 2002:a37:b041:: with SMTP id z62mr10806908qke.487.1585949088252; Fri, 03 Apr 2020 14:24:48 -0700 (PDT) Received: from pryzbyj (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id 85sm7390374qke.128.2020.04.03.14.24.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 03 Apr 2020 14:24:47 -0700 (PDT) Received: by pryzbyj (Postfix, from userid 1000) id 85579800A79; Fri, 3 Apr 2020 16:24:45 -0500 (CDT) Date: Fri, 3 Apr 2020 16:24:45 -0500 From: Justin Pryzby To: Robert Haas Cc: David Steele , Andres Freund , Noah Misch , Stephen Frost , Amit Kapila , Suraj Kharage , tushar , Rajkumar Raghuwanshi , Rushabh Lathia , Tels , Andrew Dunstan , PostgreSQL Hackers , Jeevan Chalke , vignesh C Subject: Re: backup manifests Message-ID: <20200403212445.GB12283@telsasoft.com> References: <20200402172318.3kvfrrewxmemzpia@alap3.anarazel.de> <20200402182346.6iffoadxu2hsbi2s@alap3.anarazel.de> <20200402194750.7ze74b4t6b6o4cui@alap3.anarazel.de> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="nFreZHaLTZJo0R7j" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --nFreZHaLTZJo0R7j Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Fri, Apr 03, 2020 at 03:22:23PM -0400, Robert Haas wrote: > I've pushed all the patches. I didn't manage to look at this in advance but have some doc fixes. word-diff: diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index 536de9a698..d84afb7b18 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -2586,7 +2586,7 @@ The commands accepted in replication mode are: and sent along with the backup. The manifest is a list of every file present in the backup with the exception of any WAL files that may be included. It also stores the size, last modification time, and [-an optional-]{+optionally a+} checksum for each file. A value of force-escape forces all filenames to be hex-encoded; otherwise, this type of encoding is performed only for files whose names are non-UTF8 octet sequences. @@ -2602,7 +2602,7 @@ The commands accepted in replication mode are: MANIFEST_CHECKSUMS Specifies the {+checksum+} algorithm that should be applied to each file included in the backup manifest. Currently, the available algorithms are NONE, CRC32C, SHA224, SHA256, diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index c778e061f3..922688e227 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -604,7 +604,7 @@ PostgreSQL documentation not contain any checksums. Otherwise, it will contain a checksum of each file in the backup using the specified algorithm. In addition, the manifest will always contain a SHA256 checksum of its own [-contents.-]{+content.+} The SHA algorithms are significantly more CPU-intensive than CRC32C, so selecting one of them may increase the time required to complete the backup. @@ -614,7 +614,7 @@ PostgreSQL documentation of each file for users who wish to verify that the backup has not been tampered with, while the CRC32C algorithm provides a checksum which is much faster to calculate and good at catching errors due to accidental changes but is not resistant to [-targeted-]{+malicious+} modifications. Note that, to be useful against an adversary who has access to the backup, the backup manifest would need to be stored securely elsewhere or otherwise verified not to have been modified since the backup was taken. diff --git a/doc/src/sgml/ref/pg_validatebackup.sgml b/doc/src/sgml/ref/pg_validatebackup.sgml index 19888dc196..748ac439a6 100644 --- a/doc/src/sgml/ref/pg_validatebackup.sgml +++ b/doc/src/sgml/ref/pg_validatebackup.sgml @@ -41,12 +41,12 @@ PostgreSQL documentation It is important to note that[-that-] the validation which is performed by pg_validatebackup does not and [-can not-]{+cannot+} include every check which will be performed by a running server when attempting to make use of the backup. Even if you use this tool, you should still perform test restores and verify that the resulting databases work as expected and that they[-appear to-] contain the correct data. However, pg_validatebackup can detect many problems that commonly occur due to storage problems or user error. @@ -73,7 +73,7 @@ PostgreSQL documentation a backup_manifest file in the target directory or about anything inside pg_wal, even though these files won't be listed in the backup manifest. Only files are checked; the presence or absence [-or-]{+of+} directories is not verified, except indirectly: if a directory is missing, any files it should have contained will necessarily also be missing. @@ -84,7 +84,7 @@ PostgreSQL documentation for any files for which the computed checksum does not match the checksum stored in the manifest. This step is not performed for any files which produced errors in the previous step, since they are already known to have problems. [-Also, files-]{+Files+} which were ignored in the previous step are also ignored in this step. @@ -123,7 +123,7 @@ PostgreSQL documentation Options The following command-line options control the [-behavior.-]{+behavior of this program.+} diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c index 3b18e733cd..aa72a6ff10 100644 --- a/src/backend/replication/basebackup.c +++ b/src/backend/replication/basebackup.c @@ -1148,7 +1148,7 @@ AddFileToManifest(manifest_info *manifest, const char *spcoid, } /* * Each file's entry [-need-]{+needs+} to be separated from any entry that follows by a * comma, but there's no comma before the first one or after the last one. * To make that work, adding a file to the manifest starts by terminating * the most recently added line, with a comma if appropriate, but does not -- Justin --nFreZHaLTZJo0R7j Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-docs-backup-manifests.patch" From 0bc8211dd325e8fec55eab4a3089ed1768563502 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 3 Apr 2020 16:17:28 -0500 Subject: [PATCH] docs: backup manifests commit 0d8c9c1210c44b36ec2efcb223a1dfbe897a3661 Author: Robert Haas --- doc/src/sgml/protocol.sgml | 4 ++-- doc/src/sgml/ref/pg_basebackup.sgml | 4 ++-- doc/src/sgml/ref/pg_validatebackup.sgml | 12 ++++++------ src/backend/replication/basebackup.c | 2 +- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index 536de9a698..d84afb7b18 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -2586,7 +2586,7 @@ The commands accepted in replication mode are: and sent along with the backup. The manifest is a list of every file present in the backup with the exception of any WAL files that may be included. It also stores the size, last modification time, and - an optional checksum for each file. + optionally a checksum for each file. A value of force-escape forces all filenames to be hex-encoded; otherwise, this type of encoding is performed only for files whose names are non-UTF8 octet sequences. @@ -2602,7 +2602,7 @@ The commands accepted in replication mode are: MANIFEST_CHECKSUMS - Specifies the algorithm that should be applied to each file included + Specifies the checksum algorithm that should be applied to each file included in the backup manifest. Currently, the available algorithms are NONE, CRC32C, SHA224, SHA256, diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index c778e061f3..922688e227 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -604,7 +604,7 @@ PostgreSQL documentation not contain any checksums. Otherwise, it will contain a checksum of each file in the backup using the specified algorithm. In addition, the manifest will always contain a SHA256 - checksum of its own contents. The SHA algorithms + checksum of its own content. The SHA algorithms are significantly more CPU-intensive than CRC32C, so selecting one of them may increase the time required to complete the backup. @@ -614,7 +614,7 @@ PostgreSQL documentation of each file for users who wish to verify that the backup has not been tampered with, while the CRC32C algorithm provides a checksum which is much faster to calculate and good at catching errors due to accidental - changes but is not resistant to targeted modifications. Note that, to + changes but is not resistant to malicious modifications. Note that, to be useful against an adversary who has access to the backup, the backup manifest would need to be stored securely elsewhere or otherwise verified not to have been modified since the backup was taken. diff --git a/doc/src/sgml/ref/pg_validatebackup.sgml b/doc/src/sgml/ref/pg_validatebackup.sgml index 19888dc196..748ac439a6 100644 --- a/doc/src/sgml/ref/pg_validatebackup.sgml +++ b/doc/src/sgml/ref/pg_validatebackup.sgml @@ -41,12 +41,12 @@ PostgreSQL documentation - It is important to note that that the validation which is performed by - pg_validatebackup does not and can not include + It is important to note that the validation which is performed by + pg_validatebackup does not and cannot include every check which will be performed by a running server when attempting to make use of the backup. Even if you use this tool, you should still perform test restores and verify that the resulting databases work as - expected and that they appear to contain the correct data. However, + expected and that they contain the correct data. However, pg_validatebackup can detect many problems that commonly occur due to storage problems or user error. @@ -73,7 +73,7 @@ PostgreSQL documentation a backup_manifest file in the target directory or about anything inside pg_wal, even though these files won't be listed in the backup manifest. Only files are checked; - the presence or absence or directories is not verified, except + the presence or absence of directories is not verified, except indirectly: if a directory is missing, any files it should have contained will necessarily also be missing. @@ -84,7 +84,7 @@ PostgreSQL documentation for any files for which the computed checksum does not match the checksum stored in the manifest. This step is not performed for any files which produced errors in the previous step, since they are already known - to have problems. Also, files which were ignored in the previous step are + to have problems. Files which were ignored in the previous step are also ignored in this step. @@ -123,7 +123,7 @@ PostgreSQL documentation Options - The following command-line options control the behavior. + The following command-line options control the behavior of this program. diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c index 3b18e733cd..aa72a6ff10 100644 --- a/src/backend/replication/basebackup.c +++ b/src/backend/replication/basebackup.c @@ -1148,7 +1148,7 @@ AddFileToManifest(manifest_info *manifest, const char *spcoid, } /* - * Each file's entry need to be separated from any entry that follows by a + * Each file's entry needs to be separated from any entry that follows by a * comma, but there's no comma before the first one or after the last one. * To make that work, adding a file to the manifest starts by terminating * the most recently added line, with a comma if appropriate, but does not -- 2.17.0 --nFreZHaLTZJo0R7j--