Date: Thu, 7 Jul 2022 10:51:42 -0700
From: Nathan Bossart <nathandbossart@gmail.com>
To: David Steele <david@pgmasters.net>
Cc: Robert Haas <robertmhaas@gmail.com>,
	Fujii Masao <masao.fujii@oss.nttdata.com>,
	Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: remove more archiving overhead
Message-ID: <20220707175142.GA2254092@nathanxps13>
References: <20220222011948.GA3850532@nathanxps13>
 <20220222173711.GA3852671@nathanxps13>
 <20220222195229.GA42988@nathanxps13>
 <20220224.141349.1809640551882183816.horikyota.ntt@gmail.com>
 <20220224175553.GA660690@nathanxps13>
 <20220407222327.GA1363808@nathanxps13>
 <768e6cb3-256d-9c0b-1797-62420ffca7ae@oss.nttdata.com>
 <CA+TgmoaeTe4oUT39A4nt=8LD6UJ5u0vcmGc5+Aksn-4oKRb8-w@mail.gmail.com>
 <17147b74-5465-9a7a-ae27-305d1200482c@pgmasters.net>
 <20220707161825.GA2205004@nathanxps13>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="SLDf9lqlvOQaIe6s"
Content-Disposition: inline
In-Reply-To: <20220707161825.GA2205004@nathanxps13>
Archived-At: 
 <https://www.postgresql.org/message-id/20220707175142.GA2254092%40nathanxps13>
Precedence: bulk


--SLDf9lqlvOQaIe6s
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Thu, Jul 07, 2022 at 09:18:25AM -0700, Nathan Bossart wrote:
> On Thu, Jul 07, 2022 at 10:46:23AM -0400, David Steele wrote:
>> On 7/7/22 10:37, Robert Haas wrote:
>>> I don't object, but I just started to wonder whether the need to
>>> handle re-archiving of the same file cleanly is as well-documented as
>>> it ought to be.
>> 
>> +1, but I don't think that needs to stand in the way of this patch, which
>> looks sensible to me as-is. I think that's what you meant, but just wanted
>> to be sure.
> 
> Yeah, this seems like something that should be documented.  I can pick this
> up.  I believe this is an existing problem, but this patch could make it
> more likely.

Here is a first try at documenting this.  I'm not thrilled about the
placement, since it feels a bit buried in the backup docs, but this is
where this sort of thing lives today.  It also seems odd to stress the
importance of avoiding overwriting pre-existing archives in case multiple
servers are archiving to the same place while only offering solutions with
obvious race conditions.  Even basic_archive is subject to this now that
durable_rename_excl() no longer exists.  Perhaps we should make a note of
that, too.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

--SLDf9lqlvOQaIe6s
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
	filename="v1-0001-add-note-about-re-archiving-in-docs.patch"

From 2b563823ca8dc57ea386832ea0b8c4c472645117 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathandbossart@gmail.com>
Date: Thu, 7 Jul 2022 10:43:30 -0700
Subject: [PATCH v1 1/1] add note about re-archiving in docs

---
 doc/src/sgml/backup.sgml | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)

diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml
index 73a774d3d7..dd1c360455 100644
--- a/doc/src/sgml/backup.sgml
+++ b/doc/src/sgml/backup.sgml
@@ -685,10 +685,24 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 &amp;&amp; cp pg_wal/0
    </para>
 
    <para>
-    It is advisable to test your proposed archive library to ensure that it
-    indeed does not overwrite an existing file, <emphasis>and that it returns
-    <literal>false</literal> in this case</emphasis>.
-    The example command above for Unix ensures this by including a separate
+    In rare cases, <productname>PostgreSQL</productname> may attempt to
+    re-archive a WAL file that was previously archived.  For example, if the
+    system crashes before the server makes a durable record of archival success,
+    the server will attempt to archive the file again after restarting (provided
+    archiving is still enabled).  It is advisable to test your proposed archive
+    library to ensure that it indeed does not overwrite an existing file.  When
+    a pre-existing file is encountered, the archive library may return
+    <literal>true</literal> if the WAL file has identical contents to the
+    pre-existing archive.  Alternatively, the archive library can return
+    <literal>false</literal> anytime a pre-existing file is encountered, but
+    this will require manual action by an administrator to resolve.  If a
+    pre-existing file contains different contents than the WAL file being
+    archived, the archive library <emphasis>must</emphasis> return false.
+   </para>
+
+   <para>
+    The example command above for Unix avoids overwriting a pre-existing archive
+    by including a separate
     <command>test</command> step.  On some Unix platforms, <command>cp</command> has
     switches such as <option>-i</option> that can be used to do the same thing
     less verbosely, but you should not rely on these without verifying that
-- 
2.25.1


--SLDf9lqlvOQaIe6s--