From: Nathan Bossart <nathandbossart@gmail.com>
To: David Steele <david@pgmasters.net>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Fujii Masao <masao.fujii@oss.nttdata.com>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: remove more archiving overhead
Date: Thu, 7 Jul 2022 10:51:42 -0700
Message-ID: <20220707175142.GA2254092@nathanxps13> (raw)
In-Reply-To: <20220707161825.GA2205004@nathanxps13>
References: <20220222011948.GA3850532@nathanxps13>
<20220222173711.GA3852671@nathanxps13>
<20220222195229.GA42988@nathanxps13>
<20220224.141349.1809640551882183816.horikyota.ntt@gmail.com>
<20220224175553.GA660690@nathanxps13>
<20220407222327.GA1363808@nathanxps13>
<768e6cb3-256d-9c0b-1797-62420ffca7ae@oss.nttdata.com>
<CA+TgmoaeTe4oUT39A4nt=8LD6UJ5u0vcmGc5+Aksn-4oKRb8-w@mail.gmail.com>
<17147b74-5465-9a7a-ae27-305d1200482c@pgmasters.net>
<20220707161825.GA2205004@nathanxps13>
On Thu, Jul 07, 2022 at 09:18:25AM -0700, Nathan Bossart wrote:
> On Thu, Jul 07, 2022 at 10:46:23AM -0400, David Steele wrote:
>> On 7/7/22 10:37, Robert Haas wrote:
>>> I don't object, but I just started to wonder whether the need to
>>> handle re-archiving of the same file cleanly is as well-documented as
>>> it ought to be.
>>
>> +1, but I don't think that needs to stand in the way of this patch, which
>> looks sensible to me as-is. I think that's what you meant, but just wanted
>> to be sure.
>
> Yeah, this seems like something that should be documented. I can pick this
> up. I believe this is an existing problem, but this patch could make it
> more likely.
Here is a first try at documenting this. I'm not thrilled about the
placement, since it feels a bit buried in the backup docs, but this is
where this sort of thing lives today. It also seems odd to stress the
importance of avoiding overwriting pre-existing archives in case multiple
servers are archiving to the same place while only offering solutions with
obvious race conditions. Even basic_archive is subject to this now that
durable_rename_excl() no longer exists. Perhaps we should make a note of
that, too.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.comAttachments:v1-0001-add-note-about-re-archiving-in-docs.patch (text/x-diff, 2.2K) download
From 2b563823ca8dc57ea386832ea0b8c4c472645117 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathandbossart@gmail.com>
Date: Thu, 7 Jul 2022 10:43:30 -0700
Subject: [PATCH v1 1/1] add note about re-archiving in docs
---
doc/src/sgml/backup.sgml | 22 ++++++++++++++++++----
1 file changed, 18 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgmlindex 73a774d3d7..dd1c360455 100644--- a/doc/src/sgml/backup.sgml+++ b/doc/src/sgml/backup.sgml@@ -685,10 +685,24 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0
</para>
<para>
- It is advisable to test your proposed archive library to ensure that it- indeed does not overwrite an existing file, <emphasis>and that it returns- <literal>false</literal> in this case</emphasis>.- The example command above for Unix ensures this by including a separate+ In rare cases, <productname>PostgreSQL</productname> may attempt to+ re-archive a WAL file that was previously archived. For example, if the+ system crashes before the server makes a durable record of archival success,+ the server will attempt to archive the file again after restarting (provided+ archiving is still enabled). It is advisable to test your proposed archive+ library to ensure that it indeed does not overwrite an existing file. When+ a pre-existing file is encountered, the archive library may return+ <literal>true</literal> if the WAL file has identical contents to the+ pre-existing archive. Alternatively, the archive library can return+ <literal>false</literal> anytime a pre-existing file is encountered, but+ this will require manual action by an administrator to resolve. If a+ pre-existing file contains different contents than the WAL file being+ archived, the archive library <emphasis>must</emphasis> return false.+ </para>++ <para>+ The example command above for Unix avoids overwriting a pre-existing archive+ by including a separate
<command>test</command> step. On some Unix platforms, <command>cp</command> has
switches such as <option>-i</option> that can be used to do the same thing
less verbosely, but you should not rely on these without verifying that
--
2.25.1
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: nathandbossart@gmail.com, david@pgmasters.net, robertmhaas@gmail.com, masao.fujii@oss.nttdata.com, horikyota.ntt@gmail.com
Subject: Re: remove more archiving overhead
In-Reply-To: <20220707175142.GA2254092@nathanxps13>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox