From: Bossart, Nathan <bossartn@amazon.com>
To: David Steele <david@pgmasters.net>
To: Fujii Masao <masao.fujii@oss.nttdata.com>
To: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: archive modules
Date: Wed, 10 Nov 2021 18:22:40 +0000
Message-ID: <A30D8D33-8944-4898-BCA8-C77C18258247@amazon.com> (raw)
In-Reply-To: <53a180a7-bf1b-38c5-7fb4-4088dc022b98@pgmasters.net>
References: <668D2428-F73B-475E-87AE-F89D67942270@amazon.com>
<e437bf15-9917-7b78-f326-90b9c57ba8ae@oss.nttdata.com>
<F9F7CCA3-489D-479E-941F-84C5C1D1D3CB@amazon.com>
<db01f425-2689-276d-f81a-e8cd560816b2@oss.nttdata.com>
<53a180a7-bf1b-38c5-7fb4-4088dc022b98@pgmasters.net>
On 11/10/21, 8:10 AM, "David Steele" <david@pgmasters.net> wrote:
> On 11/7/21 1:04 AM, Fujii Masao wrote:
>> It's helpful if you share how much this approach reduces
>> the amount of overhead.
>
> FWIW we have a test for this in pgBackRest. Running
> `archive_command=pgbackrest archive-push ...` 1000 times via system()
> yields an average of 3ms per execution. pgBackRest reports ~1ms of time
> here so the system() overhead is ~2ms. These times are on my very fast
> workstation and in my experience servers are quite a bit slower.
In the previous thread [0], I noted a 50% speedup for a basic
archiving strategy that involved copying the file to a different
directory.
> I would prefer this module to be in core as our standard implementation
> and load by default in a vanilla install.
Hm. I think I disagree with putting it in contrib and with making it
the default archive library. The first reason is backward
compatibility. There has already been quite a bit of discussion about
this, and I don't see how we can get away with anything except for
maintaining the existing behavior for now. Maybe we could move to a
better default down the road, but I'm hesitant to press that issue too
much at the moment.
The second reason is that the basic_archive module has a couple of
deficiencies. For example, it doesn't handle inconvenient server
crashes well (e.g., after archiving but before we've renamed the
.ready file). A way to fix this might be to compare the archive file
with the to-be-archived file and to succeed if they are exactly the
same. Another problem is that there is no handling for multiple
servers using basic_archive to write WAL to the same location. This
is because basic_archive first copies data to a temporary file that is
always named "archtemp." This might be fixed by appending a random
string to the temporary file or by locking it somehow, but there are
still a few things left to figure out.
I think it'd be awesome to eventually fix all these issues in
basic_archive and to recommend it as a proper archiving strategy, but
I'm worried that this will introduce a significant amount of
complexity to this patch. I really only intended for basic_archive to
be used for testing and to demonstrate that it's possible use the
archive module infrastructure to do something useful. If folks really
want it in contrib, I'd at least add a big warning about the
aforementioned problems in its docs.
Nathan
[0] https://postgr.es/m/E9035E94-EC76-436E-B6C9-1C03FBD8EF54%40amazon.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bossartn@amazon.com, david@pgmasters.net, masao.fujii@oss.nttdata.com
Subject: Re: archive modules
In-Reply-To: <A30D8D33-8944-4898-BCA8-C77C18258247@amazon.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox