From: Fujii Masao <masao.fujii@oss.nttdata.com>
To: Bossart, Nathan <bossartn@amazon.com>
To: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: archive modules
Date: Sun, 7 Nov 2021 14:04:57 +0900
Message-ID: <db01f425-2689-276d-f81a-e8cd560816b2@oss.nttdata.com> (raw)
In-Reply-To: <F9F7CCA3-489D-479E-941F-84C5C1D1D3CB@amazon.com>
References: <668D2428-F73B-475E-87AE-F89D67942270@amazon.com>
<e437bf15-9917-7b78-f326-90b9c57ba8ae@oss.nttdata.com>
<F9F7CCA3-489D-479E-941F-84C5C1D1D3CB@amazon.com>
On 2021/11/03 0:03, Bossart, Nathan wrote:
> On 11/1/21, 9:44 PM, "Fujii Masao" <masao.fujii@oss.nttdata.com> wrote:
>> What is the main motivation of this patch? I was thinking that
>> it's for parallelizing WAL archiving. But as far as I read
>> the patch very briefly, WAL file name is still passed to
>> the archive callback function one by one.
>
> The main motivation is provide a way to archive without shelling out.
> This reduces the amount of overhead, which can improve archival rate
> significantly.
It's helpful if you share how much this approach reduces
the amount of overhead.
> It should also make it easier to archive more safely.
> For example, many of the common shell commands used for archiving
> won't fsync the data, but it isn't too hard to do so via C.
But probably we can do the same thing even by using the existing
shell interface? For example, we can implement and provide
the C program of the archive command that fsync's the file?
Users can just use it in archive_command.
> The
> current proposal doesn't introduce any extra infrastructure for
> batching or parallelism, but it is probably still possible. I would
> like to eventually add batching, but for now I'm only focused on
> introducing basic archive module support.
Understood. I agree that it's reasonable to implement them gradually.
>> Are you planning to extend this mechanism to other WAL
>> archiving-related commands like restore_command? I can imagine
>> that those who use archive library (rather than shell) would
>> like to use the same mechanism for WAL restore.
>
> I would like to do this eventually, but my current proposal is limited
> to archive_command.
Understood.
>> I think that it's worth adding this module into core
>> rather than handling it as test module. It provides very basic
>> WAL archiving feature, but (I guess) it's enough for some users.
>
> Do you think it should go into contrib?
Yes, at least for me..
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: masao.fujii@oss.nttdata.com, bossartn@amazon.com
Subject: Re: archive modules
In-Reply-To: <db01f425-2689-276d-f81a-e8cd560816b2@oss.nttdata.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox