From: Bossart, Nathan <bossartn@amazon.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
To: masao.fujii@gmail.com <masao.fujii@gmail.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: archive status ".ready" files may be created too early
Date: Sat, 21 Dec 2019 01:18:24 +0000
Message-ID: <E43070BF-0136-41E3-BA07-F513F5ADE16C@amazon.com> (raw)
In-Reply-To: <335D05B1-B5FB-48F6-A040-AB9AB4465910@amazon.com>
References: <CBDDFA01-6E40-46BB-9F98-9340F4379505@amazon.com>
<CAHGQGwHMQW7HzG6-b+mPYduPsm2r3YmLDnbT1bjRiZAPezZEWA@mail.gmail.com>
<20191218.173658.4136447812388306.horikyota.ntt@gmail.com>
<335D05B1-B5FB-48F6-A040-AB9AB4465910@amazon.com>
On 12/18/19, 8:34 AM, "Bossart, Nathan" <bossartn@amazon.com> wrote:
> On 12/17/19, 2:26 AM, "Kyotaro Horiguchi" <horikyota.ntt@gmail.com> wrote:
>> If so, we could amend also that case by marking the last segment as
>> .ready when XLogWrite writes the first bytes of the next segment. (As
>> the further corner-case, it still doesn't work if a contination record
>> spans over trhee or more segments.. But I don't (or want not to) think
>> we don't need to consider that case..)
>
> I'm working on a new version of the patch that will actually look at
> the WAL page metadata to determine when it is safe to mark a segment
> as ready for archival. It seems relatively easy to figure out whether
> a page is the last one for the current WAL record.
I stand corrected. My attempts to add logic to check the WAL records
added quite a bit more complexity than seemed reasonable to maintain
in this code path. For example, I didn't anticipate things like
XLOG_SWITCH records.
I am still concerned about the corner case you noted, but I have yet
to find a practical way to handle it. You suggested waiting until
writing the first bytes of the next segment before marking a segment
as ready, but I'm not sure that fixes this problem either, and I
wonder if it could result in waiting arbitrarily long before creating
a ".ready" file in some cases. Perhaps I am misunderstanding your
suggestion.
Another thing I noticed is that any changes in this area could impact
archive_timeout. If we reset the archive_timeout timer when we mark
the segments ready, we could force WAL switches more often. If we do
not move the timer logic, we could be resetting it before the file is
ready for the archiver. However, these differences might be subtle
enough to be okay.
Nathan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bossartn@amazon.com, horikyota.ntt@gmail.com, masao.fujii@gmail.com
Subject: Re: archive status ".ready" files may be created too early
In-Reply-To: <E43070BF-0136-41E3-BA07-F513F5ADE16C@amazon.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox