From: Bossart, Nathan <bossartn@amazon.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
To: masao.fujii@gmail.com <masao.fujii@gmail.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: archive status ".ready" files may be created too early
Date: Wed, 18 Dec 2019 16:34:05 +0000
Message-ID: <335D05B1-B5FB-48F6-A040-AB9AB4465910@amazon.com> (raw)
In-Reply-To: <20191218.173658.4136447812388306.horikyota.ntt@gmail.com>
References: <CBDDFA01-6E40-46BB-9F98-9340F4379505@amazon.com>
<CAHGQGwHMQW7HzG6-b+mPYduPsm2r3YmLDnbT1bjRiZAPezZEWA@mail.gmail.com>
<20191218.173658.4136447812388306.horikyota.ntt@gmail.com>
On 12/17/19, 2:26 AM, "Kyotaro Horiguchi" <horikyota.ntt@gmail.com> wrote:
> But I think I found a corner case where the patch doesn't work. As I
> mentioned in another message, if WAL buffer was full,
> AdvanceXLInsertBuffer calls XLogWrite to write out the victim buffer
> regardless whether the last record in the page was the first half of a
> continuation record. XLogWrite can mark the segment as .ready even
> with the patch.
>
> Is that correct? And do you think the corner case is worth amending?
I certainly think it is worth trying to prevent potential WAL archive
corruption in known corner cases. Your comment highlights a potential
shortcoming of my patch. AFAICT there is no guarantee that
XLogWrite() is called with a complete WAL record. Even if that
assumption is true at the moment, it might not hold up over time.
> If so, we could amend also that case by marking the last segment as
> .ready when XLogWrite writes the first bytes of the next segment. (As
> the further corner-case, it still doesn't work if a contination record
> spans over trhee or more segments.. But I don't (or want not to) think
> we don't need to consider that case..)
I'm working on a new version of the patch that will actually look at
the WAL page metadata to determine when it is safe to mark a segment
as ready for archival. It seems relatively easy to figure out whether
a page is the last one for the current WAL record.
Nathan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bossartn@amazon.com, horikyota.ntt@gmail.com, masao.fujii@gmail.com
Subject: Re: archive status ".ready" files may be created too early
In-Reply-To: <335D05B1-B5FB-48F6-A040-AB9AB4465910@amazon.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox