From: Bossart, Nathan <bossartn@amazon.com>
To: Andres Freund <andres@anarazel.de>
Cc: alvherre@alvh.no-ip.org <alvherre@alvh.no-ip.org>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: x4mmm@yandex-team.ru <x4mmm@yandex-team.ru>
Cc: a.lubennikova@postgrespro.ru <a.lubennikova@postgrespro.ru>
Cc: hlinnaka@iki.fi <hlinnaka@iki.fi>
Cc: matsumura.ryo@fujitsu.com <matsumura.ryo@fujitsu.com>
Cc: masao.fujii@gmail.com <masao.fujii@gmail.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: archive status ".ready" files may be created too early
Date: Tue, 31 Aug 2021 23:31:15 +0000
Message-ID: <83EB9B6A-A6B6-40AB-8B70-0F536CB54BDA@amazon.com> (raw)
In-Reply-To: <20210831202929.3f3jzlzowis4g7gu@alap3.anarazel.de>
References: <20210829023722.cz6nbehyupp3hemk@alap3.anarazel.de>
<202108301951.x66z53s4lq6p@alvherre.pgsql>
<20210830210529.bd7wmqodt747xa2y@alap3.anarazel.de>
<C6234E05-6391-485F-92C3-10F7EE4D9612@amazon.com>
<20210831023839.7brycllwyyapbrfv@alap3.anarazel.de>
<282F96BA-948C-41DD-8A68-3E5ACE4D92C5@amazon.com>
<20210831074338.mt7hpaveuyvb723d@alap3.anarazel.de>
<21D63383-9336-4B64-BF12-BFE33563D7A2@amazon.com>
<20210831171836.ww43agvczffdultp@alap3.anarazel.de>
<20DC0D7B-E2DE-46BF-9DE4-D4D53A37E97D@amazon.com>
<20210831202929.3f3jzlzowis4g7gu@alap3.anarazel.de>
On 8/31/21, 1:30 PM, "Andres Freund" <andres@anarazel.de> wrote:
> On 2021-08-31 18:09:36 +0000, Bossart, Nathan wrote:
>> What appears to happen in this case is that bgwriter eventually creates a
>> xl_running_xacts record and nudges walwriter to flush it to disk, at which
>> point the .ready file(s) will be created. That's admittedly a bit fragile.
>
> That's not guaranteed to happen. If e.g. the partial record is a checkpoint or
> a xl_running_xacts, we'll not trigger further WAL writes in the background,
> unless autovacuum ends up doing something.
Right. Per the attached patch, a simple way to handle that could be
to teach XLogBackgroundFlush() to flush to the "earliest" segment
boundary if it doesn't find anything else to do. I think you could
still miss creating a .ready file for the previous segment in single-
user mode, though.
Nathan
Attachments:flush-to-boundary.patch (application/octet-stream, 1.3K) downloaddiff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex 24165ab03e..3e74bec42d 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -3127,6 +3127,16 @@ XLogBackgroundFlush(void)
flexible = false; /* ensure it all gets written */
}
+ /* if we have flushed that far, consider segment boundaries */+ if (WriteRqst.Write <= LogwrtResult.Flush)+ {+ SpinLockAcquire(&XLogCtl->segtrack_lck);+ if (XLogCtl->earliestSegBoundary != MaxXLogSegNo)+ WriteRqst.Write = XLogCtl->earliestSegBoundaryEndPtr;+ SpinLockRelease(&XLogCtl->segtrack_lck);+ Assert(!flexible);+ }+
/*
* If already known flushed, we're done. Just need to check if we are
* holding an open file handle to a logfile that's no longer in use,
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.cindex 6a1e16edc2..8ca0d8e616 100644--- a/src/backend/postmaster/walwriter.c+++ b/src/backend/postmaster/walwriter.c@@ -253,7 +253,8 @@ WalWriterMain(void)
* here to handle a race condition where WAL is flushed to disk prior
* to registering the segment boundary.
*/
- NotifySegmentsReadyForArchive(GetFlushRecPtr());+ if (XLogArchivingActive())+ NotifySegmentsReadyForArchive(GetFlushRecPtr());
/*
* Do what we're here for; then, if XLogBackgroundFlush() found useful
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bossartn@amazon.com, andres@anarazel.de, alvherre@alvh.no-ip.org, robertmhaas@gmail.com, horikyota.ntt@gmail.com, x4mmm@yandex-team.ru, a.lubennikova@postgrespro.ru, hlinnaka@iki.fi, matsumura.ryo@fujitsu.com, masao.fujii@gmail.com
Subject: Re: archive status ".ready" files may be created too early
In-Reply-To: <83EB9B6A-A6B6-40AB-8B70-0F536CB54BDA@amazon.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox