From: Bossart, Nathan <bossartn@amazon.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: x4mmm@yandex-team.ru <x4mmm@yandex-team.ru>
Cc: a.lubennikova@postgrespro.ru <a.lubennikova@postgrespro.ru>
Cc: hlinnaka@iki.fi <hlinnaka@iki.fi>
Cc: matsumura.ryo@fujitsu.com <matsumura.ryo@fujitsu.com>
Cc: masao.fujii@gmail.com <masao.fujii@gmail.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: archive status ".ready" files may be created too early
Date: Mon, 15 Mar 2021 16:34:29 +0000
Message-ID: <E63E5670-6CC3-4B09-9686-A77CF94FE4A8@amazon.com> (raw)
In-Reply-To: <20210127.113500.1720002622159938202.horikyota.ntt@gmail.com>
References: <20201218.144205.14089619073535731.horikyota.ntt@gmail.com>
<60C21920-AB0D-43AF-BDB9-98E70153B654@yandex-team.ru>
<68120830-3A34-4C4F-942F-6739DAA664CF@amazon.com>
<20210127.113500.1720002622159938202.horikyota.ntt@gmail.com>
On 2/18/21, 4:10 PM, "Bossart, Nathan" <bossartn@amazon.com> wrote:
> Alright, I've attached a new patch set for this.
>
> 0001 is similar to the last patch I sent in this thread, although it
> contains a few fixes. The main difference is that we no longer
> initialize lastNotifiedSeg in StartupXLOG(). Instead, we initialize
> it in XLogWrite() where we previously were creating the archive status
> files. This ensures that standby servers do not create many
> unnecessary archive status files after promotion.
>
> 0002 adds logic for persisting the last notified segment through
> crashes. This is needed because a poorly-timed crash could otherwise
> cause us to skip marking segments as ready-for-archival altogether.
> This file is only used for primary servers, as there exists a separate
> code path for marking segments as ready-for-archive for standbys.
>
> I considered attempting to prevent this bug from affecting standby
> servers by withholding WAL for a segment until the previous segment
> has been marked ready-for-archival. However, that would require us to
> track record boundaries even with archiving turned off. Also, my
> patch relied on the assumption that the flush pointer advances along
> record boundaries except for records that span multiple segments.
> This assumption is likely not always true, and even if it is, it seems
> pretty fragile. Furthermore, I suspect that there are still problems
> with standbys since the code path responsible for creating archive
> status files on standbys has even less context about the WAL record
> boundaries. IMO patches 0001 and 0002 should be the focus for now,
> and related bugs for standby servers should be picked up in a new
> thread.
>
> I ended up not touching archive_timeout at all. The documentation for
> this parameter seems to be written ambiguously enough such that any
> small differences in behavior with these patches is still acceptable.
> I don't expect that users will see much change. In the worst case,
> the timer for archive_timeout may get reset a bit before the segment's
> archive status file is created.
I've attached a set of rebased patches.
Nathan
Attachments:download all (2 files, .tar.gz)v2-0001-Avoid-creating-archive-status-.ready-files-too-ea.patch (application/octet-stream, 13.7K) download
From e94b83986ae03a49e6259a3944736d9933d7b2bd Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Wed, 17 Feb 2021 00:29:36 +0000
Subject: [PATCH v2 1/2] Avoid creating archive status ".ready" files too
early.
WAL records may span multiple segments, but XLogWrite() does not
wait for the entire record to be written out to disk before
creating archive status files. Instead, as soon as the last WAL
page of the segment is written, the archive status file will be
created. If PostgreSQL crashes before it is able to write the rest
of the record, it will end up reusing segments that have already
been marked as ready-for-archival. However, the archiver process
may have already processed the old version of the segment, so the
wrong version of the segment may be backed-up. This backed-up
segment will cause operations such as point-in-time restores to
fail.
To fix this, we keep track of records that span across segments and
ensure that segments are only marked ready-for-archival once such
records have been completely written to disk.
---
src/backend/access/transam/xlog.c | 272 ++++++++++++++++++++++++++++++-
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/include/access/xlogdefs.h | 3 +
3 files changed, 269 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex f4d1ce5dea..893b9ce951 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -531,6 +531,13 @@ typedef enum ExclusiveBackupState
*/
static SessionBackupState sessionBackupState = SESSION_BACKUP_NONE;
+/* entries for RecordBoundaryMap, used to mark segments ready for archival */+typedef struct RecordBoundaryEntry+{+ XLogSegNo seg; /* must be first */+ XLogRecPtr pos;+} RecordBoundaryEntry;+
/*
* Shared state data for WAL insertion.
*/
@@ -732,6 +739,12 @@ typedef struct XLogCtlData
*/
XLogRecPtr lastFpwDisableRecPtr;
+ /*+ * The last segment we've marked ready for archival. Protected by info_lck.+ * This value should only be updated while holding ArchNotifyLock.+ */+ XLogSegNo lastNotifiedSeg;+
slock_t info_lck; /* locks shared variables shown above */
} XLogCtlData;
@@ -745,6 +758,12 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
*/
static ControlFileData *ControlFile = NULL;
+/*+ * Record boundary map, used for marking segments as ready for archival.+ * Protected by ArchNotifyLock.+ */+static HTAB *RecordBoundaryMap = NULL;+
/*
* Calculate the amount of space left on the page after 'endptr'. Beware
* multiple evaluation!
@@ -975,6 +994,12 @@ static XLogRecPtr XLogBytePosToRecPtr(uint64 bytepos);
static XLogRecPtr XLogBytePosToEndRecPtr(uint64 bytepos);
static uint64 XLogRecPtrToBytePos(XLogRecPtr ptr);
static void checkXLogConsistency(XLogReaderState *record);
+static void RegisterRecordBoundaryEntry(XLogSegNo seg, XLogRecPtr pos);+static void NotifySegmentsReadyForArchive(void);+static XLogSegNo GetLastNotifiedSegment(void);+static void SetLastNotifiedSegment(XLogSegNo seg);+static XLogSegNo GetLatestRecordBoundarySegment(void);+static void RemoveRecordBoundariesUpTo(XLogSegNo seg);
static void WALInsertLockAcquire(void);
static void WALInsertLockAcquireExclusive(void);
@@ -1022,6 +1047,8 @@ XLogInsertRecord(XLogRecData *rdata,
info == XLOG_SWITCH);
XLogRecPtr StartPos;
XLogRecPtr EndPos;
+ XLogSegNo StartSeg;+ XLogSegNo EndSeg;
bool prevDoPageWrites = doPageWrites;
/* we assume that all of the record header is in the first chunk */
@@ -1180,6 +1207,31 @@ XLogInsertRecord(XLogRecData *rdata,
SpinLockRelease(&XLogCtl->info_lck);
}
+ /*+ * Record the record boundary if we crossed the segment boundary. This is+ * used to ensure that segments are not marked ready for archival before the+ * entire record has been flushed to disk.+ *+ * Note that we do not use XLByteToPrevSeg() for determining the ending+ * segment. This is done so that a record that fits perfectly into the end+ * of the segment is marked ready for archival as soon as the flushed+ * pointer jumps to the next segment.+ */+ XLByteToSeg(StartPos, StartSeg, wal_segment_size);+ XLByteToSeg(EndPos, EndSeg, wal_segment_size);++ if (StartSeg != EndSeg &&+ XLogArchivingActive())+ {+ RegisterRecordBoundaryEntry(EndSeg, EndPos);++ /*+ * There's a chance that the record was already flushed to disk and we+ * missed marking segments as ready for archive, so try to do that now.+ */+ NotifySegmentsReadyForArchive();+ }+
/*
* If this was an XLOG_SWITCH record, flush the record and the empty
* padding space that fills the rest of the segment, and perform
@@ -1277,6 +1329,192 @@ XLogInsertRecord(XLogRecData *rdata,
return EndPos;
}
+/*+ * RegisterRecordBoundaryEntry+ *+ * This enters a new entry into the record boundary map, which is used for+ * determing when it is safe to mark a segment as ready for archival. An entry+ * with the given key (the segment number) must not already exist in the map.+ * Also, the caller is responsible for ensuring that XLByteToSeg() would return+ * the same segment number for the given record pointer.+ */+static void+RegisterRecordBoundaryEntry(XLogSegNo seg, XLogRecPtr pos)+{+ RecordBoundaryEntry *entry;+ bool found;++ LWLockAcquire(ArchNotifyLock, LW_EXCLUSIVE);++ entry = (RecordBoundaryEntry *) hash_search(RecordBoundaryMap,+ (void *) &seg, HASH_ENTER,+ &found);+ if (found)+ elog(ERROR, "record boundary entry for segment already exists");++ entry->pos = pos;++ LWLockRelease(ArchNotifyLock);+}++/*+ * NotifySegmentsReadyForArchive+ *+ * This function marks segments as ready for archival, given that it is safe to+ * do so. It is safe to call this function repeatedly, even if nothing has+ * changed since the last time it was called.+ */+static void+NotifySegmentsReadyForArchive(void)+{+ XLogRecPtr flushed;+ XLogSegNo flushed_seg;+ XLogSegNo latest_boundary_seg;+ XLogSegNo last_notified;++ /*+ * We first do a quick sanity check to see if we can bail out without taking+ * the ArchNotifyLock at all. It is expected that this function will run+ * frequently and that it will need to do nothing the vast majority of the+ * time.+ *+ * Specifically, we bail out if the shared memory value for the last+ * notified segment has not yet been initialized or if we've already marked+ * the segment prior to the segment that contains "flushed" as ready for+ * archival. We intentionally use XLByteToSeg() instead of+ * XLByteToPrevSeg() so that we don't skip notifying when a record fits+ * perfectly into the end of a segment. ("flushed" should point to the+ * first byte of the record _after_ the one that is known to be flushed to+ * disk.)+ */+ last_notified = GetLastNotifiedSegment();+ if (XLogSegNoIsInvalid(last_notified))+ return;++ flushed = GetFlushRecPtr();+ XLByteToSeg(flushed, flushed_seg, wal_segment_size);+ if (last_notified >= flushed_seg - 1)+ return;++ /*+ * At this point, we must acquire ArchNotifyLock before proceeding. In this+ * section, we look for the latest record boundary in RecordBoundaryMap that+ * is less than or equal to the current "flushed" pointer, and we notify the+ * archiver that all segments up to (but not including) that boundary's+ * associated segment are ready for archival.+ */+ LWLockAcquire(ArchNotifyLock, LW_EXCLUSIVE);++ latest_boundary_seg = GetLatestRecordBoundarySegment();+ if (!XLogSegNoIsInvalid(latest_boundary_seg))+ {+ XLogSegNo i;++ /* create the archive status files */+ for (i = GetLastNotifiedSegment() + 1; i < latest_boundary_seg; i++)+ XLogArchiveNotifySeg(i);++ /* update shared memory */+ SetLastNotifiedSegment(latest_boundary_seg - 1);++ /* remove old boundaries from the map */+ RemoveRecordBoundariesUpTo(latest_boundary_seg);+ }++ LWLockRelease(ArchNotifyLock);+}++/*+ * GetLatestRecordBoundarySegment+ *+ * This function finds the latest record boundary in RecordBoundaryMap that is+ * less than or equal to the current "flushed" pointer and returns its+ * associated segment number, given that it is greater than the last notified+ * segment. Otherwise, InvalidXLogSegNo is returned.+ *+ * Caller is expected to be holding ArchNotifyLock.+ */+static XLogSegNo+GetLatestRecordBoundarySegment(void)+{+ XLogRecPtr flushed;+ XLogSegNo flushed_seg;+ XLogSegNo last_notified;++ flushed = GetFlushRecPtr();+ XLByteToSeg(flushed, flushed_seg, wal_segment_size);+ last_notified = GetLastNotifiedSegment();++ for (XLogSegNo i = flushed_seg; i > last_notified; i--)+ {+ RecordBoundaryEntry *entry;++ entry = (RecordBoundaryEntry *) hash_search(RecordBoundaryMap,+ (void *) &i, HASH_FIND,+ NULL);++ if (entry != NULL && flushed >= entry->pos)+ return entry->seg;+ }++ return InvalidXLogSegNo;+}++/*+ * RemoveOldRecordBoundaries+ *+ * This function removes all entries in the RecordBoundaryMap with segment+ * numbers up to an including seg.+ *+ * Caller is expected to be holding ArchNotifyLock.+ */+static void+RemoveRecordBoundariesUpTo(XLogSegNo seg)+{+ RecordBoundaryEntry *entry;+ HASH_SEQ_STATUS status;++ hash_seq_init(&status, RecordBoundaryMap);++ while ((entry = (RecordBoundaryEntry *) hash_seq_search(&status)) != NULL)+ {+ if (entry->seg <= seg)+ (void) hash_search(RecordBoundaryMap, (void *) &entry->seg,+ HASH_REMOVE, NULL);+ }+}++/*+ * GetLastNotifiedSegment+ *+ * Retrieves last notified segment from shared memory.+ */+XLogSegNo+GetLastNotifiedSegment(void)+{+ XLogSegNo seg;++ SpinLockAcquire(&XLogCtl->info_lck);+ seg = XLogCtl->lastNotifiedSeg;+ SpinLockRelease(&XLogCtl->info_lck);++ return seg;+}++/*+ * SetLastNotifiedSegment+ *+ * Sets last notified segment in shared memory. Callers should hold+ * ArchNotifyLock exclusively when calling this function.+ */+static void+SetLastNotifiedSegment(XLogSegNo seg)+{+ SpinLockAcquire(&XLogCtl->info_lck);+ XLogCtl->lastNotifiedSeg = seg;+ SpinLockRelease(&XLogCtl->info_lck);+}+
/*
* Reserves the right amount of space for a record of given size from the WAL.
* *StartPos is set to the beginning of the reserved section, *EndPos to
@@ -2601,11 +2839,11 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
* later. Doing it here ensures that one and only one backend will
* perform this fsync.
*
- * This is also the right place to notify the Archiver that the- * segment is ready to copy to archival storage, and to update the- * timer for archive_timeout, and to signal for a checkpoint if- * too many logfile segments have been used since the last- * checkpoint.+ * This is also the right place to update the timer for+ * archive_timeout and to signal for a checkpoint if too many+ * logfile segments have been used since the last checkpoint. If+ * lastNotifiedSeg hasn't been initialized yet, we need to do that,+ * too.
*/
if (finishing_seg)
{
@@ -2616,8 +2854,13 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
LogwrtResult.Flush = LogwrtResult.Write; /* end of page */
- if (XLogArchivingActive())- XLogArchiveNotifySeg(openLogSegNo);+ if (XLogArchivingActive() &&+ XLogSegNoIsInvalid(GetLastNotifiedSegment()))+ {+ LWLockAcquire(ArchNotifyLock, LW_EXCLUSIVE);+ SetLastNotifiedSegment(openLogSegNo - 1);+ LWLockRelease(ArchNotifyLock);+ }
XLogCtl->lastSegSwitchTime = (pg_time_t) time(NULL);
XLogCtl->lastSegSwitchLSN = LogwrtResult.Flush;
@@ -2705,6 +2948,9 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
XLogCtl->LogwrtRqst.Flush = LogwrtResult.Flush;
SpinLockRelease(&XLogCtl->info_lck);
}
++ if (XLogArchivingActive())+ NotifySegmentsReadyForArchive();
}
/*
@@ -5115,6 +5361,9 @@ XLOGShmemSize(void)
/* and the buffers themselves */
size = add_size(size, mul_size(XLOG_BLCKSZ, XLOGbuffers));
+ /* stuff for marking segments as ready for archival */+ size = add_size(size, hash_estimate_size(16, sizeof(RecordBoundaryEntry)));+
/*
* Note: we don't count ControlFileData, it comes out of the "slop factor"
* added by CreateSharedMemoryAndSemaphores. This lets us use this
@@ -5132,6 +5381,7 @@ XLOGShmemInit(void)
char *allocptr;
int i;
ControlFileData *localControlFile;
+ HASHCTL info;
#ifdef WAL_DEBUG
@@ -5230,6 +5480,14 @@ XLOGShmemInit(void)
SpinLockInit(&XLogCtl->ulsn_lck);
InitSharedLatch(&XLogCtl->recoveryWakeupLatch);
ConditionVariableInit(&XLogCtl->recoveryNotPausedCV);
++ /* Initialize stuff for marking segments as ready for archival. */+ XLogCtl->lastNotifiedSeg = InvalidXLogSegNo;+ memset(&info, 0, sizeof(info));+ info.keysize = sizeof(XLogSegNo);+ info.entrysize = sizeof(RecordBoundaryEntry);+ RecordBoundaryMap = ShmemInitHash("Record Boundary Table", 16, 16, &info,+ HASH_ELEM | HASH_BLOBS);
}
/*
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txtindex 6c7cf6c295..d39225bf94 100644--- a/src/backend/storage/lmgr/lwlocknames.txt+++ b/src/backend/storage/lmgr/lwlocknames.txt@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+ArchNotifyLock 48diff --git a/src/include/access/xlogdefs.h b/src/include/access/xlogdefs.hindex 0940b64ca6..8038f708c7 100644--- a/src/include/access/xlogdefs.h+++ b/src/include/access/xlogdefs.h@@ -47,6 +47,9 @@ typedef uint64 XLogRecPtr;
*/
typedef uint64 XLogSegNo;
+#define InvalidXLogSegNo 0xFFFFFFFFFFFFFFFF+#define XLogSegNoIsInvalid(s) ((s) == InvalidXLogSegNo)+
/*
* TimeLineID (TLI) - identifies different database histories to prevent
* confusion after restoring a prior state of a database installation.
--
2.16.6
v2-0002-Keep-track-of-notified-ready-for-archive-position.patch (application/octet-stream, 10.6K) download
From ec1d48d4f301df54393afccb231490f2a9110cb3 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Thu, 18 Feb 2021 23:21:49 +0000
Subject: [PATCH v2 2/2] Keep track of notified-ready-for-archive position
through crashes.
If a WAL record that spans across multiple segments is fully
flushed to disk but the server crashes before the segment can be
marked as ready-for-archive, we will fail to mark it as ready-for-
archive when the server starts again. To fix, we add a
last_notified file in the archive_status directory that is used to
restore the corresponding value in shared memory.
This file is only used for primary servers, as standbys have their
own code path for marking segments as ready-for-archive that isn't
aware of WAL record boundaries. If the server starts up in point-
in-time recovery, standby mode, or with archive_mode turned off, we
delete this file. In those cases, the file will get initialized
normally if and when it begins writing its own WAL in the future.
---
src/backend/access/transam/xlog.c | 224 +++++++++++++++++++++++++++++++++++++-
src/bin/pg_resetwal/pg_resetwal.c | 5 +-
2 files changed, 226 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex 893b9ce951..fa8a58d1cf 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -538,6 +538,9 @@ typedef struct RecordBoundaryEntry
XLogRecPtr pos;
} RecordBoundaryEntry;
+/* File for persisting the last notified segment through crashes. */+#define LAST_NOTIFIED_FILE (XLOGDIR "/archive_status/last_notified")+
/*
* Shared state data for WAL insertion.
*/
@@ -994,12 +997,22 @@ static XLogRecPtr XLogBytePosToRecPtr(uint64 bytepos);
static XLogRecPtr XLogBytePosToEndRecPtr(uint64 bytepos);
static uint64 XLogRecPtrToBytePos(XLogRecPtr ptr);
static void checkXLogConsistency(XLogReaderState *record);
++typedef struct LastNotifiedSegment+{+ XLogSegNo segno;+ bool crash_handling;+} LastNotifiedSegment;+
static void RegisterRecordBoundaryEntry(XLogSegNo seg, XLogRecPtr pos);
static void NotifySegmentsReadyForArchive(void);
static XLogSegNo GetLastNotifiedSegment(void);
static void SetLastNotifiedSegment(XLogSegNo seg);
static XLogSegNo GetLatestRecordBoundarySegment(void);
static void RemoveRecordBoundariesUpTo(XLogSegNo seg);
+static void InitializeLastNotifiedSegment(void);+static LastNotifiedSegment ReadLastNotifiedSegmentFile(void);+static void WriteLastNotifiedSegmentFile(XLogSegNo segno, bool crash_handling);
static void WALInsertLockAcquire(void);
static void WALInsertLockAcquireExclusive(void);
@@ -1410,9 +1423,27 @@ NotifySegmentsReadyForArchive(void)
{
XLogSegNo i;
- /* create the archive status files */+ /* create the archive_status files and update last_notified */
for (i = GetLastNotifiedSegment() + 1; i < latest_boundary_seg; i++)
+ {+ /*+ * We must update the last_notified file _before_ creating the+ * archive status file for the segment. This ensures that we can+ * recover in the event of a crash. RemoveXlogFile()+ * recycles/removes segments prior to clearing the archive status+ * file, so if after a crash the segment exists but its+ * archive status file does not, we know we must create its archive+ * status file.+ *+ * It's also possible that we crash in the process of notifying+ * several segments in this loop, but we will not know to create+ * archive status files for those after a crash. Instead, we will+ * attempt to notify them again after a later segment becomes+ * eligible for archival.+ */+ WriteLastNotifiedSegmentFile(i, true);
XLogArchiveNotifySeg(i);
+ }
/* update shared memory */
SetLastNotifiedSegment(latest_boundary_seg - 1);
@@ -1515,6 +1546,191 @@ SetLastNotifiedSegment(XLogSegNo seg)
SpinLockRelease(&XLogCtl->info_lck);
}
+/*+ * InitializeLastNotifiedSegment+ *+ * If the last_notified file does not exist, no action is taken.+ *+ * If archiving is turned off, the server is a standby, or the server is+ * perfoming point-in-time recovery, the last_notified file is deleted. (Once+ * the standby is promoted or recovery completes, a new last_notified file will+ * be created in XLogWrite().)+ *+ * Otherwise, initialize lastNotifiedSeg with the value in the file and make+ * sure that segment's archive status is created.+ */+static void+InitializeLastNotifiedSegment(void)+{+ char fname[MAXPGPATH];+ char path[MAXPGPATH];+ struct stat filestats;+ LastNotifiedSegment seg;+ int rc;++ rc = stat(LAST_NOTIFIED_FILE, &filestats);+ if (rc != 0 && errno != ENOENT)+ ereport(PANIC,+ (errcode_for_file_access(),+ errmsg("could not stat file \"%s\": %m",+ LAST_NOTIFIED_FILE)));++ /*+ * If the last_notified file does not exist, there's nothing to do. If this+ * file is supposed to exist, it will be created in the future by+ * XLogWrite().+ */+ if (rc != 0)+ return;++ /*+ * If the server is a standby, is performing point-in-time recovery, or+ * archiving is turned off, delete the last_notified file and move on. Once+ * the standby is promoted or recovery completes, a new last_notified file+ * will be created in XLogWrite().+ */+ if (StandbyModeRequested ||+ ArchiveRecoveryRequested ||+ !XLogArchivingActive())+ {+ durable_unlink(LAST_NOTIFIED_FILE, PANIC);+ return;+ }++ /*+ * Retrieve the latest notified segment from the file.+ */+ seg = ReadLastNotifiedSegmentFile();+ XLogCtl->lastNotifiedSeg = seg.segno;++ /*+ * If crash_handling was disabled for the segment (e.g., the value was+ * purely for initialization purposes), there is no need to ensure that an+ * archive status file was created.+ */+ if (!seg.crash_handling)+ return;++ /*+ * If the segment still exists but there is no corresponding archive status+ * file, the server must have crashed between updating last_notified and+ * creating the archive status file. We must create an archive status file+ * for the segment now.+ */+ XLogFilePath(path, ThisTimeLineID, seg.segno, wal_segment_size);+ rc = stat(path, &filestats);+ if (rc != 0 && errno != ENOENT)+ ereport(PANIC,+ (errcode_for_file_access(),+ errmsg("could not stat file \"%s\": %m",+ path)));++ XLogFileName(fname, ThisTimeLineID, seg.segno, wal_segment_size);+ if (rc == 0 && !XLogArchiveIsReadyOrDone(fname))+ XLogArchiveNotify(fname);+}++/*+ * ReadLastNotifiedSegmentFile+ *+ * Reads the segment number from the last_notified file. The caller is+ * responsible for ensuring that the file exists.+ */+static LastNotifiedSegment+ReadLastNotifiedSegmentFile(void)+{+ LastNotifiedSegment seg;+ int readBytes;+ int fd;++ fd = OpenTransientFile(LAST_NOTIFIED_FILE, O_RDONLY | PG_BINARY);+ if (fd < 0)+ ereport(PANIC,+ (errcode_for_file_access(),+ errmsg("could not open file \"%s\": %m",+ LAST_NOTIFIED_FILE)));++ readBytes = read(fd, &seg, sizeof(seg));+ if (readBytes != sizeof(seg))+ {+ if (readBytes < 0)+ ereport(PANIC,+ (errcode_for_file_access(),+ errmsg("could not read file \"%s\": %m",+ LAST_NOTIFIED_FILE)));+ else+ ereport(PANIC,+ (errcode(ERRCODE_DATA_CORRUPTED),+ errmsg("could not read file \"%s\": read %d of %zu",+ LAST_NOTIFIED_FILE, readBytes, sizeof(seg))));+ }++ if (CloseTransientFile(fd) != 0)+ ereport(PANIC,+ (errcode_for_file_access(),+ errmsg("could not close file \"%s\": %m",+ LAST_NOTIFIED_FILE)));++ return seg;+}++/*+ * WriteLastNotifiedSegmentFile+ *+ * Writes the given segment number to the last_notified file. The caller is+ * responsible for ensuring that no other backends can perform this at the same+ * time.+ */+static void+WriteLastNotifiedSegmentFile(XLogSegNo segno, bool crash_handling)+{+ LastNotifiedSegment seg;+ int fd;+ char path[MAXPGPATH];++ memset(&seg, 0, sizeof(seg));+ seg.segno = segno;+ seg.crash_handling = crash_handling;++ snprintf(path, MAXPGPATH, "%s.tmp", LAST_NOTIFIED_FILE);++ /* make sure no old temp file is remaining */+ if (unlink(path) < 0 && errno != ENOENT)+ ereport(PANIC,+ (errcode_for_file_access(),+ errmsg("could not remove file \"%s\": %m",+ path)));++ fd = OpenTransientFile(path,+ O_CREAT | O_EXCL | O_WRONLY | PG_BINARY);+ if (fd < 0)+ ereport(PANIC,+ (errcode_for_file_access(),+ errmsg("could not create file \"%s\": %m",+ path)));++ errno = 0;+ if ((write(fd, &seg, sizeof(seg))) != sizeof(seg))+ {+ /* if write didn't set errno, assume problem is no disk space */+ if (errno == 0)+ errno = ENOSPC;+ ereport(PANIC,+ (errcode_for_file_access(),+ errmsg("could not write to file \"%s\": %m",+ path)));+ }++ if (CloseTransientFile(fd) != 0)+ ereport(PANIC,+ (errcode_for_file_access(),+ errmsg("could not close file \"%s\": %m",+ path)));++ /* fsync, rename to permanent file, fsync file and directory */+ durable_rename(path, LAST_NOTIFIED_FILE, PANIC);+}+
/*
* Reserves the right amount of space for a record of given size from the WAL.
* *StartPos is set to the beginning of the reserved section, *EndPos to
@@ -2858,7 +3074,10 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
XLogSegNoIsInvalid(GetLastNotifiedSegment()))
{
LWLockAcquire(ArchNotifyLock, LW_EXCLUSIVE);
+
SetLastNotifiedSegment(openLogSegNo - 1);
+ WriteLastNotifiedSegmentFile(openLogSegNo - 1, false);+
LWLockRelease(ArchNotifyLock);
}
@@ -7278,6 +7497,9 @@ StartupXLOG(void)
InRecovery = true;
}
+ /* If necessary, read the last_notified file. */+ InitializeLastNotifiedSegment();+
/* REDO */
if (InRecovery)
{
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.cindex 805dafef07..fe58a00854 100644--- a/src/bin/pg_resetwal/pg_resetwal.c+++ b/src/bin/pg_resetwal/pg_resetwal.c@@ -1074,11 +1074,12 @@ KillExistingArchiveStatus(void)
while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
- if (strspn(xlde->d_name, "0123456789ABCDEF") == XLOG_FNAME_LEN &&+ if ((strcmp(xlde->d_name, "last_notified") == 0) ||+ (strspn(xlde->d_name, "0123456789ABCDEF") == XLOG_FNAME_LEN &&
(strcmp(xlde->d_name + XLOG_FNAME_LEN, ".ready") == 0 ||
strcmp(xlde->d_name + XLOG_FNAME_LEN, ".done") == 0 ||
strcmp(xlde->d_name + XLOG_FNAME_LEN, ".partial.ready") == 0 ||
- strcmp(xlde->d_name + XLOG_FNAME_LEN, ".partial.done") == 0))+ strcmp(xlde->d_name + XLOG_FNAME_LEN, ".partial.done") == 0)))
{
snprintf(path, sizeof(path), "%s/%s", ARCHSTATDIR, xlde->d_name);
if (unlink(path) < 0)
--
2.16.6
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bossartn@amazon.com, horikyota.ntt@gmail.com, x4mmm@yandex-team.ru, a.lubennikova@postgrespro.ru, hlinnaka@iki.fi, matsumura.ryo@fujitsu.com, masao.fujii@gmail.com
Subject: Re: archive status ".ready" files may be created too early
In-Reply-To: <E63E5670-6CC3-4B09-9686-A77CF94FE4A8@amazon.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox