public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v10] Avoid creating archive status ".ready" files too early. 10+ messages / 6 participants [nested] [flat]
* [PATCH v10] Avoid creating archive status ".ready" files too early. @ 2021-08-17 03:52 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Nathan Bossart @ 2021-08-17 03:52 UTC (permalink / raw) WAL records may span multiple segments, but XLogWrite() does not wait for the entire record to be written out to disk before creating archive status files. Instead, as soon as the last WAL page of the segment is written, the archive status file will be created. If PostgreSQL crashes before it is able to write the rest of the record, it will end up reusing segments that have already been marked as ready-for-archival. However, the archiver process may have already processed the old version of the segment, so the wrong version of the segment may be backed-up. This backed-up segment will cause operations such as point-in-time restores to fail. To fix this, we keep track of records that span across segments and ensure that segments are only marked ready-for-archival once such records have been completely written to disk. --- src/backend/access/transam/timeline.c | 2 +- src/backend/access/transam/xlog.c | 291 ++++++++++++++++++++++- src/backend/access/transam/xlogarchive.c | 17 +- src/backend/postmaster/walwriter.c | 7 + src/backend/replication/walreceiver.c | 6 +- src/backend/storage/lmgr/lwlocknames.txt | 1 + src/include/access/xlog.h | 1 + src/include/access/xlogarchive.h | 4 +- src/include/access/xlogdefs.h | 1 + src/tools/pgindent/typedefs.list | 1 + 10 files changed, 309 insertions(+), 22 deletions(-) diff --git a/src/backend/access/transam/timeline.c b/src/backend/access/transam/timeline.c index 8d0903c175..acd5c2431d 100644 --- a/src/backend/access/transam/timeline.c +++ b/src/backend/access/transam/timeline.c @@ -452,7 +452,7 @@ writeTimeLineHistory(TimeLineID newTLI, TimeLineID parentTLI, if (XLogArchivingActive()) { TLHistoryFileName(histfname, newTLI); - XLogArchiveNotify(histfname); + XLogArchiveNotify(histfname, true); } } diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index e51a7a749d..d2ccf2a7bb 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -512,6 +512,16 @@ typedef enum ExclusiveBackupState */ static SessionBackupState sessionBackupState = SESSION_BACKUP_NONE; +/* + * Entries for SegmentBoundaryMap. Each such entry represents a WAL + * record that ends in endpos and crosses a WAL segment boundary. + */ +typedef struct SegmentBoundaryEntry +{ + XLogSegNo seg; /* must be first */ + XLogRecPtr endpos; +} SegmentBoundaryEntry; + /* * Shared state data for WAL insertion. */ @@ -723,6 +733,12 @@ typedef struct XLogCtlData */ XLogRecPtr lastFpwDisableRecPtr; + /* + * The last segment we've marked ready for archival. Protected by + * info_lck. + */ + XLogSegNo lastNotifiedSeg; + slock_t info_lck; /* locks shared variables shown above */ } XLogCtlData; @@ -736,6 +752,12 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +/* + * Segment boundary map, used for marking segments as ready for archival. + * Protected by SegmentBoundaryLock. + */ +static HTAB *SegmentBoundaryMap = NULL; + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -962,6 +984,13 @@ static XLogRecPtr XLogBytePosToRecPtr(uint64 bytepos); static XLogRecPtr XLogBytePosToEndRecPtr(uint64 bytepos); static uint64 XLogRecPtrToBytePos(XLogRecPtr ptr); static void checkXLogConsistency(XLogReaderState *record); +static void RegisterSegmentBoundaryEntry(XLogSegNo seg, XLogRecPtr pos); +static XLogSegNo GetLastNotifiedSegment(void); +static void SetLastNotifiedSegment(XLogSegNo seg); +static bool GetLatestSegmentBoundary(XLogSegNo last_notified, + XLogRecPtr flushed, + XLogSegNo *latest_boundary_seg); +static void RemoveSegmentBoundariesUpTo(XLogSegNo seg); static void WALInsertLockAcquire(void); static void WALInsertLockAcquireExclusive(void); @@ -1158,6 +1187,9 @@ XLogInsertRecord(XLogRecData *rdata, */ if (StartPos / XLOG_BLCKSZ != EndPos / XLOG_BLCKSZ) { + XLogSegNo StartSeg; + XLogSegNo EndSeg; + SpinLockAcquire(&XLogCtl->info_lck); /* advance global request to include new block(s) */ if (XLogCtl->LogwrtRqst.Write < EndPos) @@ -1165,12 +1197,38 @@ XLogInsertRecord(XLogRecData *rdata, /* update local result copy while I have the chance */ LogwrtResult = XLogCtl->LogwrtResult; SpinLockRelease(&XLogCtl->info_lck); + + /* + * If we crossed the segment boundary, record it. This is used to + * ensure that segments are not marked ready for archival before the + * entire record has been flushed to disk. + * + * Note that we do not use XLByteToPrevSeg() for determining the + * ending segment. This is done so that a record that fits perfectly + * into the end of the segment is marked ready for archival as soon as + * the flushed pointer jumps to the next segment. + */ + XLByteToSeg(StartPos, StartSeg, wal_segment_size); + XLByteToSeg(EndPos, EndSeg, wal_segment_size); + + if (StartSeg != EndSeg && XLogArchivingActive()) + { + RegisterSegmentBoundaryEntry(EndSeg, EndPos); + + /* + * There's a chance that the record was already flushed to disk + * and we missed marking segments as ready for archive. If this + * happens, we nudge the WALWriter, which will take care of + * notifying segments as needed. + */ + if (LogwrtResult.Flush > EndPos && ProcGlobal->walwriterLatch) + SetLatch(ProcGlobal->walwriterLatch); + } } /* * If this was an XLOG_SWITCH record, flush the record and the empty - * padding space that fills the rest of the segment, and perform - * end-of-segment actions (eg, notifying archiver). + * padding space that fills the rest of the segment. */ if (isLogSwitch) { @@ -1264,6 +1322,192 @@ XLogInsertRecord(XLogRecData *rdata, return EndPos; } +/* + * RegisterSegmentBoundaryEntry + * + * This enters a new entry into the segment boundary map, which is used for + * determing when it is safe to mark a segment as ready for archival. An entry + * with the given key (the segment number) must not already exist in the map. + * Also, the caller is responsible for ensuring that XLByteToSeg() would return + * the same segment number for the given record pointer. + */ +static void +RegisterSegmentBoundaryEntry(XLogSegNo seg, XLogRecPtr pos) +{ + SegmentBoundaryEntry *entry; + bool found; + + LWLockAcquire(SegmentBoundaryLock, LW_EXCLUSIVE); + + entry = (SegmentBoundaryEntry *) hash_search(SegmentBoundaryMap, + (void *) &seg, HASH_ENTER, + &found); + if (found) + elog(ERROR, "entry for segment already exists"); + + entry->endpos = pos; + + LWLockRelease(SegmentBoundaryLock); +} + +/* + * NotifySegmentsReadyForArchive + * + * This function marks segments as ready for archival, given that it is safe to + * do so. It is safe to call this function repeatedly, even if nothing has + * changed since the last time it was called. + */ +void +NotifySegmentsReadyForArchive(XLogRecPtr flushRecPtr) +{ + XLogSegNo flushed_seg; + XLogSegNo latest_boundary_seg; + XLogSegNo last_notified; + + /* + * We first do a quick sanity check to see if we can bail out without + * taking the SegmentBoundaryLock at all. It is expected that this + * function will run frequently and that it will need to do nothing the + * vast majority of the time. + * + * Specifically, we bail out if we've already marked the segment prior to + * the segment that contains flushRecPtr as ready for archival. We + * intentionally use XLByteToSeg() instead of XLByteToPrevSeg() so that we + * don't skip notifying when a record fits perfectly into the end of a + * segment. (flushRecPtr should point to the first byte of the record + * _after_ the one that is known to be flushed to disk.) + */ + last_notified = GetLastNotifiedSegment(); + XLByteToSeg(flushRecPtr, flushed_seg, wal_segment_size); + if (last_notified >= flushed_seg - 1) + return; + + LWLockAcquire(SegmentBoundaryLock, LW_EXCLUSIVE); + + /* Reobtain lastNotifiedSeg in case someone else changed it. */ + last_notified = GetLastNotifiedSegment(); + + /* Retrieve the latest segment boundary to use for notifying segments. */ + if (GetLatestSegmentBoundary(last_notified, flushRecPtr, &latest_boundary_seg)) + { + /* + * Update shared memory and discard segment boundaries that are no + * longer needed. + * + * It is safe to update shared memory before we attempt to create the + * .ready files. If our calls to XLogArchiveNotifySeg() fail, + * RemoveOldXlogFiles() will retry it as needed. + */ + SetLastNotifiedSegment(latest_boundary_seg - 1); + RemoveSegmentBoundariesUpTo(latest_boundary_seg); + + LWLockRelease(SegmentBoundaryLock); + + /* + * Notify archiver about segments that are ready for archival (by + * creating the corresponding .ready files). + */ + for (XLogSegNo seg = last_notified + 1; seg < latest_boundary_seg; seg++) + XLogArchiveNotifySeg(seg, false); + + PgArchWakeup(); + } + else + LWLockRelease(SegmentBoundaryLock); +} + +/* + * GetLatestSegmentBoundary + * + * This function finds the latest segment boundary in SegmentBoundaryMap that is + * less than or equal to the given "flushed" pointer and beyond the last + * notified segment. If such a segment is found, latest_boundary_seg is + * populated and true is returned. Otherwise, false is returned. + */ +static bool +GetLatestSegmentBoundary(XLogSegNo last_notified, XLogRecPtr flushed, + XLogSegNo *latest_boundary_seg) +{ + XLogSegNo flushed_seg; + XLogSegNo seg; + + Assert(LWLockHeldByMe(SegmentBoundaryLock)); + Assert(latest_boundary_seg != NULL); + + XLByteToSeg(flushed, flushed_seg, wal_segment_size); + + for (seg = flushed_seg; seg > last_notified; seg--) + { + SegmentBoundaryEntry *entry; + + entry = (SegmentBoundaryEntry *) hash_search(SegmentBoundaryMap, + (void *) &seg, HASH_FIND, + NULL); + + if (entry != NULL && flushed >= entry->endpos) + { + *latest_boundary_seg = entry->seg; + return true; + } + } + + return false; +} + +/* + * RemoveSegmentBoundariesUpTo + * + * This function removes all entries in the SegmentBoundaryMap with segment + * numbers up to and including seg. + */ +static void +RemoveSegmentBoundariesUpTo(XLogSegNo seg) +{ + SegmentBoundaryEntry *entry; + HASH_SEQ_STATUS status; + + Assert(LWLockHeldByMeInMode(SegmentBoundaryLock, LW_EXCLUSIVE)); + + hash_seq_init(&status, SegmentBoundaryMap); + + while ((entry = (SegmentBoundaryEntry *) hash_seq_search(&status)) != NULL) + { + if (entry->seg <= seg) + (void) hash_search(SegmentBoundaryMap, (void *) &entry->seg, + HASH_REMOVE, NULL); + } +} + +/* + * GetLastNotifiedSegment + * + * Retrieves last notified segment from shared memory. + */ +XLogSegNo +GetLastNotifiedSegment(void) +{ + XLogSegNo seg; + + SpinLockAcquire(&XLogCtl->info_lck); + seg = XLogCtl->lastNotifiedSeg; + SpinLockRelease(&XLogCtl->info_lck); + + return seg; +} + +/* + * SetLastNotifiedSegment + * + * Sets last notified segment in shared memory. + */ +static void +SetLastNotifiedSegment(XLogSegNo seg) +{ + SpinLockAcquire(&XLogCtl->info_lck); + XLogCtl->lastNotifiedSeg = seg; + SpinLockRelease(&XLogCtl->info_lck); +} + /* * Reserves the right amount of space for a record of given size from the WAL. * *StartPos is set to the beginning of the reserved section, *EndPos to @@ -2421,6 +2665,7 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible) /* We should always be inside a critical section here */ Assert(CritSectionCount > 0); + Assert(LWLockHeldByMe(WALWriteLock)); /* * Update local LogwrtResult (caller probably did this already, but...) @@ -2586,11 +2831,13 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible) * later. Doing it here ensures that one and only one backend will * perform this fsync. * - * This is also the right place to notify the Archiver that the - * segment is ready to copy to archival storage, and to update the - * timer for archive_timeout, and to signal for a checkpoint if - * too many logfile segments have been used since the last - * checkpoint. + * If WAL archiving is active, we attempt to notify the archiver + * of any segments that are now ready for archival. + * + * This is also the right place to update the timer for + * archive_timeout and to signal for a checkpoint if too many + * logfile segments have been used since the last checkpoint. + * */ if (finishing_seg) { @@ -2602,7 +2849,7 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible) LogwrtResult.Flush = LogwrtResult.Write; /* end of page */ if (XLogArchivingActive()) - XLogArchiveNotifySeg(openLogSegNo); + NotifySegmentsReadyForArchive(LogwrtResult.Flush); XLogCtl->lastSegSwitchTime = (pg_time_t) time(NULL); XLogCtl->lastSegSwitchLSN = LogwrtResult.Flush; @@ -2690,6 +2937,9 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible) XLogCtl->LogwrtRqst.Flush = LogwrtResult.Flush; SpinLockRelease(&XLogCtl->info_lck); } + + if (XLogArchivingActive()) + NotifySegmentsReadyForArchive(LogwrtResult.Flush); } /* @@ -5117,6 +5367,9 @@ XLOGShmemSize(void) /* and the buffers themselves */ size = add_size(size, mul_size(XLOG_BLCKSZ, XLOGbuffers)); + /* stuff for marking segments as ready for archival */ + size = add_size(size, hash_estimate_size(16, sizeof(SegmentBoundaryEntry))); + /* * Note: we don't count ControlFileData, it comes out of the "slop factor" * added by CreateSharedMemoryAndSemaphores. This lets us use this @@ -5134,6 +5387,7 @@ XLOGShmemInit(void) char *allocptr; int i; ControlFileData *localControlFile; + HASHCTL info; #ifdef WAL_DEBUG @@ -5227,12 +5481,20 @@ XLOGShmemInit(void) XLogCtl->InstallXLogFileSegmentActive = false; XLogCtl->SharedPromoteIsTriggered = false; XLogCtl->WalWriterSleeping = false; + XLogCtl->lastNotifiedSeg = MaxXLogSegNo; SpinLockInit(&XLogCtl->Insert.insertpos_lck); SpinLockInit(&XLogCtl->info_lck); SpinLockInit(&XLogCtl->ulsn_lck); InitSharedLatch(&XLogCtl->recoveryWakeupLatch); ConditionVariableInit(&XLogCtl->recoveryNotPausedCV); + + /* Initialize stuff for marking segments as ready for archival. */ + memset(&info, 0, sizeof(info)); + info.keysize = sizeof(XLogSegNo); + info.entrysize = sizeof(SegmentBoundaryEntry); + SegmentBoundaryMap = ShmemInitHash("Segment Boundary Table", 16, 16, &info, + HASH_ELEM | HASH_BLOBS); } /* @@ -7873,6 +8135,17 @@ StartupXLOG(void) XLogCtl->LogwrtRqst.Write = EndOfLog; XLogCtl->LogwrtRqst.Flush = EndOfLog; + /* + * Initialize XLogCtl->lastNotifiedSeg to the previous WAL file. + */ + if (XLogArchivingActive()) + { + XLogSegNo EndOfLogSeg; + + XLByteToSeg(EndOfLog, EndOfLogSeg, wal_segment_size); + SetLastNotifiedSegment(EndOfLogSeg - 1); + } + /* * Update full_page_writes in shared memory and write an XLOG_FPW_CHANGE * record before resource manager writes cleanup WAL records or checkpoint @@ -8000,7 +8273,7 @@ StartupXLOG(void) XLogArchiveCleanup(partialfname); durable_rename(origpath, partialpath, ERROR); - XLogArchiveNotify(partialfname); + XLogArchiveNotify(partialfname, true); } } } diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c index 26b023e754..b9c19b2085 100644 --- a/src/backend/access/transam/xlogarchive.c +++ b/src/backend/access/transam/xlogarchive.c @@ -433,7 +433,7 @@ KeepFileRestoredFromArchive(const char *path, const char *xlogfname) if (XLogArchiveMode != ARCHIVE_MODE_ALWAYS) XLogArchiveForceDone(xlogfname); else - XLogArchiveNotify(xlogfname); + XLogArchiveNotify(xlogfname, true); /* * If the existing file was replaced, since walsenders might have it open, @@ -462,9 +462,12 @@ KeepFileRestoredFromArchive(const char *path, const char *xlogfname) * by the archiver, e.g. we write 0000000100000001000000C6.ready * and the archiver then knows to archive XLOGDIR/0000000100000001000000C6, * then when complete, rename it to 0000000100000001000000C6.done + * + * Optionally, nudge the archiver process so that it'll notice the file we + * create. */ void -XLogArchiveNotify(const char *xlog) +XLogArchiveNotify(const char *xlog, bool nudge) { char archiveStatusPath[MAXPGPATH]; FILE *fd; @@ -489,8 +492,8 @@ XLogArchiveNotify(const char *xlog) return; } - /* Notify archiver that it's got something to do */ - if (IsUnderPostmaster) + /* If caller requested, let archiver know it's got work to do */ + if (nudge) PgArchWakeup(); } @@ -498,12 +501,12 @@ XLogArchiveNotify(const char *xlog) * Convenience routine to notify using segment number representation of filename */ void -XLogArchiveNotifySeg(XLogSegNo segno) +XLogArchiveNotifySeg(XLogSegNo segno, bool nudge) { char xlog[MAXFNAMELEN]; XLogFileName(xlog, ThisTimeLineID, segno, wal_segment_size); - XLogArchiveNotify(xlog); + XLogArchiveNotify(xlog, nudge); } /* @@ -608,7 +611,7 @@ XLogArchiveCheckDone(const char *xlog) return true; /* Retry creation of the .ready file */ - XLogArchiveNotify(xlog); + XLogArchiveNotify(xlog, true); return false; } diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 626fae8454..6a1e16edc2 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -248,6 +248,13 @@ WalWriterMain(void) /* Process any signals received recently */ HandleWalWriterInterrupts(); + /* + * Notify the archiver of any WAL segments that are ready. We do this + * here to handle a race condition where WAL is flushed to disk prior + * to registering the segment boundary. + */ + NotifySegmentsReadyForArchive(GetFlushRecPtr()); + /* * Do what we're here for; then, if XLogBackgroundFlush() found useful * work to do, reset hibernation counter. diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 9a2bc37fd7..60de3be92c 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -622,7 +622,7 @@ WalReceiverMain(void) if (XLogArchiveMode != ARCHIVE_MODE_ALWAYS) XLogArchiveForceDone(xlogfname); else - XLogArchiveNotify(xlogfname); + XLogArchiveNotify(xlogfname, true); } recvFile = -1; @@ -760,7 +760,7 @@ WalRcvFetchTimeLineHistoryFiles(TimeLineID first, TimeLineID last) if (XLogArchiveMode != ARCHIVE_MODE_ALWAYS) XLogArchiveForceDone(fname); else - XLogArchiveNotify(fname); + XLogArchiveNotify(fname, true); pfree(fname); pfree(content); @@ -915,7 +915,7 @@ XLogWalRcvWrite(char *buf, Size nbytes, XLogRecPtr recptr) if (XLogArchiveMode != ARCHIVE_MODE_ALWAYS) XLogArchiveForceDone(xlogfname); else - XLogArchiveNotify(xlogfname); + XLogArchiveNotify(xlogfname, true); } recvFile = -1; diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt index 6c7cf6c295..14a742d655 100644 --- a/src/backend/storage/lmgr/lwlocknames.txt +++ b/src/backend/storage/lmgr/lwlocknames.txt @@ -53,3 +53,4 @@ XactTruncationLock 44 # 45 was XactTruncationLock until removal of BackendRandomLock WrapLimitsVacuumLock 46 NotifyQueueTailLock 47 +SegmentBoundaryLock 48 diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 0a8ede700d..6b6ae81c2d 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -315,6 +315,7 @@ extern XLogRecPtr GetInsertRecPtr(void); extern XLogRecPtr GetFlushRecPtr(void); extern XLogRecPtr GetLastImportantRecPtr(void); extern void RemovePromoteSignalFiles(void); +extern void NotifySegmentsReadyForArchive(XLogRecPtr flushRecPtr); extern bool PromoteIsTriggered(void); extern bool CheckPromoteSignal(void); diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h index 3edd1a976c..935b4cb02d 100644 --- a/src/include/access/xlogarchive.h +++ b/src/include/access/xlogarchive.h @@ -23,8 +23,8 @@ extern bool RestoreArchivedFile(char *path, const char *xlogfname, extern void ExecuteRecoveryCommand(const char *command, const char *commandName, bool failOnSignal); extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname); -extern void XLogArchiveNotify(const char *xlog); -extern void XLogArchiveNotifySeg(XLogSegNo segno); +extern void XLogArchiveNotify(const char *xlog, bool nudge); +extern void XLogArchiveNotifySeg(XLogSegNo segno, bool nudge); extern void XLogArchiveForceDone(const char *xlog); extern bool XLogArchiveCheckDone(const char *xlog); extern bool XLogArchiveIsBusy(const char *xlog); diff --git a/src/include/access/xlogdefs.h b/src/include/access/xlogdefs.h index 60348d1850..9b455e88e3 100644 --- a/src/include/access/xlogdefs.h +++ b/src/include/access/xlogdefs.h @@ -46,6 +46,7 @@ typedef uint64 XLogRecPtr; * XLogSegNo - physical log file sequence number. */ typedef uint64 XLogSegNo; +#define MaxXLogSegNo ((XLogSegNo) 0xFFFFFFFFFFFFFFFF) /* * TimeLineID (TLI) - identifies different database histories to prevent diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 37cf4b2f76..79694b049e 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2334,6 +2334,7 @@ SecBufferDesc SecLabelItem SecLabelStmt SeenRelsEntry +SegmentBoundaryEntry SelectLimit SelectStmt Selectivity -- 2.30.2 --aabamu5eazcf6rc5-- ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Built-in Raft replication @ 2025-04-16 04:58 Andrey Borodin <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Andrey Borodin @ 2025-04-16 04:58 UTC (permalink / raw) To: Ashutosh Bapat <[email protected]>; +Cc: Tom Lane <[email protected]>; Konstantin Osipov <[email protected]>; Greg Sabino Mullane <[email protected]>; Nikolay Samokhvalov <[email protected]>; PostgreSQL Hackers <[email protected]> > On 16 Apr 2025, at 09:33, Ashutosh Bapat <[email protected]> wrote: > > In my experience, the load of managing hundreds of replicas which all > participate in RAFT protocol becomes more than regular transaction > load. So making every replica a RAFT participant will affect the > ability to deploy hundreds of replica. No need to make all standbys voting. And no need to make plain topology. pg_consul is using 2/3 or 3/5 HA groups, and cascades all others from HA group. Existing tools already solve the original problem, Konstantin is just proposing to solve it in some standard “official” way. > We may build an extension which > has a similar role in PostgreSQL world as zookeeper in Hadoop. Patroni, pg_consul and others already use zookeeper, etcd and similar systems for consensus. Is it any better as extension than as etcd? > It can > be then used for other distributed systems as well - like shared > nothing clusters based on FDW. I didn’t get FDW analogy. Why other distributed systems should choose Postgres extension over Zookeeper? > There's already a proposal to bring > CREATE SERVER to the world of logical replication - so I see these two > worlds uniting in future. Again, I’m lost here. Which two worlds? > The way I imagine it is some PostgreSQL > instances, which have this extension installed, will act as a RAFT > cluster (similar to Zookeeper ensemble or etcd cluster). That’s exactly what is proposed here. > The > distributed system based on logical replication or FDW or both will > use this ensemble to manage its shared state. The same ensemble can be > shared across multiple distributed clusters if it has scaling > capabilities. Yes, shared DCS are common these days. AFAIK, we use one Zookeeper instance per hundred Postgres clusters to coordinate pg_consuls. Actually, scalability is opposite to topic of this thread. Let me explain. Currently, Postgres automatic failover tools rely on databases with built-in automatic failover. Konstantin is proposing to shorten this loop and make Postgres use its build-in automatic failover. So, existing tooling allows you to have 3 hosts for DCS, with majority of 2 hosts able to elect new leader in case of failover. And you can have only 2 hosts for Postgres - Primary and Standby. You can have 2 big Postgres machines with 64 CPUs. And 3 one-CPU hosts for Zookeper\etcd. If you use build-in failover you have to resort to 3 big Postgres machines because you need 2/3 majority. Of course, you can install MySQL-stype arbiter - host that had no real PGDATA, only participates in voting. But this is a solution to problem induced by built-in autofailover. Best regards, Andrey Borodin. ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Built-in Raft replication @ 2025-04-16 06:18 Ashutosh Bapat <[email protected]> parent: Andrey Borodin <[email protected]> 0 siblings, 3 replies; 10+ messages in thread From: Ashutosh Bapat @ 2025-04-16 06:18 UTC (permalink / raw) To: Andrey Borodin <[email protected]>; +Cc: Tom Lane <[email protected]>; Konstantin Osipov <[email protected]>; Greg Sabino Mullane <[email protected]>; Nikolay Samokhvalov <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Apr 16, 2025 at 10:29 AM Andrey Borodin <[email protected]> wrote: > > > We may build an extension which > > has a similar role in PostgreSQL world as zookeeper in Hadoop. > > Patroni, pg_consul and others already use zookeeper, etcd and similar systems for consensus. > Is it any better as extension than as etcd? I feel so. An extension runs from within a postgresql process, uses the same protocol as PostgreSQL whereas etcd is another process and another protocol. > > > It can > > be then used for other distributed systems as well - like shared > > nothing clusters based on FDW. > > I didn’t get FDW analogy. Why other distributed systems should choose Postgres extension over Zookeeper? By other distributed systems I mean PostgreSQL distributed systems - FDW based native sharding or native replication or a system which uses both. > > > There's already a proposal to bring > > CREATE SERVER to the world of logical replication - so I see these two > > worlds uniting in future. > > Again, I’m lost here. Which two worlds? Logical replication and FDW based native sharding. > > > The > > distributed system based on logical replication or FDW or both will > > use this ensemble to manage its shared state. The same ensemble can be > > shared across multiple distributed clusters if it has scaling > > capabilities. > > Yes, shared DCS are common these days. AFAIK, we use one Zookeeper instance per hundred Postgres clusters to coordinate pg_consuls. > > Actually, scalability is opposite to topic of this thread. Let me explain. > Currently, Postgres automatic failover tools rely on databases with built-in automatic failover. Konstantin is proposing to shorten this loop and make Postgres use its build-in automatic failover. > > So, existing tooling allows you to have 3 hosts for DCS, with majority of 2 hosts able to elect new leader in case of failover. > And you can have only 2 hosts for Postgres - Primary and Standby. You can have 2 big Postgres machines with 64 CPUs. And 3 one-CPU hosts for Zookeper\etcd. > > If you use build-in failover you have to resort to 3 big Postgres machines because you need 2/3 majority. Of course, you can install MySQL-stype arbiter - host that had no real PGDATA, only participates in voting. But this is a solution to problem induced by built-in autofailover. Users find it a waste of resources to deploy 3 big PostgreSQL instances just for HA where 2 suffice even if they deploy 3 lightweight DCS instances. Having only some of the nodes act as DCS and others purely PostgreSQL nodes will reduce waste of resources. -- Best Wishes, Ashutosh Bapat ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Built-in Raft replication @ 2025-04-16 06:27 Andrey Borodin <[email protected]> parent: Ashutosh Bapat <[email protected]> 2 siblings, 1 reply; 10+ messages in thread From: Andrey Borodin @ 2025-04-16 06:27 UTC (permalink / raw) To: Ashutosh Bapat <[email protected]>; +Cc: Tom Lane <[email protected]>; Konstantin Osipov <[email protected]>; Greg Sabino Mullane <[email protected]>; Nikolay Samokhvalov <[email protected]>; PostgreSQL Hackers <[email protected]> > On 16 Apr 2025, at 11:18, Ashutosh Bapat <[email protected]> wrote: > > Having only some of the nodes act as DCS > and others purely PostgreSQL nodes will reduce waste of resources. But typically you need more DCS nodes than PostgreSQL nodes. Did you mean “Having only some of nodes act as PostgreSQL and others purely DCS nodes will reduce waste of resources”? Best regards, Andrey Borodin. ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Built-in Raft replication @ 2025-04-16 06:45 Ashutosh Bapat <[email protected]> parent: Andrey Borodin <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Ashutosh Bapat @ 2025-04-16 06:45 UTC (permalink / raw) To: Andrey Borodin <[email protected]>; +Cc: Tom Lane <[email protected]>; Konstantin Osipov <[email protected]>; Greg Sabino Mullane <[email protected]>; Nikolay Samokhvalov <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Apr 16, 2025 at 11:57 AM Andrey Borodin <[email protected]> wrote: > > > > > On 16 Apr 2025, at 11:18, Ashutosh Bapat <[email protected]> wrote: > > > > Having only some of the nodes act as DCS > > and others purely PostgreSQL nodes will reduce waste of resources. > > But typically you need more DCS nodes than PostgreSQL nodes. Did you mean In a small HA setup this might be true. But not when there are many replicas. But ... > “Having only some nodes act as PostgreSQL and others purely DCS nodes will reduce waste of resources”? I mean, whatever the setup may be one shouldn't require to deploy a big PostgreSQL server just because DCS needs majority. -- Best Wishes, Ashutosh Bapat ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Built-in Raft replication @ 2025-04-16 12:53 Alastair Turner <[email protected]> parent: Ashutosh Bapat <[email protected]> 2 siblings, 1 reply; 10+ messages in thread From: Alastair Turner @ 2025-04-16 12:53 UTC (permalink / raw) To: Ashutosh Bapat <[email protected]>; +Cc: Andrey Borodin <[email protected]>; Tom Lane <[email protected]>; Konstantin Osipov <[email protected]>; Greg Sabino Mullane <[email protected]>; Nikolay Samokhvalov <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, 16 Apr 2025 at 07:18, Ashutosh Bapat <[email protected]> wrote: > On Wed, Apr 16, 2025 at 10:29 AM Andrey Borodin <[email protected]> > wrote: > > > > If you use build-in failover you have to resort to 3 big Postgres > machines because you need 2/3 majority. Of course, you can install > MySQL-stype arbiter - host that had no real PGDATA, only participates in > voting. But this is a solution to problem induced by built-in autofailover. > > Users find it a waste of resources to deploy 3 big PostgreSQL > instances just for HA where 2 suffice even if they deploy 3 > lightweight DCS instances. Having only some of the nodes act as DCS > and others purely PostgreSQL nodes will reduce waste of resources. > > The experience of other projects/products with automated failover based on quorum shows that this is a critical issue for adoption. In the In-memory Data Grid space (Coherence, Geode/GemFire) the question of how to ensure that some nodes didn't carry any data comes up early in many architecture discussions. When RabbitMQ shipped their Quorum Queues feature, the first and hardest area of pushback was around all nodes hosting message content. It's not just about the requirement for compute resources, it's also about bandwidth and latency. Many large organisations have, for historical reasons, pairs of data centres with very good point-to-point connectivity. As the requirement for quorum witnesses has come up for all sorts of things, including storage arrays, they have built arbiter/witness sites at branches, colocation providers or even on the public cloud. More than not holding user data or processing queries, the arbiter can't even be sent the replication stream for the user data in the database, it just won't fit down the pipe. Which feels like a very difficult requirement to meet if the replication model for all data is being changed to a quorum model. Regards Alastair ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Built-in Raft replication @ 2025-04-16 14:07 Konstantin Osipov <[email protected]> parent: Alastair Turner <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Konstantin Osipov @ 2025-04-16 14:07 UTC (permalink / raw) To: Alastair Turner <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; Andrey Borodin <[email protected]>; Tom Lane <[email protected]>; Greg Sabino Mullane <[email protected]>; Nikolay Samokhvalov <[email protected]>; PostgreSQL Hackers <[email protected]> * Alastair Turner <[email protected]> [25/04/16 15:58]: > > > If you use build-in failover you have to resort to 3 big Postgres > > machines because you need 2/3 majority. Of course, you can install > > MySQL-stype arbiter - host that had no real PGDATA, only participates in > > voting. But this is a solution to problem induced by built-in autofailover. > > > > Users find it a waste of resources to deploy 3 big PostgreSQL > > instances just for HA where 2 suffice even if they deploy 3 > > lightweight DCS instances. Having only some of the nodes act as DCS > > and others purely PostgreSQL nodes will reduce waste of resources. > > > > The experience of other projects/products with automated failover based on > quorum shows that this is a critical issue for adoption. In the In-memory > Data Grid space (Coherence, Geode/GemFire) the question of how to ensure > that some nodes didn't carry any data comes up early in many architecture > discussions. When RabbitMQ shipped their Quorum Queues feature, the first > and hardest area of pushback was around all nodes hosting message content. > > It's not just about the requirement for compute resources, it's also about > bandwidth and latency. Many large organisations have, for historical > reasons, pairs of data centres with very good point-to-point connectivity. > As the requirement for quorum witnesses has come up for all sorts of > things, including storage arrays, they have built arbiter/witness sites at > branches, colocation providers or even on the public cloud. More than not > holding user data or processing queries, the arbiter can't even be sent the > replication stream for the user data in the database, it just won't fit > down the pipe. > > Which feels like a very difficult requirement to meet if the replication > model for all data is being changed to a quorum model. I agree master/replica deployment layouts are very popular and are not going to directly benefit from raft. They'll still work, but no automation will be available, just like today with Patroni. However, if the storage cost is an argument, then the logical path is to disaggregate storage/compute altogether, i.e. use projects like neon. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Built-in Raft replication @ 2025-04-16 19:29 Greg Sabino Mullane <[email protected]> parent: Ashutosh Bapat <[email protected]> 2 siblings, 1 reply; 10+ messages in thread From: Greg Sabino Mullane @ 2025-04-16 19:29 UTC (permalink / raw) To: Ashutosh Bapat <[email protected]>; +Cc: Andrey Borodin <[email protected]>; Tom Lane <[email protected]>; Konstantin Osipov <[email protected]>; Nikolay Samokhvalov <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Apr 16, 2025 at 2:18 AM Ashutosh Bapat <[email protected]> wrote: > Users find it a waste of resources to deploy 3 big PostgreSQL instances > just for HA where 2 suffice even if they deploy 3 lightweight DCS > instances. Having only some of the nodes act as DCS and others purely > PostgreSQL nodes will reduce waste of resources. > A big problem is that putting your DCS into Postgres means your whole system is now super-sensitive to IO/WAL-streaming issues, and a busy database doing database stuff is going to start affecting the DCS stuff. With three lightweight DCS servers, you don't really need to worry about how stressed the database servers are. In that way, I feel etcd et al. are adhering to the unix philosophy of "do one thing, and do it well." Cheers, Greg -- Crunchy Data - https://www.crunchydata.com Enterprise Postgres Software Products & Tech Support ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Built-in Raft replication @ 2025-04-16 19:35 Konstantin Osipov <[email protected]> parent: Greg Sabino Mullane <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Konstantin Osipov @ 2025-04-16 19:35 UTC (permalink / raw) To: Greg Sabino Mullane <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; Andrey Borodin <[email protected]>; Tom Lane <[email protected]>; Nikolay Samokhvalov <[email protected]>; PostgreSQL Hackers <[email protected]> * Greg Sabino Mullane <[email protected]> [25/04/16 22:33]: > > Users find it a waste of resources to deploy 3 big PostgreSQL instances > > just for HA where 2 suffice even if they deploy 3 lightweight DCS > > instances. Having only some of the nodes act as DCS and others purely > > PostgreSQL nodes will reduce waste of resources. > > > > A big problem is that putting your DCS into Postgres means your whole > system is now super-sensitive to IO/WAL-streaming issues, and a busy > database doing database stuff is going to start affecting the DCS stuff. Affecting in what way? Do you have a scenario in mind where an external state provider would act differently (better)? > With three lightweight DCS servers, you don't really need to worry about > how stressed the database servers are. In that way, I feel etcd et al. are > adhering to the unix philosophy of "do one thing, and do it well." -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Built-in Raft replication @ 2025-04-16 21:45 Alastair Turner <[email protected]> parent: Konstantin Osipov <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Alastair Turner @ 2025-04-16 21:45 UTC (permalink / raw) To: Konstantin Osipov <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; Andrey Borodin <[email protected]>; Tom Lane <[email protected]>; Greg Sabino Mullane <[email protected]>; Nikolay Samokhvalov <[email protected]>; PostgreSQL Hackers <[email protected]> Hi Konstantin On Wed, 16 Apr 2025 at 15:07, Konstantin Osipov <[email protected]> wrote: > * Alastair Turner <[email protected]> [25/04/16 15:58]: > > > > > If you use build-in failover you have to resort to 3 big Postgres > > > machines because you need 2/3 majority. Of course, you can install > > > MySQL-stype arbiter - host that had no real PGDATA, only participates > in > > > voting. But this is a solution to problem induced by built-in > autofailover. > > > > > > Users find it a waste of resources to deploy 3 big PostgreSQL > > > instances just for HA where 2 suffice even if they deploy 3 > > > lightweight DCS instances. Having only some of the nodes act as DCS > > > and others purely PostgreSQL nodes will reduce waste of resources. > > > > > > The experience of other projects/products with automated failover > based on > > quorum shows that this is a critical issue for adoption. In the In-memory > > Data Grid space (Coherence, Geode/GemFire) the question of how to ensure > > that some nodes didn't carry any data comes up early in many architecture > > discussions. When RabbitMQ shipped their Quorum Queues feature, the first > > and hardest area of pushback was around all nodes hosting message > content. > > > > It's not just about the requirement for compute resources, it's also > about > > bandwidth and latency. Many large organisations have, for historical > > reasons, pairs of data centres with very good point-to-point > connectivity. > > As the requirement for quorum witnesses has come up for all sorts of > > things, including storage arrays, they have built arbiter/witness sites > at > > branches, colocation providers or even on the public cloud. More than not > > holding user data or processing queries, the arbiter can't even be sent > the > > replication stream for the user data in the database, it just won't fit > > down the pipe. > > > > Which feels like a very difficult requirement to meet if the replication > > model for all data is being changed to a quorum model. > > I agree master/replica deployment layouts are very popular and are > not going to directly benefit from raft. They'll still work, but no > automation will be available, just like today with Patroni. > > Users of Patroni and etcd setups can get automation for two-site primary/replica pairs by putting a third etcd node on a third site. Which only requires moving the membership/leadership data to the arbiter site, not all database activity. > However, if the storage cost is an argument, then the logical path is to > disaggregate storage/compute altogether, i.e. use projects like > neon. > > The issue is not generally storage, but network. There may simply not be enough bandwidth available to transmit the whole WAL to the arbiter site. Many on-premises IT setups have this limitation in some form. If your proposal would leave these large, traditional user organisations (which account for thousands of Postgres HA pairs or DR pairs) doing what they currently do with wraparound tooling like Patroni, and create a new, in core, option for balanced 3, 5, 7... member groups, then I don't think it's worth doing. Regards, Alastair ^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2025-04-16 21:45 UTC | newest] Thread overview: 10+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-08-17 03:52 [PATCH v10] Avoid creating archive status ".ready" files too early. Nathan Bossart <[email protected]> 2025-04-16 04:58 Re: Built-in Raft replication Andrey Borodin <[email protected]> 2025-04-16 06:18 ` Re: Built-in Raft replication Ashutosh Bapat <[email protected]> 2025-04-16 06:27 ` Re: Built-in Raft replication Andrey Borodin <[email protected]> 2025-04-16 06:45 ` Re: Built-in Raft replication Ashutosh Bapat <[email protected]> 2025-04-16 12:53 ` Re: Built-in Raft replication Alastair Turner <[email protected]> 2025-04-16 14:07 ` Re: Built-in Raft replication Konstantin Osipov <[email protected]> 2025-04-16 21:45 ` Re: Built-in Raft replication Alastair Turner <[email protected]> 2025-04-16 19:29 ` Re: Built-in Raft replication Greg Sabino Mullane <[email protected]> 2025-04-16 19:35 ` Re: Built-in Raft replication Konstantin Osipov <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox