agora inbox for [email protected]
help / color / mirror / Atom feed[PATCH v13 3/6] snapshot scalability: Introduce dense array of in-progress xids.
25+ messages / 6 participants
[nested] [flat]
* [PATCH v13 3/6] snapshot scalability: Introduce dense array of in-progress xids.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
The new array contains the xids for all connected backends / in-use
PGPROC entries in a dense manner (in contrast to the PGPROC/PGXACT
arrays which can have unused entries interspersed).
This improves performance because GetSnapshotData() always needs to
scan the xids of all live procarray entries and now there's no need to
go through the procArray->pgprocnos indirection anymore.
As the set of running top-level xids changes rarely, compared to the
number of snapshots taken, this substantially increases the likelihood
of most data required for a snapshot being in l2 cache. In
read-mostly workloads scanning the xids[] array will sufficient to
build a snapshot, as most backends will not have an xid assigned.
To keep the xid array dense ProcArrayRemove() needs to move entries
behind the to-be-removed proc's one further up in the array. Obviously
moving array entries cannot happen while a backend sets it
xid. I.e. locking needs to prevent that array entries are moved while
a backend modifies its xid.
To avoid locking ProcArrayLock in GetNewTransactionId() - a fairly hot
spot already - ProcArrayAdd() / ProcArrayRemove() now needs to hold
XidGenLock in addition to ProcArrayLock. Adding / Removing a procarray
entry is not a very frequent operation, even taking 2PC into account.
Due to the above, the dense array entries can only be read or modified
while holding ProcArrayLock and/or XidGenLock. This prevents a
concurrent ProcArrayRemove() from shifting the dense array while it is
accessed concurrently.
While the new dense array is very good when needing to look at all
xids it is less suitable when accessing a single backend's xid. In
particular it would be problematic to have to acquire a lock to access
a backend's own xid. Therefore a backend's xid is not just stored in
the dense array, but also in PGPROC. This also allows a backend to
only access the shared xid value when the backend had acquired an
xid.
The infrastructure added in this commit will be used for the remaining
PGXACT fields in subsequent commits. They are kept separate to make
review easier.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 79 +++++-
src/backend/access/heap/heapam_visibility.c | 8 +-
src/backend/access/transam/README | 33 +--
src/backend/access/transam/clog.c | 8 +-
src/backend/access/transam/twophase.c | 31 +--
src/backend/access/transam/varsup.c | 20 +-
src/backend/commands/vacuum.c | 2 +-
src/backend/storage/ipc/procarray.c | 282 +++++++++++++-------
src/backend/storage/ipc/sinvaladt.c | 4 +-
src/backend/storage/lmgr/lock.c | 3 +-
src/backend/storage/lmgr/proc.c | 26 +-
11 files changed, 335 insertions(+), 161 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 5e4b028a5f9..146bca84bd6 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -89,6 +89,17 @@ typedef enum
* distinguished from a real one at need by the fact that it has pid == 0.
* The semaphore and lock-activity fields in a prepared-xact PGPROC are unused,
* but its myProcLocks[] lists are valid.
+ *
+ * Mirrored fields:
+ *
+ * Some fields in PGPROC (see "mirrored in ..." comment) are mirrored into an
+ * element of more densely packed ProcGlobal arrays. These arrays are indexed
+ * by PGPROC->pgxactoff. Both copies need to be maintained coherently.
+ *
+ * NB: The pgxactoff indexed value can *never* be accessed without holding
+ * locks.
+ *
+ * See PROC_HDR for details.
*/
struct PGPROC
{
@@ -101,6 +112,12 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+
+ TransactionId xid; /* id of top-level transaction currently being
+ * executed by this proc, if running and XID
+ * is assigned; else InvalidTransactionId.
+ * mirrored in ProcGlobal->xids[pgxactoff] */
+
TransactionId xmin; /* minimal running XID as it was when we were
* starting our xact, excluding LAZY VACUUM:
* vacuum must not remove tuples deleted by
@@ -110,6 +127,9 @@ struct PGPROC
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
int pid; /* Backend's process ID; 0 if prepared xact */
+
+ int pgxactoff; /* offset into various ProcGlobal->arrays
+ * with data mirrored from this PGPROC */
int pgprocno;
/* These fields are zero while a backend is still starting up: */
@@ -224,10 +244,6 @@ extern PGDLLIMPORT struct PGXACT *MyPgXact;
*/
typedef struct PGXACT
{
- TransactionId xid; /* id of top-level transaction currently being
- * executed by this proc, if running and XID
- * is assigned; else InvalidTransactionId */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
@@ -236,6 +252,57 @@ typedef struct PGXACT
/*
* There is one ProcGlobal struct for the whole database cluster.
+ *
+ * Adding/Removing an entry into the procarray requires holding *both*
+ * ProcArrayLock and XidGenLock in exclusive mode (in that order). Both are
+ * needed because the dense arrays (see below) are accessed from
+ * GetNewTransactionId() and GetSnapshotData(), and we don't want to add
+ * further contention by both using the same lock. Adding/Removing a procarray
+ * entry is much less frequent.
+ *
+ * Some fields in PGPROC are mirrored into more densely packed arrays (like
+ * xids), with one entry for each backend. These arrays only contain entries
+ * for PGPROCs that have been added to the shared array with ProcArrayAdd()
+ * (in contrast to PGPROC array which has unused PGPROCs interspersed).
+ *
+ * The dense arrays are indexed indexed by PGPROC->pgxactoff. Any concurrent
+ * ProcArrayAdd() / ProcArrayRemove() can lead to pgxactoff of a procarray
+ * member to change. Therefore it is only safe to use PGPROC->pgxactoff to
+ * access the dense array while holding either ProcArrayLock or XidGenLock.
+ *
+ * As long as a PGPROC is in the procarray, the mirrored values need to be
+ * maintained in both places in a coherent manner.
+ *
+ * The denser separate arrays are beneficial for three main reasons: First, to
+ * allow for as tight loops accessing the data as possible. Second, to prevent
+ * updates of frequently changing data (e.g. xmin) from invalidating
+ * cachelines also containing less frequently changing data (e.g. xid,
+ * vacuumFlags). Third to condense frequently accessed data into as few
+ * cachelines as possible.
+ *
+ * There are two main reasons to have the data mirrored between these dense
+ * arrays and PGPROC. First, as explained above, a PGPROC's array entries can
+ * only be accessed with either ProcArrayLock or XidGenLock held, whereas the
+ * PGPROC entries do not require that (obviously there may still be locking
+ * requirements around the individual field, separate from the concerns
+ * here). That is particularly important for a backend to efficiently checks
+ * it own values, which it often can safely do without locking. Second, the
+ * PGPROC fields allow to avoid unnecessary accesses and modification to the
+ * dense arrays. A backend's own PGPROC is more likely to be in a local cache,
+ * whereas the cachelines for the dense array will be modified by other
+ * backends (often removing it from the cache for other cores/sockets). At
+ * commit/abort time a check of the PGPROC value can avoid accessing/dirtying
+ * the corresponding array value.
+ *
+ * Basically it makes sense to access the PGPROC variable when checking a
+ * single backend's data, especially when already looking at the PGPROC for
+ * other reasons already. It makes sense to look at the "dense" arrays if we
+ * need to look at many / most entries, because we then benefit from the
+ * reduced indirection and better cross-process cache-ability.
+ *
+ * When entering a PGPROC for 2PC transactions with ProcArrayAdd(), the data
+ * in the dense arrays is initialized from the PGPROC while it already holds
+ * ProcArrayLock.
*/
typedef struct PROC_HDR
{
@@ -243,6 +310,10 @@ typedef struct PROC_HDR
PGPROC *allProcs;
/* Array of PGXACT structures (not including dummies for prepared txns) */
PGXACT *allPgXact;
+
+ /* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
+ TransactionId *xids;
+
/* Length of allProcs array */
uint32 allProcCount;
/* Head of list of free PGPROC structures */
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index f117ee160a3..6dec0c8311b 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -11,12 +11,12 @@
* shared buffer content lock on the buffer containing the tuple.
*
* NOTE: When using a non-MVCC snapshot, we must check
- * TransactionIdIsInProgress (which looks in the PGXACT array)
+ * TransactionIdIsInProgress (which looks in the PGPROC array)
* before TransactionIdDidCommit/TransactionIdDidAbort (which look in
* pg_xact). Otherwise we have a race condition: we might decide that a
* just-committed transaction crashed, because none of the tests succeed.
* xact.c is careful to record commit/abort in pg_xact before it unsets
- * MyPgXact->xid in the PGXACT array. That fixes that problem, but it
+ * MyProc->xid in the PGPROC array. That fixes that problem, but it
* also means there is a window where TransactionIdIsInProgress and
* TransactionIdDidCommit will both return true. If we check only
* TransactionIdDidCommit, we could consider a tuple committed when a
@@ -956,7 +956,7 @@ HeapTupleSatisfiesDirty(HeapTuple htup, Snapshot snapshot,
* coding where we tried to set the hint bits as soon as possible, we instead
* did TransactionIdIsInProgress in each call --- to no avail, as long as the
* inserting/deleting transaction was still running --- which was more cycles
- * and more contention on the PGXACT array.
+ * and more contention on ProcArrayLock.
*/
static bool
HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot,
@@ -1445,7 +1445,7 @@ HeapTupleSatisfiesNonVacuumable(HeapTuple htup, Snapshot snapshot,
* HeapTupleSatisfiesMVCC) and, therefore, any hint bits that can be set
* should already be set. We assume that if no hint bits are set, the xmin
* or xmax transaction is still running. This is therefore faster than
- * HeapTupleSatisfiesVacuum, because we don't consult PGXACT nor CLOG.
+ * HeapTupleSatisfiesVacuum, because we consult neither procarray nor CLOG.
* It's okay to return false when in doubt, but we must return true only
* if the tuple is removable.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index c15b5540a09..2d7979011ce 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -251,10 +251,10 @@ enforce, and it assists with some other issues as explained below.) The
implementation of this is that GetSnapshotData takes the ProcArrayLock in
shared mode (so that multiple backends can take snapshots in parallel),
but ProcArrayEndTransaction must take the ProcArrayLock in exclusive mode
-while clearing MyPgXact->xid at transaction end (either commit or abort).
-(To reduce context switching, when multiple transactions commit nearly
-simultaneously, we have one backend take ProcArrayLock and clear the XIDs
-of multiple processes at once.)
+while clearing the ProcGlobal->xids[] entry at transaction end (either
+commit or abort). (To reduce context switching, when multiple transactions
+commit nearly simultaneously, we have one backend take ProcArrayLock and
+clear the XIDs of multiple processes at once.)
ProcArrayEndTransaction also holds the lock while advancing the shared
latestCompletedXid variable. This allows GetSnapshotData to use
@@ -278,12 +278,13 @@ present in the ProcArray, or not running anymore. (This guarantee doesn't
apply to subtransaction XIDs, because of the possibility that there's not
room for them in the subxid array; instead we guarantee that they are
present or the overflow flag is set.) If a backend released XidGenLock
-before storing its XID into MyPgXact, then it would be possible for another
-backend to allocate and commit a later XID, causing latestCompletedXid to
-pass the first backend's XID, before that value became visible in the
-ProcArray. That would break GetOldestXmin, as discussed below.
+before storing its XID into ProcGlobal->xids[], then it would be possible
+for another backend to allocate and commit a later XID, causing
+latestCompletedXid to pass the first backend's XID, before that value
+became visible in the ProcArray. That would break ComputeXidHorizons,
+as discussed below.
-We allow GetNewTransactionId to store the XID into MyPgXact->xid (or the
+We allow GetNewTransactionId to store the XID into ProcGlobal->xids[] (or the
subxid array) without taking ProcArrayLock. This was once necessary to
avoid deadlock; while that is no longer the case, it's still beneficial for
performance. We are thereby relying on fetch/store of an XID to be atomic,
@@ -382,13 +383,13 @@ Top-level transactions do not have a parent, so they leave their pg_subtrans
entries set to the default value of zero (InvalidTransactionId).
pg_subtrans is used to check whether the transaction in question is still
-running --- the main Xid of a transaction is recorded in the PGXACT struct,
-but since we allow arbitrary nesting of subtransactions, we can't fit all Xids
-in shared memory, so we have to store them on disk. Note, however, that for
-each transaction we keep a "cache" of Xids that are known to be part of the
-transaction tree, so we can skip looking at pg_subtrans unless we know the
-cache has been overflowed. See storage/ipc/procarray.c for the gory details.
-
+running --- the main Xid of a transaction is recorded in ProcGlobal->xids[],
+with a copy in PGPROC->xid, but since we allow arbitrary nesting of
+subtransactions, we can't fit all Xids in shared memory, so we have to store
+them on disk. Note, however, that for each transaction we keep a "cache" of
+Xids that are known to be part of the transaction tree, so we can skip looking
+at pg_subtrans unless we know the cache has been overflowed. See
+storage/ipc/procarray.c for the gory details.
slru.c is the supporting mechanism for both pg_xact and pg_subtrans. It
implements the LRU policy for in-memory buffer pages. The high-level routines
for pg_xact are implemented in transam.c, while the low-level functions are in
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index dd2f4d5bc7e..a4599e96610 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -285,15 +285,15 @@ TransactionIdSetPageStatus(TransactionId xid, int nsubxids,
* updates for multiple backends so that the number of times XactSLRULock
* needs to be acquired is reduced.
*
- * For this optimization to be safe, the XID in MyPgXact and the subxids
- * in MyProc must be the same as the ones for which we're setting the
- * status. Check that this is the case.
+ * For this optimization to be safe, the XID and subxids in MyProc must be
+ * the same as the ones for which we're setting the status. Check that
+ * this is the case.
*
* For this optimization to be efficient, we shouldn't have too many
* sub-XIDs and all of the XIDs for which we're adjusting clog should be
* on the same page. Check those conditions, too.
*/
- if (all_xact_same_page && xid == MyPgXact->xid &&
+ if (all_xact_same_page && xid == MyProc->xid &&
nsubxids <= THRESHOLD_SUBTRANS_CLOG_OPT &&
nsubxids == MyPgXact->nxids &&
memcmp(subxids, MyProc->subxids.xids,
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index eb5f4680a3d..a0398bf3a3e 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -351,7 +351,7 @@ AtAbort_Twophase(void)
/*
* This is called after we have finished transferring state to the prepared
- * PGXACT entry.
+ * PGPROC entry.
*/
void
PostPrepare_Twophase(void)
@@ -463,7 +463,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->waitStatus = PROC_WAIT_STATUS_OK;
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
- pgxact->xid = xid;
+ proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
@@ -768,7 +768,6 @@ pg_prepared_xact(PG_FUNCTION_ARGS)
{
GlobalTransaction gxact = &status->array[status->currIdx++];
PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
- PGXACT *pgxact = &ProcGlobal->allPgXact[gxact->pgprocno];
Datum values[5];
bool nulls[5];
HeapTuple tuple;
@@ -783,7 +782,7 @@ pg_prepared_xact(PG_FUNCTION_ARGS)
MemSet(values, 0, sizeof(values));
MemSet(nulls, 0, sizeof(nulls));
- values[0] = TransactionIdGetDatum(pgxact->xid);
+ values[0] = TransactionIdGetDatum(proc->xid);
values[1] = CStringGetTextDatum(gxact->gid);
values[2] = TimestampTzGetDatum(gxact->prepared_at);
values[3] = ObjectIdGetDatum(gxact->owner);
@@ -829,9 +828,8 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
{
GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
- PGXACT *pgxact = &ProcGlobal->allPgXact[gxact->pgprocno];
- if (pgxact->xid == xid)
+ if (gxact->xid == xid)
{
result = gxact;
break;
@@ -987,8 +985,7 @@ void
StartPrepare(GlobalTransaction gxact)
{
PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
- PGXACT *pgxact = &ProcGlobal->allPgXact[gxact->pgprocno];
- TransactionId xid = pgxact->xid;
+ TransactionId xid = gxact->xid;
TwoPhaseFileHeader hdr;
TransactionId *children;
RelFileNode *commitrels;
@@ -1140,15 +1137,15 @@ EndPrepare(GlobalTransaction gxact)
/*
* Mark the prepared transaction as valid. As soon as xact.c marks
- * MyPgXact as not running our XID (which it will do immediately after
+ * MyProc as not running our XID (which it will do immediately after
* this function returns), others can commit/rollback the xact.
*
* NB: a side effect of this is to make a dummy ProcArray entry for the
- * prepared XID. This must happen before we clear the XID from MyPgXact,
- * else there is a window where the XID is not running according to
- * TransactionIdIsInProgress, and onlookers would be entitled to assume
- * the xact crashed. Instead we have a window where the same XID appears
- * twice in ProcArray, which is OK.
+ * prepared XID. This must happen before we clear the XID from MyProc /
+ * ProcGlobal->xids[], else there is a window where the XID is not running
+ * according to TransactionIdIsInProgress, and onlookers would be entitled
+ * to assume the xact crashed. Instead we have a window where the same
+ * XID appears twice in ProcArray, which is OK.
*/
MarkAsPrepared(gxact, false);
@@ -1404,7 +1401,6 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
{
GlobalTransaction gxact;
PGPROC *proc;
- PGXACT *pgxact;
TransactionId xid;
char *buf;
char *bufptr;
@@ -1423,8 +1419,7 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
*/
gxact = LockGXact(gid, GetUserId());
proc = &ProcGlobal->allProcs[gxact->pgprocno];
- pgxact = &ProcGlobal->allPgXact[gxact->pgprocno];
- xid = pgxact->xid;
+ xid = gxact->xid;
/*
* Read and validate 2PC state data. State data will typically be stored
@@ -1726,7 +1721,7 @@ CheckPointTwoPhase(XLogRecPtr redo_horizon)
for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
{
/*
- * Note that we are using gxact not pgxact so this works in recovery
+ * Note that we are using gxact not PGPROC so this works in recovery
* also
*/
GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index 2ef0f4991ca..4c91b343ecd 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -38,7 +38,8 @@ VariableCache ShmemVariableCache = NULL;
* Allocate the next FullTransactionId for a new transaction or
* subtransaction.
*
- * The new XID is also stored into MyPgXact before returning.
+ * The new XID is also stored into MyProc->xid/ProcGlobal->xids[] before
+ * returning.
*
* Note: when this is called, we are actually already inside a valid
* transaction, since XIDs are now not allocated until the transaction
@@ -65,7 +66,8 @@ GetNewTransactionId(bool isSubXact)
if (IsBootstrapProcessingMode())
{
Assert(!isSubXact);
- MyPgXact->xid = BootstrapTransactionId;
+ MyProc->xid = BootstrapTransactionId;
+ ProcGlobal->xids[MyProc->pgxactoff] = BootstrapTransactionId;
return FullTransactionIdFromEpochAndXid(0, BootstrapTransactionId);
}
@@ -190,10 +192,10 @@ GetNewTransactionId(bool isSubXact)
* latestCompletedXid is present in the ProcArray, which is essential for
* correct OldestXmin tracking; see src/backend/access/transam/README.
*
- * Note that readers of PGXACT xid fields should be careful to fetch the
- * value only once, rather than assume they can read a value multiple
- * times and get the same answer each time. Note we are assuming that
- * TransactionId and int fetch/store are atomic.
+ * Note that readers of ProcGlobal->xids/PGPROC->xid should be careful
+ * to fetch the value for each proc only once, rather than assume they can
+ * read a value multiple times and get the same answer each time. Note we
+ * are assuming that TransactionId and int fetch/store are atomic.
*
* The same comments apply to the subxact xid count and overflow fields.
*
@@ -219,7 +221,11 @@ GetNewTransactionId(bool isSubXact)
* answer later on when someone does have a reason to inquire.)
*/
if (!isSubXact)
- MyPgXact->xid = xid; /* LWLockRelease acts as barrier */
+ {
+ /* LWLockRelease acts as barrier */
+ MyProc->xid = xid;
+ ProcGlobal->xids[MyProc->pgxactoff] = xid;
+ }
else
{
int nxids = MyPgXact->nxids;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 22228f5684f..648e12c78d8 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1724,7 +1724,7 @@ vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params)
*
* Note: these flags remain set until CommitTransaction or
* AbortTransaction. We don't want to clear them until we reset
- * MyPgXact->xid/xmin, otherwise GetOldestNonRemovableTransactionId()
+ * MyProc->xid/xmin, otherwise GetOldestNonRemovableTransactionId()
* might appear to go backwards, which is probably Not Good.
*/
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index a016816ae86..e5617ddfb41 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -9,8 +9,9 @@
* one is as a means of determining the set of currently running transactions.
*
* Because of various subtle race conditions it is critical that a backend
- * hold the correct locks while setting or clearing its MyPgXact->xid field.
- * See notes in src/backend/access/transam/README.
+ * hold the correct locks while setting or clearing its xid (in
+ * ProcGlobal->xids[]/MyProc->xid). See notes in
+ * src/backend/access/transam/README.
*
* The process arrays now also include structures representing prepared
* transactions. The xid and subxids fields of these are valid, as are the
@@ -436,7 +437,9 @@ ProcArrayAdd(PGPROC *proc)
ProcArrayStruct *arrayP = procArray;
int index;
+ /* See ProcGlobal comment explaining why both locks are held */
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
+ LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
if (arrayP->numProcs >= arrayP->maxProcs)
{
@@ -445,7 +448,6 @@ ProcArrayAdd(PGPROC *proc)
* fixed supply of PGPROC structs too, and so we should have failed
* earlier.)
*/
- LWLockRelease(ProcArrayLock);
ereport(FATAL,
(errcode(ERRCODE_TOO_MANY_CONNECTIONS),
errmsg("sorry, too many clients already")));
@@ -471,10 +473,25 @@ ProcArrayAdd(PGPROC *proc)
}
memmove(&arrayP->pgprocnos[index + 1], &arrayP->pgprocnos[index],
- (arrayP->numProcs - index) * sizeof(int));
+ (arrayP->numProcs - index) * sizeof(*arrayP->pgprocnos));
+ memmove(&ProcGlobal->xids[index + 1], &ProcGlobal->xids[index],
+ (arrayP->numProcs - index) * sizeof(*ProcGlobal->xids));
+
arrayP->pgprocnos[index] = proc->pgprocno;
+ ProcGlobal->xids[index] = proc->xid;
+
arrayP->numProcs++;
+ for (; index < arrayP->numProcs; index++)
+ {
+ allProcs[arrayP->pgprocnos[index]].pgxactoff = index;
+ }
+
+ /*
+ * Release in reversed acquisition order, to reduce frequency of having to
+ * wait for XidGenLock while holding ProcArrayLock.
+ */
+ LWLockRelease(XidGenLock);
LWLockRelease(ProcArrayLock);
}
@@ -500,36 +517,59 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
DisplayXidCache();
#endif
+ /* See ProcGlobal comment explaining why both locks are held */
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
+ LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[proc->pgxactoff]].pgxactoff == proc->pgxactoff);
if (TransactionIdIsValid(latestXid))
{
- Assert(TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
+ Assert(TransactionIdIsValid(ProcGlobal->xids[proc->pgxactoff]));
/* Advance global latestCompletedXid while holding the lock */
MaintainLatestCompletedXid(latestXid);
+
+ ProcGlobal->xids[proc->pgxactoff] = 0;
}
else
{
/* Shouldn't be trying to remove a live transaction here */
- Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
+ Assert(!TransactionIdIsValid(ProcGlobal->xids[proc->pgxactoff]));
}
+ Assert(TransactionIdIsValid(ProcGlobal->xids[proc->pgxactoff] == 0));
+
for (index = 0; index < arrayP->numProcs; index++)
{
if (arrayP->pgprocnos[index] == proc->pgprocno)
{
/* Keep the PGPROC array sorted. See notes above */
memmove(&arrayP->pgprocnos[index], &arrayP->pgprocnos[index + 1],
- (arrayP->numProcs - index - 1) * sizeof(int));
+ (arrayP->numProcs - index - 1) * sizeof(*arrayP->pgprocnos));
+ memmove(&ProcGlobal->xids[index], &ProcGlobal->xids[index + 1],
+ (arrayP->numProcs - index - 1) * sizeof(*ProcGlobal->xids));
+
arrayP->pgprocnos[arrayP->numProcs - 1] = -1; /* for debugging */
arrayP->numProcs--;
+
+ for (; index < arrayP->numProcs; index++)
+ {
+ allProcs[arrayP->pgprocnos[index]].pgxactoff--;
+ }
+
+ /*
+ * Release in reversed acquisition order, to reduce frequency of
+ * having to wait for XidGenLock while holding ProcArrayLock.
+ */
+ LWLockRelease(XidGenLock);
LWLockRelease(ProcArrayLock);
return;
}
}
/* Oops */
+ LWLockRelease(XidGenLock);
LWLockRelease(ProcArrayLock);
elog(LOG, "failed to find proc %p in ProcArray", proc);
@@ -562,7 +602,7 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
* else is taking a snapshot. See discussion in
* src/backend/access/transam/README.
*/
- Assert(TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
+ Assert(TransactionIdIsValid(proc->xid));
/*
* If we can immediately acquire ProcArrayLock, we clear our own XID
@@ -584,7 +624,7 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
* anyone else's calculation of a snapshot. We might change their
* estimate of global xmin, but that's OK.
*/
- Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
+ Assert(!TransactionIdIsValid(proc->xid));
proc->lxid = InvalidLocalTransactionId;
/* must be cleared with xid/xmin: */
@@ -607,7 +647,13 @@ static inline void
ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
TransactionId latestXid)
{
- pgxact->xid = InvalidTransactionId;
+ size_t pgxactoff = proc->pgxactoff;
+
+ Assert(TransactionIdIsValid(ProcGlobal->xids[pgxactoff]));
+ Assert(ProcGlobal->xids[pgxactoff] == proc->xid);
+
+ ProcGlobal->xids[pgxactoff] = InvalidTransactionId;
+ proc->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
@@ -643,7 +689,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
uint32 wakeidx;
/* We should definitely have an XID to clear. */
- Assert(TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
+ Assert(TransactionIdIsValid(proc->xid));
/* Add ourselves to the list of processes needing a group XID clear. */
proc->procArrayGroupMember = true;
@@ -748,20 +794,28 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
* This is used after successfully preparing a 2-phase transaction. We are
* not actually reporting the transaction's XID as no longer running --- it
* will still appear as running because the 2PC's gxact is in the ProcArray
- * too. We just have to clear out our own PGXACT.
+ * too. We just have to clear out our own PGPROC.
*/
void
ProcArrayClearTransaction(PGPROC *proc)
{
PGXACT *pgxact = &allPgXact[proc->pgprocno];
+ size_t pgxactoff;
/*
- * We can skip locking ProcArrayLock here, because this action does not
- * actually change anyone's view of the set of running XIDs: our entry is
- * duplicate with the gxact that has already been inserted into the
- * ProcArray.
+ * We can skip locking ProcArrayLock exclusively here, because this action
+ * does not actually change anyone's view of the set of running XIDs: our
+ * entry is duplicate with the gxact that has already been inserted into
+ * the ProcArray. But need it in shared mode for pgproc->pgxactoff to stay
+ * the same.
*/
- pgxact->xid = InvalidTransactionId;
+ LWLockAcquire(ProcArrayLock, LW_SHARED);
+
+ pgxactoff = proc->pgxactoff;
+
+ ProcGlobal->xids[pgxactoff] = InvalidTransactionId;
+ proc->xid = InvalidTransactionId;
+
proc->lxid = InvalidLocalTransactionId;
proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
@@ -773,6 +827,8 @@ ProcArrayClearTransaction(PGPROC *proc)
/* Clear the subtransaction-XID cache too */
pgxact->nxids = 0;
pgxact->overflowed = false;
+
+ LWLockRelease(ProcArrayLock);
}
/*
@@ -1167,7 +1223,7 @@ ProcArrayApplyXidAssignment(TransactionId topxid,
* there are four possibilities for finding a running transaction:
*
* 1. The given Xid is a main transaction Id. We will find this out cheaply
- * by looking at the PGXACT struct for each backend.
+ * by looking at ProcGlobal->xids.
*
* 2. The given Xid is one of the cached subxact Xids in the PGPROC array.
* We can find this out cheaply too.
@@ -1176,26 +1232,28 @@ ProcArrayApplyXidAssignment(TransactionId topxid,
* if the Xid is running on the primary.
*
* 4. Search the SubTrans tree to find the Xid's topmost parent, and then see
- * if that is running according to PGXACT or KnownAssignedXids. This is the
- * slowest way, but sadly it has to be done always if the others failed,
- * unless we see that the cached subxact sets are complete (none have
+ * if that is running according to ProcGlobal->xids[] or KnownAssignedXids.
+ * This is the slowest way, but sadly it has to be done always if the others
+ * failed, unless we see that the cached subxact sets are complete (none have
* overflowed).
*
* ProcArrayLock has to be held while we do 1, 2, 3. If we save the top Xids
* while doing 1 and 3, we can release the ProcArrayLock while we do 4.
* This buys back some concurrency (and we can't retrieve the main Xids from
- * PGXACT again anyway; see GetNewTransactionId).
+ * ProcGlobal->xids[] again anyway; see GetNewTransactionId).
*/
bool
TransactionIdIsInProgress(TransactionId xid)
{
static TransactionId *xids = NULL;
+ static TransactionId *other_xids;
int nxids = 0;
ProcArrayStruct *arrayP = procArray;
TransactionId topxid;
TransactionId latestCompletedXid;
- int i,
- j;
+ int mypgxactoff;
+ size_t numProcs;
+ int j;
/*
* Don't bother checking a transaction older than RecentXmin; it could not
@@ -1250,6 +1308,8 @@ TransactionIdIsInProgress(TransactionId xid)
errmsg("out of memory")));
}
+ other_xids = ProcGlobal->xids;
+
LWLockAcquire(ProcArrayLock, LW_SHARED);
/*
@@ -1266,20 +1326,22 @@ TransactionIdIsInProgress(TransactionId xid)
}
/* No shortcuts, gotta grovel through the array */
- for (i = 0; i < arrayP->numProcs; i++)
+ mypgxactoff = MyProc->pgxactoff;
+ numProcs = arrayP->numProcs;
+ for (size_t pgxactoff = 0; pgxactoff < numProcs; pgxactoff++)
{
- int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
+ int pgprocno;
+ PGXACT *pgxact;
+ PGPROC *proc;
TransactionId pxid;
int pxids;
- /* Ignore my own proc --- dealt with it above */
- if (proc == MyProc)
+ /* Ignore ourselves --- dealt with it above */
+ if (pgxactoff == mypgxactoff)
continue;
/* Fetch xid just once - see GetNewTransactionId */
- pxid = UINT32_ACCESS_ONCE(pgxact->xid);
+ pxid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
if (!TransactionIdIsValid(pxid))
continue;
@@ -1304,8 +1366,12 @@ TransactionIdIsInProgress(TransactionId xid)
/*
* Step 2: check the cached child-Xids arrays
*/
+ pgprocno = arrayP->pgprocnos[pgxactoff];
+ pgxact = &allPgXact[pgprocno];
pxids = pgxact->nxids;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
+ pgprocno = arrayP->pgprocnos[pgxactoff];
+ proc = &allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1336,7 +1402,7 @@ TransactionIdIsInProgress(TransactionId xid)
*/
if (RecoveryInProgress())
{
- /* none of the PGXACT entries should have XIDs in hot standby mode */
+ /* none of the PGPROC entries should have XIDs in hot standby mode */
Assert(nxids == 0);
if (KnownAssignedXidExists(xid))
@@ -1391,7 +1457,7 @@ TransactionIdIsInProgress(TransactionId xid)
Assert(TransactionIdIsValid(topxid));
if (!TransactionIdEquals(topxid, xid))
{
- for (i = 0; i < nxids; i++)
+ for (int i = 0; i < nxids; i++)
{
if (TransactionIdEquals(xids[i], topxid))
return true;
@@ -1414,6 +1480,7 @@ TransactionIdIsActive(TransactionId xid)
{
bool result = false;
ProcArrayStruct *arrayP = procArray;
+ TransactionId *other_xids = ProcGlobal->xids;
int i;
/*
@@ -1429,11 +1496,10 @@ TransactionIdIsActive(TransactionId xid)
{
int pgprocno = arrayP->pgprocnos[i];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
TransactionId pxid;
/* Fetch xid just once - see GetNewTransactionId */
- pxid = UINT32_ACCESS_ONCE(pgxact->xid);
+ pxid = UINT32_ACCESS_ONCE(other_xids[i]);
if (!TransactionIdIsValid(pxid))
continue;
@@ -1519,6 +1585,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
ProcArrayStruct *arrayP = procArray;
TransactionId kaxmin;
bool in_recovery = RecoveryInProgress();
+ TransactionId *other_xids = ProcGlobal->xids;
/* inferred after ProcArrayLock is released */
h->catalog_oldest_nonremovable = InvalidTransactionId;
@@ -1562,7 +1629,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
TransactionId xmin;
/* Fetch xid just once - see GetNewTransactionId */
- xid = UINT32_ACCESS_ONCE(pgxact->xid);
+ xid = UINT32_ACCESS_ONCE(other_xids[pgprocno]);
xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
@@ -1853,14 +1920,17 @@ Snapshot
GetSnapshotData(Snapshot snapshot)
{
ProcArrayStruct *arrayP = procArray;
+ TransactionId *other_xids = ProcGlobal->xids;
TransactionId xmin;
TransactionId xmax;
- int index;
- int count = 0;
+ size_t count = 0;
int subcount = 0;
bool suboverflowed = false;
FullTransactionId latest_completed;
TransactionId oldestxid;
+ int mypgxactoff;
+ TransactionId myxid;
+
TransactionId replication_slot_xmin = InvalidTransactionId;
TransactionId replication_slot_catalog_xmin = InvalidTransactionId;
@@ -1905,6 +1975,10 @@ GetSnapshotData(Snapshot snapshot)
LWLockAcquire(ProcArrayLock, LW_SHARED);
latest_completed = ShmemVariableCache->latestCompletedXid;
+ mypgxactoff = MyProc->pgxactoff;
+ myxid = other_xids[mypgxactoff];
+ Assert(myxid == MyProc->xid);
+
oldestxid = ShmemVariableCache->oldestXid;
/* xmax is always latestCompletedXid + 1 */
@@ -1915,57 +1989,79 @@ GetSnapshotData(Snapshot snapshot)
/* initialize xmin calculation with xmax */
xmin = xmax;
+ /* take own xid into account, saves a check inside the loop */
+ if (TransactionIdIsNormal(myxid) && NormalTransactionIdPrecedes(myxid, xmin))
+ xmin = myxid;
+
snapshot->takenDuringRecovery = RecoveryInProgress();
if (!snapshot->takenDuringRecovery)
{
+ size_t numProcs = arrayP->numProcs;
+ TransactionId *xip = snapshot->xip;
int *pgprocnos = arrayP->pgprocnos;
- int numProcs;
/*
- * Spin over procArray checking xid, xmin, and subxids. The goal is
- * to gather all active xids, find the lowest xmin, and try to record
- * subxids.
+ * First collect set of pgxactoff/xids that need to be included in the
+ * snapshot.
*/
- numProcs = arrayP->numProcs;
- for (index = 0; index < numProcs; index++)
+ for (size_t pgxactoff = 0; pgxactoff < numProcs; pgxactoff++)
{
- int pgprocno = pgprocnos[index];
- PGXACT *pgxact = &allPgXact[pgprocno];
- TransactionId xid;
+ /* Fetch xid just once - see GetNewTransactionId */
+ TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
+ int pgprocno;
+ PGXACT *pgxact;
+ uint8 vacuumFlags;
+
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+
+ /*
+ * If the transaction has no XID assigned, we can skip it; it
+ * won't have sub-XIDs either.
+ */
+ if (likely(xid == InvalidTransactionId))
+ continue;
+
+ /*
+ * We don't include our own XIDs (if any) in the snapshot. It
+ * needs to be includeded in the xmin computation, but we did so
+ * outside the loop.
+ */
+ if (pgxactoff == mypgxactoff)
+ continue;
+
+ /*
+ * The only way we are able to get here with a non-normal xid
+ * is during bootstrap - with this backend using
+ * BootstrapTransactionId. But the above test should filter
+ * that out.
+ */
+ Assert(TransactionIdIsNormal(xid));
+
+ /*
+ * If the XID is >= xmax, we can skip it; such transactions will
+ * be treated as running anyway (and any sub-XIDs will also be >=
+ * xmax).
+ */
+ if (!NormalTransactionIdPrecedes(xid, xmax))
+ continue;
+
+ pgprocno = pgprocnos[pgxactoff];
+ pgxact = &allPgXact[pgprocno];
+ vacuumFlags = pgxact->vacuumFlags;
/*
* Skip over backends doing logical decoding which manages xmin
* separately (check below) and ones running LAZY VACUUM.
*/
- if (pgxact->vacuumFlags &
- (PROC_IN_LOGICAL_DECODING | PROC_IN_VACUUM))
+ if (vacuumFlags & (PROC_IN_LOGICAL_DECODING | PROC_IN_VACUUM))
continue;
- /* Fetch xid just once - see GetNewTransactionId */
- xid = UINT32_ACCESS_ONCE(pgxact->xid);
-
- /*
- * If the transaction has no XID assigned, we can skip it; it
- * won't have sub-XIDs either. If the XID is >= xmax, we can also
- * skip it; such transactions will be treated as running anyway
- * (and any sub-XIDs will also be >= xmax).
- */
- if (!TransactionIdIsNormal(xid)
- || !NormalTransactionIdPrecedes(xid, xmax))
- continue;
-
- /*
- * We don't include our own XIDs (if any) in the snapshot, but we
- * must include them in xmin.
- */
if (NormalTransactionIdPrecedes(xid, xmin))
xmin = xid;
- if (pgxact == MyPgXact)
- continue;
/* Add XID to snapshot. */
- snapshot->xip[count++] = xid;
+ xip[count++] = xid;
/*
* Save subtransaction XIDs if possible (if we've already
@@ -1988,9 +2084,9 @@ GetSnapshotData(Snapshot snapshot)
suboverflowed = true;
else
{
- int nxids = pgxact->nxids;
+ int nsubxids = pgxact->nxids;
- if (nxids > 0)
+ if (nsubxids > 0)
{
PGPROC *proc = &allProcs[pgprocno];
@@ -1998,8 +2094,8 @@ GetSnapshotData(Snapshot snapshot)
memcpy(snapshot->subxip + subcount,
(void *) proc->subxids.xids,
- nxids * sizeof(TransactionId));
- subcount += nxids;
+ nsubxids * sizeof(TransactionId));
+ subcount += nsubxids;
}
}
}
@@ -2131,6 +2227,7 @@ GetSnapshotData(Snapshot snapshot)
}
RecentXmin = xmin;
+ Assert(TransactionIdPrecedesOrEquals(TransactionXmin, RecentXmin));
snapshot->xmin = xmin;
snapshot->xmax = xmax;
@@ -2293,7 +2390,7 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
* GetRunningTransactionData -- returns information about running transactions.
*
* Similar to GetSnapshotData but returns more information. We include
- * all PGXACTs with an assigned TransactionId, even VACUUM processes and
+ * all PGPROCs with an assigned TransactionId, even VACUUM processes and
* prepared transactions.
*
* We acquire XidGenLock and ProcArrayLock, but the caller is responsible for
@@ -2308,7 +2405,7 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
* This is never executed during recovery so there is no need to look at
* KnownAssignedXids.
*
- * Dummy PGXACTs from prepared transaction are included, meaning that this
+ * Dummy PGPROCs from prepared transaction are included, meaning that this
* may return entries with duplicated TransactionId values coming from
* transaction finishing to prepare. Nothing is done about duplicated
* entries here to not hold on ProcArrayLock more than necessary.
@@ -2327,6 +2424,7 @@ GetRunningTransactionData(void)
static RunningTransactionsData CurrentRunningXactsData;
ProcArrayStruct *arrayP = procArray;
+ TransactionId *other_xids = ProcGlobal->xids;
RunningTransactions CurrentRunningXacts = &CurrentRunningXactsData;
TransactionId latestCompletedXid;
TransactionId oldestRunningXid;
@@ -2387,7 +2485,7 @@ GetRunningTransactionData(void)
TransactionId xid;
/* Fetch xid just once - see GetNewTransactionId */
- xid = UINT32_ACCESS_ONCE(pgxact->xid);
+ xid = UINT32_ACCESS_ONCE(other_xids[index]);
/*
* We don't need to store transactions that don't have a TransactionId
@@ -2484,7 +2582,7 @@ GetRunningTransactionData(void)
* GetOldestActiveTransactionId()
*
* Similar to GetSnapshotData but returns just oldestActiveXid. We include
- * all PGXACTs with an assigned TransactionId, even VACUUM processes.
+ * all PGPROCs with an assigned TransactionId, even VACUUM processes.
* We look at all databases, though there is no need to include WALSender
* since this has no effect on hot standby conflicts.
*
@@ -2499,6 +2597,7 @@ TransactionId
GetOldestActiveTransactionId(void)
{
ProcArrayStruct *arrayP = procArray;
+ TransactionId *other_xids = ProcGlobal->xids;
TransactionId oldestRunningXid;
int index;
@@ -2521,12 +2620,10 @@ GetOldestActiveTransactionId(void)
LWLockAcquire(ProcArrayLock, LW_SHARED);
for (index = 0; index < arrayP->numProcs; index++)
{
- int pgprocno = arrayP->pgprocnos[index];
- PGXACT *pgxact = &allPgXact[pgprocno];
TransactionId xid;
/* Fetch xid just once - see GetNewTransactionId */
- xid = UINT32_ACCESS_ONCE(pgxact->xid);
+ xid = UINT32_ACCESS_ONCE(other_xids[index]);
if (!TransactionIdIsNormal(xid))
continue;
@@ -2604,8 +2701,8 @@ GetOldestSafeDecodingTransactionId(bool catalogOnly)
* If we're not in recovery, we walk over the procarray and collect the
* lowest xid. Since we're called with ProcArrayLock held and have
* acquired XidGenLock, no entries can vanish concurrently, since
- * PGXACT->xid is only set with XidGenLock held and only cleared with
- * ProcArrayLock held.
+ * ProcGlobal->xids[i] is only set with XidGenLock held and only cleared
+ * with ProcArrayLock held.
*
* In recovery we can't lower the safe value besides what we've computed
* above, so we'll have to wait a bit longer there. We unfortunately can
@@ -2614,17 +2711,17 @@ GetOldestSafeDecodingTransactionId(bool catalogOnly)
*/
if (!recovery_in_progress)
{
+ TransactionId *other_xids = ProcGlobal->xids;
+
/*
- * Spin over procArray collecting all min(PGXACT->xid)
+ * Spin over procArray collecting min(ProcGlobal->xids[i])
*/
for (index = 0; index < arrayP->numProcs; index++)
{
- int pgprocno = arrayP->pgprocnos[index];
- PGXACT *pgxact = &allPgXact[pgprocno];
TransactionId xid;
/* Fetch xid just once - see GetNewTransactionId */
- xid = UINT32_ACCESS_ONCE(pgxact->xid);
+ xid = UINT32_ACCESS_ONCE(other_xids[index]);
if (!TransactionIdIsNormal(xid))
continue;
@@ -2812,6 +2909,7 @@ BackendXidGetPid(TransactionId xid)
{
int result = 0;
ProcArrayStruct *arrayP = procArray;
+ TransactionId *other_xids = ProcGlobal->xids;
int index;
if (xid == InvalidTransactionId) /* never match invalid xid */
@@ -2823,9 +2921,8 @@ BackendXidGetPid(TransactionId xid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
- if (pgxact->xid == xid)
+ if (other_xids[index] == xid)
{
result = proc->pid;
break;
@@ -3105,7 +3202,6 @@ MinimumActiveBackends(int min)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3122,7 +3218,7 @@ MinimumActiveBackends(int min)
continue; /* do not count deleted entries */
if (proc == MyProc)
continue; /* do not count myself */
- if (pgxact->xid == InvalidTransactionId)
+ if (proc->xid == InvalidTransactionId)
continue; /* do not count if no XID assigned */
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3548,8 +3644,8 @@ XidCacheRemoveRunningXids(TransactionId xid,
*
* Note that we do not have to be careful about memory ordering of our own
* reads wrt. GetNewTransactionId() here - only this process can modify
- * relevant fields of MyProc/MyPgXact. But we do have to be careful about
- * our own writes being well ordered.
+ * relevant fields of MyProc/ProcGlobal->xids[]. But we do have to be
+ * careful about our own writes being well ordered.
*/
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
@@ -3907,7 +4003,7 @@ FullXidRelativeTo(FullTransactionId rel, TransactionId xid)
* In Hot Standby mode, we maintain a list of transactions that are (or were)
* running on the primary at the current point in WAL. These XIDs must be
* treated as running by standby transactions, even though they are not in
- * the standby server's PGXACT array.
+ * the standby server's PGPROC array.
*
* We record all XIDs that we know have been assigned. That includes all the
* XIDs seen in WAL records, plus all unobserved XIDs that we can deduce have
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index ad048bc85fa..a9477ccb4a3 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -417,9 +417,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
if (proc != NULL)
{
- PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
-
- *xid = xact->xid;
+ *xid = proc->xid;
*xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 95989ce79bd..d86566f4554 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -3974,9 +3974,8 @@ GetRunningTransactionLocks(int *nlocks)
proclock->tag.myLock->tag.locktag_type == LOCKTAG_RELATION)
{
PGPROC *proc = proclock->tag.myProc;
- PGXACT *pgxact = &ProcGlobal->allPgXact[proc->pgprocno];
LOCK *lock = proclock->tag.myLock;
- TransactionId xid = pgxact->xid;
+ TransactionId xid = proc->xid;
/*
* Don't record locks for transactions if we know they have
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index de346cd87fc..7fad49544ce 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -102,21 +102,18 @@ Size
ProcGlobalShmemSize(void)
{
Size size = 0;
+ Size TotalProcs =
+ add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
- /* MyProcs, including autovacuum workers and launcher */
- size = add_size(size, mul_size(MaxBackends, sizeof(PGPROC)));
- /* AuxiliaryProcs */
- size = add_size(size, mul_size(NUM_AUXILIARY_PROCS, sizeof(PGPROC)));
- /* Prepared xacts */
- size = add_size(size, mul_size(max_prepared_xacts, sizeof(PGPROC)));
- /* ProcStructLock */
+ size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, sizeof(slock_t));
size = add_size(size, mul_size(MaxBackends, sizeof(PGXACT)));
size = add_size(size, mul_size(NUM_AUXILIARY_PROCS, sizeof(PGXACT)));
size = add_size(size, mul_size(max_prepared_xacts, sizeof(PGXACT)));
+ size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
return size;
}
@@ -216,6 +213,17 @@ InitProcGlobal(void)
MemSet(pgxacts, 0, TotalProcs * sizeof(PGXACT));
ProcGlobal->allPgXact = pgxacts;
+ /*
+ * Allocate arrays mirroring PGPROC fields in a dense manner. See
+ * PROC_HDR.
+ *
+ * XXX: It might make sense to increase padding for these arrays, given
+ * how hotly they are accessed.
+ */
+ ProcGlobal->xids =
+ (TransactionId *) ShmemAlloc(TotalProcs * sizeof(*ProcGlobal->xids));
+ MemSet(ProcGlobal->xids, 0, TotalProcs * sizeof(*ProcGlobal->xids));
+
for (i = 0; i < TotalProcs; i++)
{
/* Common initialization for all PGPROCs, regardless of type. */
@@ -387,7 +395,7 @@ InitProcess(void)
MyProc->lxid = InvalidLocalTransactionId;
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
- MyPgXact->xid = InvalidTransactionId;
+ MyProc->xid = InvalidTransactionId;
MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
@@ -571,7 +579,7 @@ InitAuxiliaryProcess(void)
MyProc->lxid = InvalidLocalTransactionId;
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
- MyPgXact->xid = InvalidTransactionId;
+ MyProc->xid = InvalidTransactionId;
MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0004-snapshot-scalability-Move-PGXACT-vacuumFlags-to-.patch"
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-01 00:20 Andres Freund <[email protected]>
0 siblings, 3 replies; 25+ messages in thread
From: Andres Freund @ 2022-10-01 00:20 UTC (permalink / raw)
To: Dilip Kumar <[email protected]>; +Cc: Robert Haas <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
Hi,
On 2022-09-30 15:36:11 +0530, Dilip Kumar wrote:
> I have done some testing around this area to see the impact on WAL
> size especially when WAL sizes are smaller, with a very simple test
> with insert/update/delete I can see around an 11% increase in WAL size
> [1] then I did some more test with pgbench with smaller scale
> factor(1) there I do not see a significant increase in the WAL size
> although it increases WAL size around 1-2%. [2].
I think it'd be interesting to look at per-record-type stats between two
equivalent workload, to see where practical workloads suffer the most
(possibly with fpw=off, to make things more repeatable).
I think it'd be an OK tradeoff to optimize WAL usage for a few of the worst to
pay off for 56bit relfilenodes. The class of problems foreclosed is large
enough to "waste" "improvement potential" on this.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-01 01:44 Peter Geoghegan <[email protected]>
parent: Andres Freund <[email protected]>
2 siblings, 0 replies; 25+ messages in thread
From: Peter Geoghegan @ 2022-10-01 01:44 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Dilip Kumar <[email protected]>; Robert Haas <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
On Fri, Sep 30, 2022 at 5:20 PM Andres Freund <[email protected]> wrote:
> I think it'd be an OK tradeoff to optimize WAL usage for a few of the worst to
> pay off for 56bit relfilenodes. The class of problems foreclosed is large
> enough to "waste" "improvement potential" on this.
I agree overall.
A closely related but distinct question occurs to me: if we're going
to be "wasting" space on alignment padding in certain cases one way or
another, can we at least recognize those cases and take advantage at
the level of individual WAL record formats? In other words: So far
we've been discussing the importance of not going over a critical
threshold for certain WAL records. But it might also be valuable to
consider recognizing that that's inevitable, and that we might as well
make the most of it by including one or two other things.
This seems most likely to matter when managing the problem of negative
compression with per-WAL-record compression schemes for things like
arrays of page offset numbers [1]. If (say) a given compression scheme
"wastes" space for arrays of only 1-3 items, but we already know that
the relevant space will all be lost to alignment needed by code one
level down in any case, does it really count as waste? We're likely
always going to have some kind of negative compression, but you do get
to influence where and when the negative compression happens.
Not sure how relevant this will turn out to be, but seems worth
considering. More generally, thinking about how things work across
multiple layers of abstraction seems like it could be valuable in
other ways.
[1] https://postgr.es/m/CAH2-WzmLCn2Hx9tQLdmdb+9CkHKLyWD2bsz=PmRebc4dAxjy6g@mail.gmail.com
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-01 06:14 Dilip Kumar <[email protected]>
parent: Andres Freund <[email protected]>
2 siblings, 0 replies; 25+ messages in thread
From: Dilip Kumar @ 2022-10-01 06:14 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
On Sat, Oct 1, 2022 at 5:50 AM Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2022-09-30 15:36:11 +0530, Dilip Kumar wrote:
> > I have done some testing around this area to see the impact on WAL
> > size especially when WAL sizes are smaller, with a very simple test
> > with insert/update/delete I can see around an 11% increase in WAL size
> > [1] then I did some more test with pgbench with smaller scale
> > factor(1) there I do not see a significant increase in the WAL size
> > although it increases WAL size around 1-2%. [2].
>
> I think it'd be interesting to look at per-record-type stats between two
> equivalent workload, to see where practical workloads suffer the most
> (possibly with fpw=off, to make things more repeatable).
While testing pgbench, I dumped the wal sizes using waldump. So in
pgbench case, most of the record sizes increased by 4 bytes as they
include single block references and the same is true for the other
test case I sent. Here is the wal dump of what the sizes look like
for a single pgbench transaction[1]. Maybe for seeing these changes
with the different workloads we can run some of the files from the
regression test and compare the individual wal sizes.
Head:
rmgr: Heap len (rec/tot): 54/ 54, tx: 867, lsn:
0/02DD1280, prev 0/02DD1250, desc: LOCK off 44: xid 867: flags 0x01
LOCK_ONLY EXCL_LOCK , blkref #0: rel 1663/5/16424 blk 226
rmgr: Heap len (rec/tot): 171/ 171, tx: 867, lsn:
0/02DD12B8, prev 0/02DD1280, desc: UPDATE off 44 xmax 867 flags 0x11 ;
new off 30 xmax 0, blkref #0: rel 1663/5/16424 blk 1639, blkref #1:
rel 1663/5/16424 blk 226
rmgr: Btree len (rec/tot): 64/ 64, tx: 867, lsn:
0/02DD1368, prev 0/02DD12B8, desc: INSERT_LEAF off 290, blkref #0: rel
1663/5/16432 blk 39
rmgr: Heap len (rec/tot): 78/ 78, tx: 867, lsn:
0/02DD13A8, prev 0/02DD1368, desc: HOT_UPDATE off 15 xmax 867 flags
0x10 ; new off 19 xmax 0, blkref #0: rel 1663/5/16427 blk 0
rmgr: Heap len (rec/tot): 74/ 74, tx: 867, lsn:
0/02DD13F8, prev 0/02DD13A8, desc: HOT_UPDATE off 9 xmax 867 flags
0x10 ; new off 10 xmax 0, blkref #0: rel 1663/5/16425 blk 0
rmgr: Heap len (rec/tot): 79/ 79, tx: 867, lsn:
0/02DD1448, prev 0/02DD13F8, desc: INSERT off 9 flags 0x08, blkref #0:
rel 1663/5/16434 blk 0
rmgr: Transaction len (rec/tot): 46/ 46, tx: 867, lsn:
0/02DD1498, prev 0/02DD1448, desc: COMMIT 2022-10-01 11:24:03.464437
IST
Patch:
rmgr: Heap len (rec/tot): 58/ 58, tx: 818, lsn:
0/0218BEB0, prev 0/0218BE80, desc: LOCK off 34: xid 818: flags 0x01
LOCK_ONLY EXCL_LOCK , blkref #0: rel 1663/5/100004 blk 522
rmgr: Heap len (rec/tot): 175/ 175, tx: 818, lsn:
0/0218BEF0, prev 0/0218BEB0, desc: UPDATE off 34 xmax 818 flags 0x11 ;
new off 8 xmax 0, blkref #0: rel 1663/5/100004 blk 1645, blkref #1:
rel 1663/5/100004 blk 522
rmgr: Btree len (rec/tot): 68/ 68, tx: 818, lsn:
0/0218BFA0, prev 0/0218BEF0, desc: INSERT_LEAF off 36, blkref #0: rel
1663/5/100010 blk 89
rmgr: Heap len (rec/tot): 82/ 82, tx: 818, lsn:
0/0218BFE8, prev 0/0218BFA0, desc: HOT_UPDATE off 66 xmax 818 flags
0x10 ; new off 90 xmax 0, blkref #0: rel 1663/5/100007 blk 0
rmgr: Heap len (rec/tot): 78/ 78, tx: 818, lsn:
0/0218C058, prev 0/0218BFE8, desc: HOT_UPDATE off 80 xmax 818 flags
0x10 ; new off 81 xmax 0, blkref #0: rel 1663/5/100005 blk 0
rmgr: Heap len (rec/tot): 83/ 83, tx: 818, lsn:
0/0218C0A8, prev 0/0218C058, desc: INSERT off 80 flags 0x08, blkref
#0: rel 1663/5/100011 blk 0
rmgr: Transaction len (rec/tot): 46/ 46, tx: 818, lsn:
0/0218C100, prev 0/0218C0A8, desc: COMMIT 2022-10-01 11:11:03.564063
IST
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-03 12:12 Robert Haas <[email protected]>
parent: Andres Freund <[email protected]>
2 siblings, 1 reply; 25+ messages in thread
From: Robert Haas @ 2022-10-03 12:12 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
On Fri, Sep 30, 2022 at 8:20 PM Andres Freund <[email protected]> wrote:
> I think it'd be interesting to look at per-record-type stats between two
> equivalent workload, to see where practical workloads suffer the most
> (possibly with fpw=off, to make things more repeatable).
I would expect, and Dilip's results seem to confirm, the effect to be
pretty uniform: basically, nearly every record gets bigger by 4 bytes.
That's because most records contain at least one block reference, and
if they contain multiple block references, likely all but one will be
marked BKPBLOCK_SAME_REL, so we pay the cost just once.
Because of alignment padding, the practical effect is probably that
about half of the records get bigger by 8 bytes and the other half
don't get bigger at all. But I see no reason to believe that things
are any better or worse than that. Most interesting record types are
going to contain some kind of variable-length payload, so the chances
that a 4 byte size increase pushes you across a MAXALIGN boundary seem
to be no better or worse than fifty-fifty.
> I think it'd be an OK tradeoff to optimize WAL usage for a few of the worst to
> pay off for 56bit relfilenodes. The class of problems foreclosed is large
> enough to "waste" "improvement potential" on this.
I thought about trying to buy back some space elsewhere, and I think
that would be a reasonable approach to getting this committed if we
could find a way to do it. However, I don't see a terribly obvious way
of making it happen. Trying to do it by optimizing specific WAL record
types seems like a real pain in the neck, because there's tons of
different WAL records that all have the same problem. Trying to do it
in a generic way makes more sense, and the fact that we have 2 padding
bytes available in XLogRecord seems like a place to start looking, but
the way forward from there is not clear to me.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-03 17:01 Andres Freund <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 2 replies; 25+ messages in thread
From: Andres Freund @ 2022-10-03 17:01 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
Hi,
On 2022-10-03 08:12:39 -0400, Robert Haas wrote:
> On Fri, Sep 30, 2022 at 8:20 PM Andres Freund <[email protected]> wrote:
> > I think it'd be interesting to look at per-record-type stats between two
> > equivalent workload, to see where practical workloads suffer the most
> > (possibly with fpw=off, to make things more repeatable).
>
> I would expect, and Dilip's results seem to confirm, the effect to be
> pretty uniform: basically, nearly every record gets bigger by 4 bytes.
> That's because most records contain at least one block reference, and
> if they contain multiple block references, likely all but one will be
> marked BKPBLOCK_SAME_REL, so we pay the cost just once.
But it doesn't really matter that much if an already large record gets a bit
bigger. Whereas it does matter if it's a small record. Focussing on optimizing
the record types where the increase is large seems like a potential way
forward to me, even if we can't find something generic.
> I thought about trying to buy back some space elsewhere, and I think
> that would be a reasonable approach to getting this committed if we
> could find a way to do it. However, I don't see a terribly obvious way
> of making it happen.
I think there's plenty potential...
> Trying to do it by optimizing specific WAL record
> types seems like a real pain in the neck, because there's tons of
> different WAL records that all have the same problem.
I am not so sure about that. Improving a bunch of the most frequent small
records might buy you back enough on just about every workload to be OK.
I put the top record sizes for an installcheck run with full_page_writes off
at the bottom. Certainly our regression tests aren't generally
representative. But I think it still decently highlights how just improving a
few records could buy you back more than enough.
> Trying to do it in a generic way makes more sense, and the fact that we have
> 2 padding bytes available in XLogRecord seems like a place to start looking,
> but the way forward from there is not clear to me.
Random idea: xl_prev is large. Store a full xl_prev in the page header, but
only store a 2 byte offset from the page header xl_prev within each record.
Greetings,
Andres Freund
by total size:
Type N (%) Record size (%) FPI size (%) Combined size (%)
---- - --- ----------- --- -------- --- ------------- ---
Heap/INSERT 1041666 ( 50.48) 106565255 ( 50.54) 0 ( 0.00) 106565255 ( 43.92)
Btree/INSERT_LEAF 352196 ( 17.07) 24067672 ( 11.41) 0 ( 0.00) 24067672 ( 9.92)
Heap/DELETE 250852 ( 12.16) 13546008 ( 6.42) 0 ( 0.00) 13546008 ( 5.58)
Hash/INSERT 108499 ( 5.26) 7811928 ( 3.70) 0 ( 0.00) 7811928 ( 3.22)
Transaction/COMMIT 16053 ( 0.78) 6402657 ( 3.04) 0 ( 0.00) 6402657 ( 2.64)
Gist/PAGE_UPDATE 57225 ( 2.77) 5217100 ( 2.47) 0 ( 0.00) 5217100 ( 2.15)
Gin/UPDATE_META_PAGE 23943 ( 1.16) 4539970 ( 2.15) 0 ( 0.00) 4539970 ( 1.87)
Gin/INSERT 27004 ( 1.31) 3623998 ( 1.72) 0 ( 0.00) 3623998 ( 1.49)
Gist/PAGE_SPLIT 448 ( 0.02) 3391244 ( 1.61) 0 ( 0.00) 3391244 ( 1.40)
SPGist/ADD_LEAF 38968 ( 1.89) 3341696 ( 1.58) 0 ( 0.00) 3341696 ( 1.38)
...
XLOG/FPI 7228 ( 0.35) 378924 ( 0.18) 29788166 ( 93.67) 30167090 ( 12.43)
...
Gin/SPLIT 141 ( 0.01) 13011 ( 0.01) 1187588 ( 3.73) 1200599 ( 0.49)
...
-------- -------- -------- --------
Total 2063609 210848282 [86.89%] 31802766 [13.11%] 242651048 [100%]
(Included XLOG/FPI and Gin/SPLIT to explain why there's FPIs despite running with fpw=off)
sorted by number of records:
Heap/INSERT 1041666 ( 50.48) 106565255 ( 50.54) 0 ( 0.00) 106565255 ( 43.92)
Btree/INSERT_LEAF 352196 ( 17.07) 24067672 ( 11.41) 0 ( 0.00) 24067672 ( 9.92)
Heap/DELETE 250852 ( 12.16) 13546008 ( 6.42) 0 ( 0.00) 13546008 ( 5.58)
Hash/INSERT 108499 ( 5.26) 7811928 ( 3.70) 0 ( 0.00) 7811928 ( 3.22)
Gist/PAGE_UPDATE 57225 ( 2.77) 5217100 ( 2.47) 0 ( 0.00) 5217100 ( 2.15)
SPGist/ADD_LEAF 38968 ( 1.89) 3341696 ( 1.58) 0 ( 0.00) 3341696 ( 1.38)
Gin/INSERT 27004 ( 1.31) 3623998 ( 1.72) 0 ( 0.00) 3623998 ( 1.49)
Gin/UPDATE_META_PAGE 23943 ( 1.16) 4539970 ( 2.15) 0 ( 0.00) 4539970 ( 1.87)
Standby/LOCK 18451 ( 0.89) 775026 ( 0.37) 0 ( 0.00) 775026 ( 0.32)
Transaction/COMMIT 16053 ( 0.78) 6402657 ( 3.04) 0 ( 0.00) 6402657 ( 2.64)
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-03 17:40 Matthias van de Meent <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 25+ messages in thread
From: Matthias van de Meent @ 2022-10-03 17:40 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
On Mon, 3 Oct 2022, 19:01 Andres Freund, <[email protected]> wrote:
>
> Hi,
>
> On 2022-10-03 08:12:39 -0400, Robert Haas wrote:
> > On Fri, Sep 30, 2022 at 8:20 PM Andres Freund <[email protected]> wrote:
> > > I think it'd be interesting to look at per-record-type stats between two
> > > equivalent workload, to see where practical workloads suffer the most
> > > (possibly with fpw=off, to make things more repeatable).
> >
> > I would expect, and Dilip's results seem to confirm, the effect to be
> > pretty uniform: basically, nearly every record gets bigger by 4 bytes.
> > That's because most records contain at least one block reference, and
> > if they contain multiple block references, likely all but one will be
> > marked BKPBLOCK_SAME_REL, so we pay the cost just once.
>
> But it doesn't really matter that much if an already large record gets a bit
> bigger. Whereas it does matter if it's a small record. Focussing on optimizing
> the record types where the increase is large seems like a potential way
> forward to me, even if we can't find something generic.
>
>
> > I thought about trying to buy back some space elsewhere, and I think
> > that would be a reasonable approach to getting this committed if we
> > could find a way to do it. However, I don't see a terribly obvious way
> > of making it happen.
>
> I think there's plenty potential...
>
>
> > Trying to do it by optimizing specific WAL record
> > types seems like a real pain in the neck, because there's tons of
> > different WAL records that all have the same problem.
>
> I am not so sure about that. Improving a bunch of the most frequent small
> records might buy you back enough on just about every workload to be OK.
>
> I put the top record sizes for an installcheck run with full_page_writes off
> at the bottom. Certainly our regression tests aren't generally
> representative. But I think it still decently highlights how just improving a
> few records could buy you back more than enough.
>
>
> > Trying to do it in a generic way makes more sense, and the fact that we have
> > 2 padding bytes available in XLogRecord seems like a place to start looking,
> > but the way forward from there is not clear to me.
>
> Random idea: xl_prev is large. Store a full xl_prev in the page header, but
> only store a 2 byte offset from the page header xl_prev within each record.
With that small xl_prev we may not detect partial page writes in
recycled segments; or other issues in the underlying file system. With
small record sizes, the chance of returning incorrect data would be
significant for small records (it would be approximately the chance of
getting a record boundary on the underlying page boundary * chance of
getting the same MAXALIGN-adjusted size record before the persistence
boundary). That issue is part of the reason why my proposed change
upthread still contains the full xl_prev.
A different idea is removing most block_ids from the record, and
optionally reducing per-block length fields to 1B. Used block ids are
effectively always sequential, and we only allow 33+4 valid values, so
we can use 2 bits to distinguish between 'block belonging to this ID
field have at most 255B of data registered' and 'blocks up to this ID
follow sequentially without own block ID'. That would save 2N-1 total
bytes for N blocks. It is scraping the barrel, but I think it is quite
possible.
Lastly, we could add XLR_BLOCK_ID_DATA_MED for values >255 containing
up to UINT16_MAX lengths. That would save 2 bytes for records that
only just pass the 255B barrier, where 2B is still a fairly
significant part of the record size.
Kind regards,
Matthias van de Meent
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-03 21:25 Andres Freund <[email protected]>
parent: Matthias van de Meent <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Andres Freund @ 2022-10-03 21:25 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: Robert Haas <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
Hi,
On 2022-10-03 19:40:30 +0200, Matthias van de Meent wrote:
> On Mon, 3 Oct 2022, 19:01 Andres Freund, <[email protected]> wrote:
> > Random idea: xl_prev is large. Store a full xl_prev in the page header, but
> > only store a 2 byte offset from the page header xl_prev within each record.
>
> With that small xl_prev we may not detect partial page writes in
> recycled segments; or other issues in the underlying file system. With
> small record sizes, the chance of returning incorrect data would be
> significant for small records (it would be approximately the chance of
> getting a record boundary on the underlying page boundary * chance of
> getting the same MAXALIGN-adjusted size record before the persistence
> boundary). That issue is part of the reason why my proposed change
> upthread still contains the full xl_prev.
What exactly is the theory for this significant increase? I don't think
xl_prev provides a meaningful protection against torn pages in the first
place?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-04 13:05 Matthias van de Meent <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Matthias van de Meent @ 2022-10-04 13:05 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
On Mon, 3 Oct 2022 at 23:26, Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2022-10-03 19:40:30 +0200, Matthias van de Meent wrote:
> > On Mon, 3 Oct 2022, 19:01 Andres Freund, <[email protected]> wrote:
> > > Random idea: xl_prev is large. Store a full xl_prev in the page header, but
> > > only store a 2 byte offset from the page header xl_prev within each record.
> >
> > With that small xl_prev we may not detect partial page writes in
> > recycled segments; or other issues in the underlying file system. With
> > small record sizes, the chance of returning incorrect data would be
> > significant for small records (it would be approximately the chance of
> > getting a record boundary on the underlying page boundary * chance of
> > getting the same MAXALIGN-adjusted size record before the persistence
> > boundary). That issue is part of the reason why my proposed change
> > upthread still contains the full xl_prev.
>
> What exactly is the theory for this significant increase? I don't think
> xl_prev provides a meaningful protection against torn pages in the first
> place?
XLog pages don't have checksums, so they do not provide torn page
protection capabilities on their own.
A singular xlog record is protected against torn page writes through
the checksum that covers the whole record - if only part of the record
was written, we can detect that through the mismatching checksum.
However, if records end at the tear boundary, we must know for certain
that any record that starts after the tear is the record that was
written after the one before the tear. Page-local references/offsets
would not work, because the record decoding doesn't know which xlog
page the record should be located on; it could be both the version of
the page before it was recycled, or the one after.
Currently, we can detect this because the value of xl_prev will point
to a record far in the past (i.e. not the expected value), but with a
page-local version of xl_prev we would be less likely to detect torn
pages (and thus be unable to handle this without risk of corruption)
due to the significant chance of the truncated xl_prev value being the
same in both the old and new record.
Example: Page { [ record A ] | tear boundary | [ record B ] } gets
recycled and receives a new record C at the place of A with the same
length.
With your proposal, record B would still be a valid record when it
follows C; as the page-local serial number/offset reference to the
previous record would still match after the torn write.
With the current situation and a full LSN in xl_prev, the mismatching
value in the xl_prev pointer allows us to detect this torn page write
and halt replay, before redoing an old (incorrect) record.
Kind regards,
Matthias van de Meent
PS. there are ideas floating around (I heard about this one from
Heikki) where we could concatenate WAL records into one combined
record that has only one shared xl_prev+crc; which would save these 12
bytes per record. However, that needs a lot of careful consideration
to make sure that the persistence guarantee of operations doesn't get
lost somewhere in the traffic.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-04 15:34 Andres Freund <[email protected]>
parent: Matthias van de Meent <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Andres Freund @ 2022-10-04 15:34 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: Robert Haas <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Heikki Linnakangas <[email protected]>
Hi,
On 2022-10-04 15:05:47 +0200, Matthias van de Meent wrote:
> On Mon, 3 Oct 2022 at 23:26, Andres Freund <[email protected]> wrote:
> > On 2022-10-03 19:40:30 +0200, Matthias van de Meent wrote:
> > > On Mon, 3 Oct 2022, 19:01 Andres Freund, <[email protected]> wrote:
> > > > Random idea: xl_prev is large. Store a full xl_prev in the page header, but
> > > > only store a 2 byte offset from the page header xl_prev within each record.
> > >
> > > With that small xl_prev we may not detect partial page writes in
> > > recycled segments; or other issues in the underlying file system. With
> > > small record sizes, the chance of returning incorrect data would be
> > > significant for small records (it would be approximately the chance of
> > > getting a record boundary on the underlying page boundary * chance of
> > > getting the same MAXALIGN-adjusted size record before the persistence
> > > boundary). That issue is part of the reason why my proposed change
> > > upthread still contains the full xl_prev.
> >
> > What exactly is the theory for this significant increase? I don't think
> > xl_prev provides a meaningful protection against torn pages in the first
> > place?
>
> XLog pages don't have checksums, so they do not provide torn page
> protection capabilities on their own.
> A singular xlog record is protected against torn page writes through
> the checksum that covers the whole record - if only part of the record
> was written, we can detect that through the mismatching checksum.
> However, if records end at the tear boundary, we must know for certain
> that any record that starts after the tear is the record that was
> written after the one before the tear. Page-local references/offsets
> would not work, because the record decoding doesn't know which xlog
> page the record should be located on; it could be both the version of
> the page before it was recycled, or the one after.
> Currently, we can detect this because the value of xl_prev will point
> to a record far in the past (i.e. not the expected value), but with a
> page-local version of xl_prev we would be less likely to detect torn
> pages (and thus be unable to handle this without risk of corruption)
> due to the significant chance of the truncated xl_prev value being the
> same in both the old and new record.
Think this is addressable, see below.
> Example: Page { [ record A ] | tear boundary | [ record B ] } gets
> recycled and receives a new record C at the place of A with the same
> length.
>
> With your proposal, record B would still be a valid record when it
> follows C; as the page-local serial number/offset reference to the
> previous record would still match after the torn write.
> With the current situation and a full LSN in xl_prev, the mismatching
> value in the xl_prev pointer allows us to detect this torn page write
> and halt replay, before redoing an old (incorrect) record.
In this concrete scenario the 8 byte xl_prev doesn't provide *any* protection?
As you specified it, C has the same length as A, so B's xl_prev will be the
same whether it's a page local offset or the full 8 bytes.
The relevant protection against issues like this isn't xl_prev, it's the
CRC. We could improve the CRC by using the "full width" LSN for xl_prev rather
than the offset.
> PS. there are ideas floating around (I heard about this one from
> Heikki) where we could concatenate WAL records into one combined
> record that has only one shared xl_prev+crc; which would save these 12
> bytes per record. However, that needs a lot of careful consideration
> to make sure that the persistence guarantee of operations doesn't get
> lost somewhere in the traffic.
One version of that is to move the CRCs to the page header, make the pages
smaller (512 bytes / 4K, depending on the hardware), and to pad out partial
pages when flushing them out. Rewriting pages is bad for hardware and prevents
having multiple WAL IOs in flight at the same time.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-04 17:36 Robert Haas <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Robert Haas @ 2022-10-04 17:36 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Heikki Linnakangas <[email protected]>
On Tue, Oct 4, 2022 at 11:34 AM Andres Freund <[email protected]> wrote:
> > Example: Page { [ record A ] | tear boundary | [ record B ] } gets
> > recycled and receives a new record C at the place of A with the same
> > length.
> >
> > With your proposal, record B would still be a valid record when it
> > follows C; as the page-local serial number/offset reference to the
> > previous record would still match after the torn write.
> > With the current situation and a full LSN in xl_prev, the mismatching
> > value in the xl_prev pointer allows us to detect this torn page write
> > and halt replay, before redoing an old (incorrect) record.
>
> In this concrete scenario the 8 byte xl_prev doesn't provide *any* protection?
> As you specified it, C has the same length as A, so B's xl_prev will be the
> same whether it's a page local offset or the full 8 bytes.
>
> The relevant protection against issues like this isn't xl_prev, it's the
> CRC. We could improve the CRC by using the "full width" LSN for xl_prev rather
> than the offset.
I'm really confused. xl_prev *is* a full-width LSN currently, as I
understand it. So in the scenario that Matthias poses, let's say the
segment was previously 000000010000000400000025 and now it's
000000010000000400000049. So if a given chunk of the page is leftover
from when the page was 000000010000000400000025, it will have xl_prev
values like 4/25xxxxxx. If it's been rewritten since the segment was
recycled, it will have xl_prev values like 4/49xxxxxx. So, we can tell
whether record B has been overwritten with a new record since the
segment was recycled. But if we stored only 2 bytes in each xl_prev
field, that would no longer be possible.
So I'm lost. It seems like Matthias has correctly identified a real
hazard, and not some weird corner case but actually something that
will happen regularly. All you need is for the old segment that got
recycled to have a record stating at the same place where the page
tore, and for the previous record to have been the same length as the
one on the new page. Given that there's only <~1024 places on a page
where a record can start, and given that in many workloads the lengths
of WAL records will be fairly uniform, this doesn't seem unlikely at
all.
A way to up the chances of detecting this case would be to store only
2 or 4 bytes of xl_prev on disk, but arrange to include the full
xl_prev value in the xl_crc calculation. Then your chances of a
collision are about 2^-32, or maybe more if you posit that CRC is a
weak and crappy algorithm, but even then it's strictly better than
just hoping that there isn't a tear point at a record boundary where
the same length record precedes the tear in both the old and new WAL
segments. However, on the flip side, even if you assume that CRC is a
fantastic algorithm with beautiful and state-of-the-art bit mixing,
the chances of it failing to notice the problem are still >0, whereas
the current algorithm that compares the full xl_prev value is a sure
thing. Because xl_prev values are never repeated, it's certain that
when a segment is recycled, any values that were legal for the old one
aren't legal in the new one.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-04 18:30 Andres Freund <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Andres Freund @ 2022-10-04 18:30 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Heikki Linnakangas <[email protected]>
Hi,
On 2022-10-04 13:36:33 -0400, Robert Haas wrote:
> On Tue, Oct 4, 2022 at 11:34 AM Andres Freund <[email protected]> wrote:
> > > Example: Page { [ record A ] | tear boundary | [ record B ] } gets
> > > recycled and receives a new record C at the place of A with the same
> > > length.
> > >
> > > With your proposal, record B would still be a valid record when it
> > > follows C; as the page-local serial number/offset reference to the
> > > previous record would still match after the torn write.
> > > With the current situation and a full LSN in xl_prev, the mismatching
> > > value in the xl_prev pointer allows us to detect this torn page write
> > > and halt replay, before redoing an old (incorrect) record.
> >
> > In this concrete scenario the 8 byte xl_prev doesn't provide *any* protection?
> > As you specified it, C has the same length as A, so B's xl_prev will be the
> > same whether it's a page local offset or the full 8 bytes.
> >
> > The relevant protection against issues like this isn't xl_prev, it's the
> > CRC. We could improve the CRC by using the "full width" LSN for xl_prev rather
> > than the offset.
>
> I'm really confused. xl_prev *is* a full-width LSN currently, as I
> understand it. So in the scenario that Matthias poses, let's say the
> segment was previously 000000010000000400000025 and now it's
> 000000010000000400000049. So if a given chunk of the page is leftover
> from when the page was 000000010000000400000025, it will have xl_prev
> values like 4/25xxxxxx. If it's been rewritten since the segment was
> recycled, it will have xl_prev values like 4/49xxxxxx. So, we can tell
> whether record B has been overwritten with a new record since the
> segment was recycled. But if we stored only 2 bytes in each xl_prev
> field, that would no longer be possible.
Oh, I think I misunderstood the scenario. I was thinking of cases where we
write out a bunch of pages, crash, only some of the pages made it to disk, we
then write new ones of the same length, and now find a record after the "real"
end of the WAL to be valid. Not sure how I mentally swallowed the "recycled".
For the recycling scenario to be a problem we'll also need to crash, with
parts of the page ending up with the new contents and parts of the page ending
up with the old "pre recycling" content, correct? Because without a crash
we'll have zeroed out the remainder of the page (well, leaving walreceiver out
of the picture, grr).
However, this can easily happen without any record boundaries on the partially
recycled page, so we rely on the CRCs to protect against this.
Here I originally wrote a more in-depth explanation of the scenario I was
thinking about, where we alread rely on CRCs to protect us. But, ooph, I think
they don't reliably, with today's design. But maybe I'm missing more things
today. Consider the following sequence:
1) we write WAL like this:
[record A][tear boundary][record B, prev A_lsn][tear boundary][record C, prev B_lsn]
2) crash, the sectors with A and C made it to disk, the one with B didn't
3) We replay A, discover B is invalid (possibly zeroes or old contents),
insert a new record B' with the same length. Now it looks like this:
[record A][tear boundary][record B', prev A_lsn][tear boundary][record C, prev B_lsn]
4) crash, the sector with B' makes it to disk
5) we replay A, B', C, because C has an xl_prev that's compatible with B'
location and a valid CRC.
Oops.
I think this can happen both within a single page and across page boundaries.
I hope I am missing something here?
> A way to up the chances of detecting this case would be to store only
> 2 or 4 bytes of xl_prev on disk, but arrange to include the full
> xl_prev value in the xl_crc calculation.
Right, that's what I was suggesting as well.
> Then your chances of a collision are about 2^-32, or maybe more if you posit
> that CRC is a weak and crappy algorithm, but even then it's strictly better
> than just hoping that there isn't a tear point at a record boundary where
> the same length record precedes the tear in both the old and new WAL
> segments. However, on the flip side, even if you assume that CRC is a
> fantastic algorithm with beautiful and state-of-the-art bit mixing, the
> chances of it failing to notice the problem are still >0, whereas the
> current algorithm that compares the full xl_prev value is a sure
> thing. Because xl_prev values are never repeated, it's certain that when a
> segment is recycled, any values that were legal for the old one aren't legal
> in the new one.
Given that we already rely on the CRCs to detect corruption within a single
record spanning tear boundaries, this doesn't cause me a lot of heartburn. But
I suspect we might need to do something about the scenario I outlined above,
which likely would also increase the protection against this issue.
I think there might be reasonable ways to increase the guarantees based on the
2 byte xl_prev approach "alone". We don't have to store the offset from the
page header as a plain offset. What about storing something like:
page_offset ^ (page_lsn >> wal_segsz_shift)
I think something like that'd result in prev_not_really_lsn typically not
simply matching after recycling. Of course it only provides so much
protection, given 16bits...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-04 18:53 Robert Haas <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Robert Haas @ 2022-10-04 18:53 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Heikki Linnakangas <[email protected]>
On Tue, Oct 4, 2022 at 2:30 PM Andres Freund <[email protected]> wrote:
> Consider the following sequence:
>
> 1) we write WAL like this:
>
> [record A][tear boundary][record B, prev A_lsn][tear boundary][record C, prev B_lsn]
>
> 2) crash, the sectors with A and C made it to disk, the one with B didn't
>
> 3) We replay A, discover B is invalid (possibly zeroes or old contents),
> insert a new record B' with the same length. Now it looks like this:
>
> [record A][tear boundary][record B', prev A_lsn][tear boundary][record C, prev B_lsn]
>
> 4) crash, the sector with B' makes it to disk
>
> 5) we replay A, B', C, because C has an xl_prev that's compatible with B'
> location and a valid CRC.
>
> Oops.
>
> I think this can happen both within a single page and across page boundaries.
>
> I hope I am missing something here?
If you are, I don't know what it is off-hand. That seems like a
plausible scenario to me. It does require the OS to write things out
of order, and I don't know how likely that is in practice, but the
answer probably isn't zero.
> I think there might be reasonable ways to increase the guarantees based on the
> 2 byte xl_prev approach "alone". We don't have to store the offset from the
> page header as a plain offset. What about storing something like:
> page_offset ^ (page_lsn >> wal_segsz_shift)
>
> I think something like that'd result in prev_not_really_lsn typically not
> simply matching after recycling. Of course it only provides so much
> protection, given 16bits...
Maybe. That does seem somewhat better, but I feel like it's hard to
reason about whether it's safe in absolute terms or just resistant to
the precise scenario Matthias postulated while remaining vulnerable to
slightly modified versions.
How about this: remove xl_prev. widen xl_crc to 64 bits. include the
CRC of the previous WAL record in the xl_crc calculation. That doesn't
cut quite as many bytes out of the record size as your proposal, but
it feels like it should strongly resist pretty much every attack of
this general type, with only the minor disadvantage that the more
expensive CRC calculation will destroy all hope of getting anything
committed.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-04 23:49 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 2 replies; 25+ messages in thread
From: Andres Freund @ 2022-10-04 23:49 UTC (permalink / raw)
To: Robert Haas <[email protected]>; Peter Geoghegan <[email protected]>; David Rowley <[email protected]>; +Cc: Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
Hi,
On 2022-10-03 10:01:25 -0700, Andres Freund wrote:
> On 2022-10-03 08:12:39 -0400, Robert Haas wrote:
> > On Fri, Sep 30, 2022 at 8:20 PM Andres Freund <[email protected]> wrote:
> > I thought about trying to buy back some space elsewhere, and I think
> > that would be a reasonable approach to getting this committed if we
> > could find a way to do it. However, I don't see a terribly obvious way
> > of making it happen.
>
> I think there's plenty potential...
I light dusted off my old varint implementation from [1] and converted the
RelFileLocator and BlockNumber from fixed width integers to varint ones. This
isn't meant as a serious patch, but an experiment to see if this is a path
worth pursuing.
A run of installcheck in a cluster with autovacuum=off, full_page_writes=off
(for increased reproducability) shows a decent saving:
master: 241106544 - 230 MB
varint: 227858640 - 217 MB
The average record size goes from 102.7 to 95.7 bytes excluding the remaining
FPIs, 118.1 to 111.0 including FPIs.
There's plenty other spots that could be converted (e.g. the record length
which rarely needs four bytes), this is just meant as a demonstration.
I used pg_waldump --stats for that range of WAL to measure the CPU overhead. A
profile does show pg_varint_decode_uint64(), but partially that seems to be
offset by the reduced amount of bytes to CRC. Maybe a ~2% overhead remains.
That would be tolerable, I think, because waldump --stats pretty much doesn't
do anything with the WAL.
But I suspect there's plenty of optimization potential in the varint
code. Right now it e.g. stores data as big endian, and the bswap instructions
do show up. And a lot of my bit-maskery could be optimized too.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-10 09:16 Dilip Kumar <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 25+ messages in thread
From: Dilip Kumar @ 2022-10-10 09:16 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Peter Geoghegan <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
On Wed, Oct 5, 2022 at 5:19 AM Andres Freund <[email protected]> wrote:
>
> I light dusted off my old varint implementation from [1] and converted the
> RelFileLocator and BlockNumber from fixed width integers to varint ones. This
> isn't meant as a serious patch, but an experiment to see if this is a path
> worth pursuing.
>
> A run of installcheck in a cluster with autovacuum=off, full_page_writes=off
> (for increased reproducability) shows a decent saving:
>
> master: 241106544 - 230 MB
> varint: 227858640 - 217 MB
>
> The average record size goes from 102.7 to 95.7 bytes excluding the remaining
> FPIs, 118.1 to 111.0 including FPIs.
>
I have also executed my original test after applying these patches on
top of the 56 bit relfilenode patch. So earlier we saw the WAL size
increased by 11% (66199.09375 kB to 73906.984375 kB) and after this
patch now the WAL generated is 58179.2265625. That means in this
particular example this patch is reducing the WAL size by 12% even
with the 56 bit relfilenode patch.
[1] https://www.postgresql.org/message-id/CAFiTN-uut%2B04AdwvBY_oK_jLvMkwXUpDJj5mXg--nek%2BucApPQ%40mail...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-10 12:10 Robert Haas <[email protected]>
parent: Dilip Kumar <[email protected]>
0 siblings, 2 replies; 25+ messages in thread
From: Robert Haas @ 2022-10-10 12:10 UTC (permalink / raw)
To: Dilip Kumar <[email protected]>; +Cc: Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
On Mon, Oct 10, 2022 at 5:16 AM Dilip Kumar <[email protected]> wrote:
> I have also executed my original test after applying these patches on
> top of the 56 bit relfilenode patch. So earlier we saw the WAL size
> increased by 11% (66199.09375 kB to 73906.984375 kB) and after this
> patch now the WAL generated is 58179.2265625. That means in this
> particular example this patch is reducing the WAL size by 12% even
> with the 56 bit relfilenode patch.
That's a very promising result, but the question in my mind is how
much work would be required to bring this patch to a committable
state?
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-10 21:22 Andres Freund <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 0 replies; 25+ messages in thread
From: Andres Freund @ 2022-10-10 21:22 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Dilip Kumar <[email protected]>; Peter Geoghegan <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
Hi,
On 2022-10-10 08:10:22 -0400, Robert Haas wrote:
> On Mon, Oct 10, 2022 at 5:16 AM Dilip Kumar <[email protected]> wrote:
> > I have also executed my original test after applying these patches on
> > top of the 56 bit relfilenode patch. So earlier we saw the WAL size
> > increased by 11% (66199.09375 kB to 73906.984375 kB) and after this
> > patch now the WAL generated is 58179.2265625. That means in this
> > particular example this patch is reducing the WAL size by 12% even
> > with the 56 bit relfilenode patch.
>
> That's a very promising result, but the question in my mind is how
> much work would be required to bring this patch to a committable
> state?
The biggest part clearly is to review the variable width integer patch. It's
not a large amount of code, but probably more complicated than average.
One complication there is that currently the patch assumes:
* Note that this function, for efficiency, reads 8 bytes, even if the
* variable integer is less than 8 bytes long. The buffer has to be
* allocated sufficiently large to account for that fact. The maximum
* amount of memory read is 9 bytes.
We could make a less efficient version without that assumption, but I think it
might be easier to just guarantee it in the xlog*.c case.
Using it in xloginsert.c is pretty darn simple, code-wise. xlogreader is bit
harder, although not for intrinsic reasons - the logic underlying
COPY_HEADER_FIELD seems unneccessary complicated to me. The minimal solution
would likely be to just wrap the varint reads in another weird macro.
Leaving the code issues themselves aside, one important thing would be to
evaluate what the performance impacts of the varint encoding/decoding are as
part of "full" server. I suspect it'll vanish in the noise, but we'd need to
validate that.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-11 08:33 Dilip Kumar <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 0 replies; 25+ messages in thread
From: Dilip Kumar @ 2022-10-11 08:33 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>
On Mon, Oct 10, 2022 at 5:40 PM Robert Haas <[email protected]> wrote:
>
> On Mon, Oct 10, 2022 at 5:16 AM Dilip Kumar <[email protected]> wrote:
> > I have also executed my original test after applying these patches on
> > top of the 56 bit relfilenode patch. So earlier we saw the WAL size
> > increased by 11% (66199.09375 kB to 73906.984375 kB) and after this
> > patch now the WAL generated is 58179.2265625. That means in this
> > particular example this patch is reducing the WAL size by 12% even
> > with the 56 bit relfilenode patch.
>
> That's a very promising result, but the question in my mind is how
> much work would be required to bring this patch to a committable
> state?
Right, the results are promising. I have done some more testing with
make installcheck WAL size (fpw=off) and I have seen a similar gain
with this patch.
1. Head: 272 MB
2. 56 bit RelfileLocator: 285 MB
3. 56 bit RelfileLocator + this patch: 261 MB
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-12 20:05 Matthias van de Meent <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 25+ messages in thread
From: Matthias van de Meent @ 2022-10-12 20:05 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Peter Geoghegan <[email protected]>; David Rowley <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Heikki Linnakangas <[email protected]>
On Wed, 5 Oct 2022 at 01:50, Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2022-10-03 10:01:25 -0700, Andres Freund wrote:
> > On 2022-10-03 08:12:39 -0400, Robert Haas wrote:
> > > On Fri, Sep 30, 2022 at 8:20 PM Andres Freund <[email protected]> wrote:
> > > I thought about trying to buy back some space elsewhere, and I think
> > > that would be a reasonable approach to getting this committed if we
> > > could find a way to do it. However, I don't see a terribly obvious way
> > > of making it happen.
> >
> > I think there's plenty potential...
>
> I light dusted off my old varint implementation from [1] and converted the
> RelFileLocator and BlockNumber from fixed width integers to varint ones. This
> isn't meant as a serious patch, but an experiment to see if this is a path
> worth pursuing.
>
> A run of installcheck in a cluster with autovacuum=off, full_page_writes=off
> (for increased reproducability) shows a decent saving:
>
> master: 241106544 - 230 MB
> varint: 227858640 - 217 MB
I think a signficant part of this improvement comes from the premise
of starting with a fresh database. tablespace OID will indeed most
likely be low, but database OID may very well be linearly distributed
if concurrent workloads in the cluster include updating (potentially
unlogged) TOASTed columns and the databases are not created in one
"big bang" but over the lifetime of the cluster. In that case DBOID
will consume 5B for a significant fraction of databases (anything with
OID >=2^28).
My point being: I don't think that we should have different WAL
performance in databases which is dependent on which OID was assigned
to that database.
In addition; this varlen encoding of relfilenode would mean that
performance would drop over time, as a relations' relfile locator is
updated to something with a wider number (through VACUUM FULL or other
relfilelocator cycling; e.g. re-importing a database). For maximum
performance, you'd have to tune your database to have the lowest
possible database, namespace and relfilelocator numbers; which (in
older clusters) implies hacking into the catalogs - which seems like
an antipattern.
I would have much less issue with this if we had separate counters per
database (and approximately incremental dbOid:s), but that's not the
case right now.
> The average record size goes from 102.7 to 95.7 bytes excluding the remaining
> FPIs, 118.1 to 111.0 including FPIs.
That's quite promising.
> There's plenty other spots that could be converted (e.g. the record length
> which rarely needs four bytes), this is just meant as a demonstration.
Agreed.
> I used pg_waldump --stats for that range of WAL to measure the CPU overhead. A
> profile does show pg_varint_decode_uint64(), but partially that seems to be
> offset by the reduced amount of bytes to CRC. Maybe a ~2% overhead remains.
>
> That would be tolerable, I think, because waldump --stats pretty much doesn't
> do anything with the WAL.
>
> But I suspect there's plenty of optimization potential in the varint
> code. Right now it e.g. stores data as big endian, and the bswap instructions
> do show up. And a lot of my bit-maskery could be optimized too.
One thing that comes to mind is that we will never see dbOid < 2^8
(and rarely < 2^14, nor spcOid less than 2^8 for that matter), so
we'll probably waste at least one or two bits in the encoding of those
values. That's not the end of the world, but it'd probably be better
if we could improve on that - up to 6% of the field's disk usage would
be wasted on an always-on bit.
----
Attached is a prototype patchset that reduces the WAL record size in
many common cases. This is a prototype, as it fails tests due to a
locking issue in prepared_xacts that I have not been able to find the
source of yet. It also could use some more polishing, but the base
case seems quite good. I haven't yet run the numbers though...
0001 - Extract xl_rminfo from xl_info
See [0] for more info as to why that's useful, the patch was pulled
from there. It is mainly used to reduce the size of 0002; and mostly
consists of find-and-replace of rmgrs extracting their bits from
xl_info.
0002 - Rework XLogRecord
This makes many fields in the xlog header optional, reducing the size
of many xlog records by several bytes. This implements the design I
shared in my earlier message [1].
0003 - Rework XLogRecordBlockHeader.
This patch could be applied on current head, and saves some bytes in
per-block data. It potentially saves some bytes per registered
block/buffer in the WAL record (max 2 bytes for the first block, after
that up to 3). See the patch's commit message in the patch for
detailed information.
Kind regards,
Matthias van de Meent
[0] https://postgr.es/m/CAEze2WgZti_Bgs-Aw3egsR5PJQpHcYZwZFCJND5MS-O_DX0-Hg%40mail.gmail.com
[1] https://postgr.es/m/CAEze2WjOFzRzPMPYhH4odSa9OCF2XeZszE3jGJhJzrpdFmyLOw@mail.gmail.com
Attachments:
[application/x-patch] 0003-Compactify-XLogRecordBlockHeader-format-where-possib.patch (9.5K, ../../CAEze2Wjd3jY_UhhOGdGGnC6NO=+NmtNOmd=JaYv-v-nwBAiXXA@mail.gmail.com/2-0003-Compactify-XLogRecordBlockHeader-format-where-possib.patch)
download | inline diff:
From b42ffc126fe714d3a25bb198fb45ef483c9e5795 Mon Sep 17 00:00:00 2001
From: Matthias van de Meent <[email protected]>
Date: Wed, 12 Oct 2022 21:16:53 +0200
Subject: [PATCH 3/3] Compactify XLogRecordBlockHeader format, where possible.
- Add 'sequential block IDs' compression
Usually, blocks are registered with sequential IDs starting with 0.
Utilize that pattern to deduplicate those fields in that common case.
- Add 'mini block data' field size
If blocks don't need 2 bytes to store length; we don't those 2 bytes.
- Omit data_length if !BKPBLOCK_HAS_DATA
FPIs have their own data length bookkeeping, so don't bother with
those 2 bytes if the field is 0.
---
src/backend/access/transam/xloginsert.c | 115 +++++++++++++++++++++++-
src/backend/access/transam/xlogreader.c | 39 +++++++-
src/include/access/xlogrecord.h | 25 ++++--
3 files changed, 167 insertions(+), 12 deletions(-)
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 2ac1d0a870..c9d54c0a1e 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -559,6 +559,7 @@ XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
XLogRecData *rdt_datas_last;
XLogRecord *rechdr = (XLogRecord *) hdr_scratch;
char *scratch = hdr_scratch;
+ uint8 *first_blockid = NULL;
Assert((info & ~(XLR_INFO_USERFLAGS)) == 0);
@@ -626,7 +627,10 @@ XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
bool include_image;
if (!regbuf->in_use)
+ {
+ first_blockid = NULL;
continue;
+ }
only_hdr = false;
/* Determine if this block needs to be backed up */
@@ -843,8 +847,115 @@ XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
prev_regbuf = regbuf;
/* Ok, copy the header to the scratch buffer */
- memcpy(scratch, &bkpb, SizeOfXLogRecordBlockHeader);
- scratch += SizeOfXLogRecordBlockHeader;
+ /* First block needs to initialize optional batching */
+ if (block_id == 0)
+ {
+ first_blockid = (uint8 *) scratch;
+
+ if (bkpb.data_length <= UINT8_MAX)
+ {
+ /*
+ * Optimistic guess: if the first registered block has no data,
+ * it is likely that subsequent blocks have little data too.
+ * If we're wrong, a subsequent block will have 1 more byte
+ * than when we would have guessed correctly. If we're right,
+ * it saves a byte for each correctly guessed block.
+ */
+ bkpb.id |= XLR_BLOCK_DATA_SMALL;
+ memcpy(scratch, &bkpb, offsetof(XLogRecordBlockHeader, data_length));
+ scratch += offsetof(XLogRecordBlockHeader, data_length);
+
+ if (bkpb.data_length == 0)
+ Assert((bkpb.fork_flags & BKPBLOCK_HAS_DATA) == 0);
+ else
+ {
+ uint8 data_length = bkpb.data_length;
+ memcpy(scratch++, &data_length, sizeof(uint8));
+ }
+ }
+ else
+ {
+ memcpy(scratch, &bkpb, SizeOfXLogRecordBlockHeader);
+ scratch += SizeOfXLogRecordBlockHeader;
+ }
+ }
+ /* in a sequence of block ids; with sequence data at first_blockid */
+ else if (first_blockid != NULL)
+ {
+ if (bkpb.data_length == 0)
+ {
+ /*
+ * Empty blocks don't care about data_length field size, as
+ * it is not emitted.
+ */
+ *first_blockid += 1;
+ *first_blockid |= XLR_BLOCK_FIRST_NP1_SEQ;
+
+ memcpy(scratch++, &bkpb.fork_flags, sizeof(bkpb.fork_flags));
+ }
+ else if (*first_blockid & XLR_BLOCK_DATA_SMALL)
+ {
+ if (bkpb.data_length <= UINT8_MAX)
+ {
+ uint8 fork_flags = bkpb.fork_flags;
+ uint8 data_length = bkpb.data_length;
+
+ *first_blockid += 1;
+ *first_blockid |= XLR_BLOCK_FIRST_NP1_SEQ;
+
+ memcpy(scratch++, &fork_flags, sizeof(uint8));
+ memcpy(scratch++, &data_length, sizeof(uint8));
+ }
+ else
+ {
+ first_blockid = NULL;
+ memcpy(scratch, &bkpb, SizeOfXLogRecordBlockHeader);
+ scratch += SizeOfXLogRecordBlockHeader;
+ }
+ }
+ else
+ {
+ /*
+ * It would save 1 byte in the length field to break the chain,
+ * but that would cost 1 byte as well, and completely disable
+ * any chances of saving 1 byte in the block header of a
+ * later block.
+ * So as long as the chain continues, we won't have a large
+ * record header.
+ * We do still ignore the block_id field in the block header.
+ */
+ *first_blockid += 1;
+ *first_blockid |= XLR_BLOCK_FIRST_NP1_SEQ;
+
+ memcpy(scratch,
+ ((char *) &bkpb) + offsetof(XLogRecordBlockHeader, fork_flags),
+ SizeOfXLogRecordBlockHeader - offsetof(XLogRecordBlockHeader, fork_flags));
+ scratch += SizeOfXLogRecordBlockHeader - offsetof(XLogRecordBlockHeader, fork_flags);
+ }
+ }
+ /* not in a sequence of block ids */
+ else
+ {
+ if (bkpb.data_length <= UINT8_MAX)
+ {
+ memcpy(scratch, &bkpb, offsetof(XLogRecordBlockHeader, data_length));
+ scratch += offsetof(XLogRecordBlockHeader, data_length);
+
+ if (bkpb.data_length == 0)
+ Assert((bkpb.fork_flags & BKPBLOCK_HAS_DATA) == 0);
+ else
+ {
+ uint8 data_length = bkpb.data_length;
+ memcpy(scratch++, &data_length, sizeof(uint8));
+ }
+ }
+ else
+ {
+ memcpy(scratch, &bkpb, SizeOfXLogRecordBlockHeader);
+ scratch += SizeOfXLogRecordBlockHeader;
+ }
+ }
+
if (include_image)
{
memcpy(scratch, &bimg, SizeOfXLogRecordBlockImageHeader);
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 56bf86cb27..cb930c9e82 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1792,6 +1792,9 @@ DecodeXLogRecord(XLogReaderState *state,
uint32 datatotal;
RelFileLocator *rlocator = NULL;
uint8 block_id;
+ bool shorthdr = false;
+ bool in_bid_seq = false;
+ uint8 bid_seq_end = 0;
decoded->header = (XLRHeaderData) {0};
decoded->lsn = lsn;
@@ -1851,7 +1854,10 @@ DecodeXLogRecord(XLogReaderState *state,
datatotal = 0;
while (remaining > datatotal)
{
- COPY_HEADER_FIELD(&block_id, sizeof(uint8));
+ if (in_bid_seq)
+ block_id++;
+ else
+ COPY_HEADER_FIELD(&block_id, sizeof(uint8));
if (block_id == XLR_BLOCK_ID_DATA_SHORT)
{
@@ -1884,12 +1890,24 @@ DecodeXLogRecord(XLogReaderState *state,
{
COPY_HEADER_FIELD(&decoded->toplevel_xid, sizeof(TransactionId));
}
- else if (block_id <= XLR_MAX_BLOCK_ID)
+ else if ((block_id & XLR_BLOCK_ID_MASK) <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
DecodedBkpBlock *blk;
uint8 fork_flags;
+ if (!in_bid_seq)
+ {
+ shorthdr = (block_id & XLR_BLOCK_DATA_SMALL) != 0;
+
+ if (block_id & XLR_BLOCK_FIRST_NP1_SEQ)
+ {
+ bid_seq_end = (block_id & XLR_BLOCK_ID_MASK) + 1;
+ block_id = decoded->max_block_id + 1;
+ in_bid_seq = true;
+ }
+ }
+
/* mark any intervening block IDs as not in use */
for (int i = decoded->max_block_id + 1; i < block_id; ++i)
decoded->blocks[i].in_use = false;
@@ -1915,8 +1933,19 @@ DecodeXLogRecord(XLogReaderState *state,
blk->has_data = ((fork_flags & BKPBLOCK_HAS_DATA) != 0);
blk->prefetch_buffer = InvalidBuffer;
+ if (!blk->has_data)
+ {
+ /* no data == no data_len, in all situations */
+ }
+ else if (shorthdr)
+ {
+ uint8 len = 0;
+ COPY_HEADER_FIELD(&len, sizeof(uint8));
+ blk->data_len = len;
+ }
+ else
+ COPY_HEADER_FIELD(&blk->data_len, sizeof(uint16));
- COPY_HEADER_FIELD(&blk->data_len, sizeof(uint16));
/* cross-check that the HAS_DATA flag is set iff data_length > 0 */
if (blk->has_data && blk->data_len == 0)
{
@@ -2033,6 +2062,10 @@ DecodeXLogRecord(XLogReaderState *state,
blk->rlocator = *rlocator;
}
COPY_HEADER_FIELD(&blk->blkno, sizeof(BlockNumber));
+
+ /* finish the sequence we're in */
+ if (block_id == bid_seq_end)
+ in_bid_seq = false;
}
else
{
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 94125f9878..862c922032 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -220,13 +220,18 @@ XLogRecordGetRMInfo(XLogRecord *record)
*
* Note that we don't attempt to align the XLogRecordBlockHeader struct!
* So, the struct must be copied to aligned local storage before use.
+ *
+ * Note that if .id & 0x80 is set; the block header is small instead.
+ * If .id & 0x40 is set, there will be id + 1 following block headers
+ * of this type, having incremental ids, but written to disk without
+ * the id field.
*/
typedef struct XLogRecordBlockHeader
{
uint8 id; /* block reference ID */
uint8 fork_flags; /* fork within the relation, and flags */
uint16 data_length; /* number of payload bytes (not including page
- * image) */
+ * image). Emitted iff BKPBLOCK_HAS_DATA */
/* If BKPBLOCK_HAS_IMAGE, an XLogRecordBlockImageHeader struct follows */
/* If BKPBLOCK_SAME_REL is not set, a RelFileLocator follows */
@@ -359,11 +364,17 @@ typedef struct XLogRecordDataHeaderLong
* need a handful of block references, but there are a few exceptions that
* need more.
*/
-#define XLR_MAX_BLOCK_ID 32
-
-#define XLR_BLOCK_ID_DATA_SHORT 255
-#define XLR_BLOCK_ID_DATA_LONG 254
-#define XLR_BLOCK_ID_ORIGIN 253
-#define XLR_BLOCK_ID_TOPLEVEL_XID 252
+#define XLR_MAX_BLOCK_ID 0x20
+
+#define XLR_BLOCK_ID_DATA_SHORT 0x3F
+#define XLR_BLOCK_ID_DATA_LONG 0x3E
+#define XLR_BLOCK_ID_ORIGIN 0x3D
+#define XLR_BLOCK_ID_TOPLEVEL_XID 0x3C
+
+#define XLR_BLOCK_ID_MASK 0x3F
+#define XLR_BLOCK_FIRST_NP1_SEQ 0x40 /* the following blocks are
+ * 0..block_id + 1, and have omitted
+ * their block ID */
+#define XLR_BLOCK_DATA_SMALL 0x80 /* data_length field is uint8 */
#endif /* XLOGRECORD_H */
--
2.30.2
[application/x-patch] 0002-Compactify-xlog-format.patch (67.3K, ../../CAEze2Wjd3jY_UhhOGdGGnC6NO=+NmtNOmd=JaYv-v-nwBAiXXA@mail.gmail.com/3-0002-Compactify-xlog-format.patch)
download | inline diff:
From eb42223e869ddc8b4d5c7a62fe8f1e6e58eedced Mon Sep 17 00:00:00 2001
From: Matthias van de Meent <[email protected]>
Date: Tue, 11 Oct 2022 19:15:58 +0200
Subject: [PATCH 2/3] Compactify xlog format
Reduces xlog header size to 16B in many cases, and <=21B in the
common case, saving 3-8 bytes /record.
Future improvements will see up to 2 more bytes saved per registered
block.
Note: prepared_xacts tests are disabled for now; because I had issues
that I couldn't trace down; resulting in a lock waiting for terminated
backends. IDK how that happened, but that's for a later day.
---
contrib/pg_walinspect/pg_walinspect.c | 4 +-
src/backend/access/heap/heapam.c | 24 +-
src/backend/access/heap/rewriteheap.c | 3 +-
src/backend/access/transam/multixact.c | 3 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/access/transam/xact.c | 6 +-
src/backend/access/transam/xlog.c | 90 +++-
src/backend/access/transam/xloginsert.c | 141 +++++-
src/backend/access/transam/xlogprefetcher.c | 2 +-
src/backend/access/transam/xlogreader.c | 438 ++++++++++++------
src/backend/access/transam/xlogrecovery.c | 31 +-
src/backend/catalog/storage.c | 8 +-
src/backend/commands/dbcommands.c | 20 +-
src/backend/replication/logical/decode.c | 6 +
src/backend/replication/logical/logical.c | 2 +-
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 2 +-
src/backend/utils/cache/inval.c | 3 +-
src/bin/pg_resetwal/pg_resetwal.c | 23 +-
src/bin/pg_rewind/parsexlog.c | 6 +-
src/bin/pg_waldump/pg_waldump.c | 2 +-
src/include/access/xloginsert.h | 2 +-
src/include/access/xlogprefetcher.h | 4 +-
src/include/access/xlogreader.h | 6 +-
src/include/access/xlogrecord.h | 173 ++++++-
src/test/regress/parallel_schedule | 7 +-
27 files changed, 748 insertions(+), 264 deletions(-)
diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c
index f4e3b40bed..0c6eeab557 100644
--- a/contrib/pg_walinspect/pg_walinspect.c
+++ b/contrib/pg_walinspect/pg_walinspect.c
@@ -38,7 +38,7 @@ PG_FUNCTION_INFO_V1(pg_get_wal_stats_till_end_of_wal);
static bool IsFutureLSN(XLogRecPtr lsn, XLogRecPtr *curr_lsn);
static XLogReaderState *InitXLogReaderState(XLogRecPtr lsn);
-static XLogRecord *ReadNextXLogRecord(XLogReaderState *xlogreader);
+static XLRHeaderData *ReadNextXLogRecord(XLogReaderState *xlogreader);
static void GetWALRecordInfo(XLogReaderState *record, Datum *values,
bool *nulls, uint32 ncols);
static XLogRecPtr ValidateInputLSNs(bool till_end_of_wal,
@@ -138,7 +138,7 @@ InitXLogReaderState(XLogRecPtr lsn)
* encounter errors if the flush pointer falls in the middle of a record. In
* that case we'll return NULL.
*/
-static XLogRecord *
+static XLRHeaderData *
ReadNextXLogRecord(XLogReaderState *xlogreader)
{
XLogRecord *record;
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 94e702cdb2..22b12da928 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2171,7 +2171,8 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
/* filtering by origin on a row level is much more efficient */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- recptr = XLogInsert(RM_HEAP_ID, rminfo);
+ recptr = XLogInsertExtended(RM_HEAP_ID, XLR_HAS_XID,
+ rminfo, InvalidCommandId);
PageSetLSN(page, recptr);
}
@@ -2519,7 +2520,8 @@ heap_multi_insert(Relation relation, TupleTableSlot **slots, int ntuples,
/* filtering by origin on a row level is much more efficient */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- recptr = XLogInsert(RM_HEAP2_ID, rminfo);
+ recptr = XLogInsertExtended(RM_HEAP2_ID, XLR_HAS_XID,
+ rminfo, InvalidCommandId);
PageSetLSN(page, recptr);
}
@@ -3018,7 +3020,8 @@ l1:
/* filtering by origin on a row level is much more efficient */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- recptr = XLogInsert(RM_HEAP_ID, XLOG_HEAP_DELETE);
+ recptr = XLogInsertExtended(RM_HEAP_ID, XLR_HAS_XID,
+ XLOG_HEAP_DELETE, InvalidCommandId);
PageSetLSN(page, recptr);
}
@@ -5813,7 +5816,8 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
XLogRegisterData((char *) &xlrec, SizeOfHeapConfirm);
XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
- recptr = XLogInsert(RM_HEAP_ID, XLOG_HEAP_CONFIRM);
+ recptr = XLogInsertExtended(RM_HEAP_ID, XLR_HAS_XID,
+ XLOG_HEAP_CONFIRM, InvalidCommandId);
PageSetLSN(page, recptr);
}
@@ -5956,7 +5960,8 @@ heap_abort_speculative(Relation relation, ItemPointer tid)
/* No replica identity & replication origin logged */
- recptr = XLogInsert(RM_HEAP_ID, XLOG_HEAP_DELETE);
+ recptr = XLogInsertExtended(RM_HEAP_ID, XLR_HAS_XID,
+ XLOG_HEAP_DELETE, InvalidCommandId);
PageSetLSN(page, recptr);
}
@@ -6067,7 +6072,8 @@ heap_inplace_update(Relation relation, HeapTuple tuple)
/* inplace updates aren't decoded atm, don't log the origin */
- recptr = XLogInsert(RM_HEAP_ID, XLOG_HEAP_INPLACE);
+ recptr = XLogInsertExtended(RM_HEAP_ID, XLR_HAS_XID,
+ XLOG_HEAP_INPLACE, InvalidCommandId);
PageSetLSN(page, recptr);
}
@@ -8439,7 +8445,8 @@ log_heap_update(Relation reln, Buffer oldbuf,
/* filtering by origin on a row level is much more efficient */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- recptr = XLogInsert(RM_HEAP_ID, rminfo);
+ recptr = XLogInsertExtended(RM_HEAP_ID, XLR_HAS_XID,
+ rminfo, InvalidCommandId);
return recptr;
}
@@ -8513,7 +8520,8 @@ log_heap_new_cid(Relation relation, HeapTuple tup)
/* will be looked at irrespective of origin */
- recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_NEW_CID);
+ recptr = XLogInsertExtended(RM_HEAP2_ID, XLR_HAS_XID,
+ XLOG_HEAP2_NEW_CID, InvalidCommandId);
return recptr;
}
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index b01b39b008..0b54dec99d 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -927,7 +927,8 @@ logical_heap_rewrite_flush_mappings(RewriteState state)
XLogRegisterData(waldata_start, len);
/* write xlog record */
- XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_REWRITE);
+ XLogInsertExtended(RM_HEAP2_ID, XLR_HAS_XID,
+ XLOG_HEAP2_REWRITE, InvalidCommandId);
pfree(waldata_start);
}
diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.c
index ca6e238542..2fe15635e9 100644
--- a/src/backend/access/transam/multixact.c
+++ b/src/backend/access/transam/multixact.c
@@ -835,7 +835,8 @@ MultiXactIdCreateFromMembers(int nmembers, MultiXactMember *members)
XLogRegisterData((char *) (&xlrec), SizeOfMultiXactCreate);
XLogRegisterData((char *) members, nmembers * sizeof(MultiXactMember));
- (void) XLogInsert(RM_MULTIXACT_ID, XLOG_MULTIXACT_CREATE_ID);
+ (void) XLogInsertExtended(RM_MULTIXACT_ID, XLR_HAS_XID,
+ XLOG_MULTIXACT_CREATE_ID, InvalidCommandId);
/* Now enter the information into the OFFSETs and MEMBERs logs */
RecordNewMultiXact(multi, offset, nmembers, members);
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 98c1ef7ec5..0bae7c71ea 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -1396,7 +1396,7 @@ ReadTwoPhaseFile(TransactionId xid, bool missing_ok)
static void
XlogReadTwoPhaseData(XLogRecPtr lsn, char **buf, int *len)
{
- XLogRecord *record;
+ XLRHeaderData *record;
XLogReaderState *xlogreader;
char *errormsg;
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 8ce44abfcf..fd9ce4d41f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -5769,7 +5769,8 @@ XactLogCommitRecord(TimestampTz commit_time,
/* we allow filtering by xacts */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- return XLogInsertExtended(RM_XACT_ID, info, rminfo);
+ return XLogInsertExtended(RM_XACT_ID, info | XLR_HAS_XID,
+ rminfo, InvalidCommandId);
}
/*
@@ -5917,7 +5918,8 @@ XactLogAbortRecord(TimestampTz abort_time,
if (TransactionIdIsValid(twophase_xid))
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- return XLogInsertExtended(RM_XACT_ID, info, rminfo);
+ return XLogInsertExtended(RM_XACT_ID, info | XLR_HAS_XID,
+ rminfo, InvalidCommandId);
}
/*
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 3058041683..02ec968697 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -742,16 +742,16 @@ XLogInsertRecord(XLogRecData *rdata,
pg_crc32c rdata_crc;
bool inserted;
XLogRecord *rechdr = (XLogRecord *) rdata->data;
- uint8 rminfo = rechdr->xl_rminfo;
- bool isLogSwitch = (rechdr->xl_rmid == RM_XLOG_ID &&
- rminfo == XLOG_SWITCH);
+ uint8 rminfo;
+ uint32 reclength;
+ bool isLogSwitch;
XLogRecPtr StartPos;
XLogRecPtr EndPos;
bool prevDoPageWrites = doPageWrites;
TimeLineID insertTLI;
/* we assume that all of the record header is in the first chunk */
- Assert(rdata->len >= SizeOfXLogRecord);
+ Assert(rdata->len >= MinXLogHeaderSize);
/* cross-check on whether we should be here or not */
if (!XLogInsertAllowed())
@@ -763,6 +763,40 @@ XLogInsertRecord(XLogRecData *rdata,
*/
insertTLI = XLogCtl->InsertTimeLineID;
+ if (rechdr->xl_info & XLR_HAS_RMINFO)
+ {
+ Size offset = MinXLogHeaderSize;
+
+ switch (rechdr->xl_info & XLR2_LEN_MASK)
+ {
+ case XLR2_LEN_ABSENT:
+ break;
+ case XLR2_LEN_1B:
+ offset += 1;
+ break;
+ case XLR2_LEN_2B:
+ offset += 2;
+ break;
+ case XLR2_LEN_4B:
+ offset += 4;
+ break;
+ default:
+ pg_unreachable();
+ }
+ if (rechdr->xl_info & XLR_HAS_XID)
+ offset += sizeof(TransactionId);
+ if (rechdr->xl_info & XLR_HAS_CID)
+ offset += sizeof(CommandId);
+
+ rminfo = *((uint8 *) (rdata->data + offset));
+ }
+ else
+ rminfo = 0;
+
+ reclength = XLogRecordGetLength(rechdr);
+
+ isLogSwitch = (rechdr->xl_rmid == RM_XLOG_ID && rminfo == XLOG_SWITCH);
+
/*----------
*
* We have now done all the preparatory work we can without holding a
@@ -845,7 +879,7 @@ XLogInsertRecord(XLogRecData *rdata,
inserted = ReserveXLogSwitch(&StartPos, &EndPos, &rechdr->xl_prev);
else
{
- ReserveXLogInsertLocation(rechdr->xl_tot_len, &StartPos, &EndPos,
+ ReserveXLogInsertLocation(reclength, &StartPos, &EndPos,
&rechdr->xl_prev);
inserted = true;
}
@@ -865,7 +899,7 @@ XLogInsertRecord(XLogRecData *rdata,
* All the record data, including the header, is now ready to be
* inserted. Copy the record in the space reserved.
*/
- CopyXLogRecordToWAL(rechdr->xl_tot_len, isLogSwitch, rdata,
+ CopyXLogRecordToWAL(reclength, isLogSwitch, rdata,
StartPos, EndPos, insertTLI);
/*
@@ -896,14 +930,18 @@ XLogInsertRecord(XLogRecData *rdata,
END_CRIT_SECTION();
- MarkCurrentTransactionIdLoggedIfAny();
+ if (rechdr->xl_info & XLR_HAS_XID)
+ MarkCurrentTransactionIdLoggedIfAny();
/*
* Mark top transaction id is logged (if needed) so that we should not try
* to log it again with the next WAL record in the current subtransaction.
*/
if (topxid_included)
+ {
+ Assert(rechdr->xl_info & XLR_HAS_XID);
MarkSubxactTopXidLogged();
+ }
/*
* Update shared LogwrtRqst.Write, if we crossed page boundary.
@@ -936,7 +974,8 @@ XLogInsertRecord(XLogRecData *rdata,
*/
if (inserted)
{
- EndPos = StartPos + SizeOfXLogRecord;
+ /* switch record is no more than a min-sized header plus rminfo */
+ EndPos = StartPos + MAXALIGN(MinXLogHeaderSize + sizeof(uint8));
if (StartPos / XLOG_BLCKSZ != EndPos / XLOG_BLCKSZ)
{
uint64 offset = XLogSegmentOffset(EndPos, wal_segment_size);
@@ -977,7 +1016,7 @@ XLogInsertRecord(XLogRecData *rdata,
/* We also need temporary space to decode the record. */
record = (XLogRecord *) recordBuf.data;
decoded = (DecodedXLogRecord *)
- palloc(DecodeXLogRecordRequiredSpace(record->xl_tot_len));
+ palloc(DecodeXLogRecordRequiredSpace(reclength));
if (!debug_reader)
debug_reader = XLogReaderAllocate(wal_segment_size, NULL,
@@ -1022,7 +1061,7 @@ XLogInsertRecord(XLogRecData *rdata,
/* Report WAL traffic to the instrumentation. */
if (inserted)
{
- pgWalUsage.wal_bytes += rechdr->xl_tot_len;
+ pgWalUsage.wal_bytes += reclength;
pgWalUsage.wal_records++;
pgWalUsage.wal_fpi += num_fpi;
}
@@ -1056,7 +1095,7 @@ ReserveXLogInsertLocation(int size, XLogRecPtr *StartPos, XLogRecPtr *EndPos,
size = MAXALIGN(size);
/* All (non xlog-switch) records should contain data. */
- Assert(size > SizeOfXLogRecord);
+ Assert(size > MinXLogHeaderSize);
/*
* The duration the spinlock needs to be held is minimized by minimizing
@@ -1107,7 +1146,7 @@ ReserveXLogSwitch(XLogRecPtr *StartPos, XLogRecPtr *EndPos, XLogRecPtr *PrevPtr)
uint64 startbytepos;
uint64 endbytepos;
uint64 prevbytepos;
- uint32 size = MAXALIGN(SizeOfXLogRecord);
+ uint32 size = MAXALIGN(MinXLogHeaderSize + sizeof(uint8));
XLogRecPtr ptr;
uint32 segleft;
@@ -1250,8 +1289,11 @@ CopyXLogRecordToWAL(int write_len, bool isLogSwitch, XLogRecData *rdata,
*/
if (isLogSwitch && XLogSegmentOffset(CurrPos, wal_segment_size) != 0)
{
- /* An xlog-switch record doesn't contain any data besides the header */
- Assert(write_len == SizeOfXLogRecord);
+ /*
+ * An xlog-switch record doesn't contain any data besides the basic
+ * header and rminfo
+ */
+ Assert(write_len == MAXALIGN(MinXLogHeaderSize + sizeof(uint8)));
/* Assert that we did reserve the right amount of space */
Assert(XLogSegmentOffset(EndPos, wal_segment_size) == 0);
@@ -4672,6 +4714,9 @@ BootStrapXLOG(void)
uint64 sysidentifier;
struct timeval tv;
pg_crc32c crc;
+ const uint32 rec_tot_len = MinXLogHeaderSize + sizeof(uint8) /* xl_tot_len */
+ + SizeOfXLogRecordDataHeaderShort + sizeof(checkPoint);
+ const uint8 rec_len = rec_tot_len;
/* allow ordinary WAL segment creation, like StartupXLOG() would */
SetInstallXLogFileSegmentActive();
@@ -4746,20 +4791,23 @@ BootStrapXLOG(void)
recptr = ((char *) page + SizeOfXLogLongPHD);
record = (XLogRecord *) recptr;
record->xl_prev = 0;
- record->xl_xid = InvalidTransactionId;
- record->xl_tot_len = SizeOfXLogRecord + SizeOfXLogRecordDataHeaderShort + sizeof(checkPoint);
- record->xl_info = XLOG_CHECKPOINT_SHUTDOWN;
+ record->xl_info = XLR2_LEN_1B;
record->xl_rmid = RM_XLOG_ID;
- recptr += SizeOfXLogRecord;
+
+ recptr += MinXLogHeaderSize;
+
+ memcpy(recptr, (char *) &rec_len, sizeof(uint8));
+ recptr += sizeof(uint8);
+
/* fill the XLogRecordDataHeaderShort struct */
*(recptr++) = (char) XLR_BLOCK_ID_DATA_SHORT;
*(recptr++) = sizeof(checkPoint);
- memcpy(recptr, &checkPoint, sizeof(checkPoint));
+ memcpy(recptr, (char *) &checkPoint, sizeof(checkPoint));
recptr += sizeof(checkPoint);
- Assert(recptr - (char *) record == record->xl_tot_len);
+ Assert(recptr - (char *) record == rec_len);
INIT_CRC32C(crc);
- COMP_CRC32C(crc, ((char *) record) + SizeOfXLogRecord, record->xl_tot_len - SizeOfXLogRecord);
+ COMP_CRC32C(crc, ((char *) record) + offsetof(XLogRecord, xl_rmid), rec_len - offsetof(XLogRecord, xl_rmid));
COMP_CRC32C(crc, (char *) record, offsetof(XLogRecord, xl_crc));
FIN_CRC32C(crc);
record->xl_crc = crc;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index cc4a262d8e..2ac1d0a870 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -118,7 +118,7 @@ static char *hdr_scratch = NULL;
#define SizeOfXLogTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
- (SizeOfXLogRecord + \
+ (MaxXLogHeaderSize + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
SizeOfXLogTransactionId)
@@ -135,10 +135,11 @@ static bool begininsert_called = false;
/* Memory context to hold the registered buffer and data references. */
static MemoryContext xloginsert_cxt;
-static XLogRecData *XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
- XLogRecPtr RedoRecPtr, bool doPageWrites,
- XLogRecPtr *fpw_lsn, int *num_fpi,
- bool *topxid_included);
+static XLogRecData *
+XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
+ XLogRecPtr RedoRecPtr, bool doPageWrites,
+ XLogRecPtr *fpw_lsn, int *num_fpi, bool *topxid_included,
+ CommandId cid);
static bool XLogCompressBackupBlock(char *page, uint16 hole_offset,
uint16 hole_length, char *dest, uint16 *dlen);
@@ -447,11 +448,15 @@ XLogSetRecordFlags(uint8 flags)
* (LSN is the XLOG point up to which the XLOG must be flushed to disk
* before the data page can be written out. This implements the basic
* WAL rule "write the log before the data".)
+ *
+ * Note: To include the XID in the record, you have to use
+ * XLogInsertExtended with info = XLR2_HAS_XID. To include CID, do the
+ * same with HAS_CID, and specify the command ID.
*/
XLogRecPtr
XLogInsert(RmgrId rmid, uint8 rminfo)
{
- return XLogInsertExtended(rmid, 0, rminfo);
+ return XLogInsertExtended(rmid, 0, rminfo, InvalidCommandId);
}
@@ -459,11 +464,15 @@ XLogInsert(RmgrId rmid, uint8 rminfo)
* Insert an XLOG record having the specified RMID, info and rminfo bytes,
* with the body of the record being the data and buffer references
* registered earlier with XLogRegister* calls.
+ *
+ * CommandId is an argument to be called by the user, while TransactionId
+ * (if needed) is taken from this backend's state: There is at any time
+ * only one running XID, while there may be more than one active CommandIds.
*
* See also XLogInsert above.
*/
XLogRecPtr
-XLogInsertExtended(RmgrId rmid, uint8 info, uint8 rminfo)
+XLogInsertExtended(RmgrId rmid, uint8 info, uint8 rminfo, CommandId cid)
{
XLogRecPtr EndPos;
@@ -475,8 +484,7 @@ XLogInsertExtended(RmgrId rmid, uint8 info, uint8 rminfo)
* The caller can set XLR_SPECIAL_REL_UPDATE and
* XLR_CHECK_CONSISTENCY; the rest are reserved for use by me.
*/
- if ((info & ~(XLR_SPECIAL_REL_UPDATE |
- XLR_CHECK_CONSISTENCY)) != 0)
+ if ((info & ~XLR_INFO_USERFLAGS) != 0)
elog(PANIC, "invalid xlog info mask %02X", info);
TRACE_POSTGRESQL_WAL_INSERT(rmid, info);
@@ -509,7 +517,8 @@ XLogInsertExtended(RmgrId rmid, uint8 info, uint8 rminfo)
GetFullPageWriteInfo(&RedoRecPtr, &doPageWrites);
rdt = XLogRecordAssemble(rmid, info, rminfo, RedoRecPtr, doPageWrites,
- &fpw_lsn, &num_fpi, &topxid_included);
+ &fpw_lsn, &num_fpi, &topxid_included,
+ cid);
EndPos = XLogInsertRecord(rdt, fpw_lsn, curinsert_flags, num_fpi,
topxid_included);
@@ -538,30 +547,57 @@ XLogInsertExtended(RmgrId rmid, uint8 info, uint8 rminfo)
static XLogRecData *
XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
XLogRecPtr RedoRecPtr, bool doPageWrites,
- XLogRecPtr *fpw_lsn, int *num_fpi, bool *topxid_included)
+ XLogRecPtr *fpw_lsn, int *num_fpi, bool *topxid_included,
+ CommandId cid)
{
XLogRecData *rdt;
uint32 total_len = 0;
int block_id;
+ bool only_hdr = true;
pg_crc32c rdata_crc;
registered_buffer *prev_regbuf = NULL;
XLogRecData *rdt_datas_last;
- XLogRecord *rechdr;
+ XLogRecord *rechdr = (XLogRecord *) hdr_scratch;
char *scratch = hdr_scratch;
+ Assert((info & ~(XLR_INFO_USERFLAGS)) == 0);
+
/*
* Note: this function can be called multiple times for the same record.
* All the modifications we do to the rdata chains below must handle that.
*/
- /* The record begins with the fixed-size header */
- rechdr = (XLogRecord *) scratch;
- scratch += SizeOfXLogRecord;
+ /*
+ * The record begins with the variable-size header data. We pre-allocate
+ * the fixed part of the xlog header section, plus the length field, as
+ * we'll only fill those at the end of the record. The rest can be
+ * pre-filled.
+ */
+ scratch += MinXLogHeaderSize + sizeof(uint32);
hdr_rdt.next = NULL;
rdt_datas_last = &hdr_rdt;
hdr_rdt.data = hdr_scratch;
+ if (IsSubxactTopXidLogPending())
+ info |= XLR_HAS_XID;
+
+ if (info & XLR_HAS_XID)
+ {
+ TransactionId xid = GetCurrentTransactionIdIfAny();
+ memcpy(scratch, (char *) &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
+ if (info & XLR_HAS_CID)
+ {
+ memcpy(scratch, (char *) &cid, sizeof(CommandId));
+ scratch += sizeof(CommandId);
+ }
+
+ if (rminfo != 0)
+ *(scratch++) = rminfo;
+
/*
* Enforce consistency checks for this record if user is looking for it.
* Do this before at the beginning of this routine to give the possibility
@@ -592,6 +628,7 @@ XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
if (!regbuf->in_use)
continue;
+ only_hdr = false;
/* Determine if this block needs to be backed up */
if (regbuf->flags & REGBUF_FORCE_IMAGE)
needs_backup = true;
@@ -832,6 +869,7 @@ XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
if ((curinsert_flags & XLOG_INCLUDE_ORIGIN) &&
replorigin_session_origin != InvalidRepOriginId)
{
+ only_hdr = false;
*(scratch++) = (char) XLR_BLOCK_ID_ORIGIN;
memcpy(scratch, &replorigin_session_origin, sizeof(replorigin_session_origin));
scratch += sizeof(replorigin_session_origin);
@@ -845,6 +883,7 @@ XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
/* Set the flag that the top xid is included in the WAL */
*topxid_included = true;
+ only_hdr = false;
*(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
memcpy(scratch, &xid, sizeof(TransactionId));
scratch += sizeof(TransactionId);
@@ -853,6 +892,8 @@ XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
/* followed by main data, if any */
if (mainrdata_len > 0)
{
+ only_hdr = false;
+
if (mainrdata_len > 255)
{
*(scratch++) = (char) XLR_BLOCK_ID_DATA_LONG;
@@ -873,16 +914,75 @@ XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
hdr_rdt.len = (scratch - hdr_scratch);
total_len += hdr_rdt.len;
+
+ /* ensure that we haven't yet set the length mask */
+ Assert((info & XLR2_LEN_MASK) == 0);
+
+ /*
+ * Here, the full xlog header has been constructed, except for the xlog
+ * record length, and the constant fields in the xlog header.
+ */
+
+ /* fill the xlog header length field and mask */
+ if (only_hdr)
+ {
+ Assert(total_len - sizeof(uint32) == XLRSizeOfHeader(info));
+ info |= XLR2_LEN_ABSENT;
+ memmove(hdr_scratch + MinXLogHeaderSize,
+ hdr_scratch + MinXLogHeaderSize + sizeof(uint32),
+ hdr_rdt.len - MinXLogHeaderSize - sizeof(uint32));
+ hdr_rdt.len = hdr_rdt.len - sizeof(uint32);
+ }
+ else if (total_len - sizeof(uint32) <= UINT8_MAX - sizeof(uint8))
+ {
+ uint8 size = (uint8) (total_len - sizeof(uint32)) + sizeof(uint8);
+ info |= XLR2_LEN_1B;
+ memmove(hdr_scratch + MinXLogHeaderSize + sizeof(uint8),
+ hdr_scratch + MinXLogHeaderSize + sizeof(uint32),
+ hdr_rdt.len - MinXLogHeaderSize - sizeof(uint32));
+ memcpy(hdr_scratch + MinXLogHeaderSize, (char *) &size, sizeof(uint8));
+ total_len = size;
+ hdr_rdt.len = hdr_rdt.len - sizeof(uint32) + sizeof(uint8);
+ }
+ else if (total_len - sizeof(uint32) <= UINT16_MAX - sizeof(uint16))
+ {
+ uint16 size = (uint16) (total_len - sizeof(uint32)) + sizeof(uint16);
+ info |= XLR2_LEN_2B;
+ memmove(hdr_scratch + MinXLogHeaderSize + sizeof(uint16),
+ hdr_scratch + MinXLogHeaderSize + sizeof(uint32),
+ hdr_rdt.len - MinXLogHeaderSize - sizeof(uint32));
+ memcpy(hdr_scratch + MinXLogHeaderSize, (char *) &size, sizeof(uint16));
+ total_len = size;
+ hdr_rdt.len = hdr_rdt.len - sizeof(uint32) + sizeof(uint16);
+ }
+ else
+ {
+ uint32 size = total_len;
+ info |= XLR2_LEN_4B;
+ memcpy(hdr_scratch + MinXLogHeaderSize, (char *) &size, sizeof(uint32));
+ total_len = size;
+ }
+
+ /*
+ * We've filled all variable-length data of the xlog header section,
+ * which allows us to start CRC-ing the data.
+ */
+
+ rechdr->xl_rmid = rmid;
+ rechdr->xl_info = info;
+
/*
* Calculate CRC of the data
*
* Note that the record header isn't added into the CRC initially since we
* don't know the prev-link yet. Thus, the CRC will represent the CRC of
- * the whole record in the order: rdata, then backup blocks, then record
- * header.
+ * the whole record in the order: xl_rmid, xl_info, varlen header fields,
+ * rdata, then backup blocks, then prevptr.
*/
INIT_CRC32C(rdata_crc);
- COMP_CRC32C(rdata_crc, hdr_scratch + SizeOfXLogRecord, hdr_rdt.len - SizeOfXLogRecord);
+ COMP_CRC32C(rdata_crc,
+ hdr_rdt.data + offsetof(XLogRecord, xl_rmid),
+ hdr_rdt.len - offsetof(XLogRecord, xl_rmid));
for (rdt = hdr_rdt.next; rdt != NULL; rdt = rdt->next)
COMP_CRC32C(rdata_crc, rdt->data, rdt->len);
@@ -891,11 +991,6 @@ XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
* once we know where in the WAL the record will be inserted. The CRC does
* not include the record header yet.
*/
- rechdr->xl_xid = GetCurrentTransactionIdIfAny();
- rechdr->xl_tot_len = total_len;
- rechdr->xl_info = info;
- rechdr->xl_rmid = rmid;
- rechdr->xl_rminfo = rminfo;
rechdr->xl_prev = InvalidXLogRecPtr;
rechdr->xl_crc = rdata_crc;
diff --git a/src/backend/access/transam/xlogprefetcher.c b/src/backend/access/transam/xlogprefetcher.c
index e5d4f36182..f885670263 100644
--- a/src/backend/access/transam/xlogprefetcher.c
+++ b/src/backend/access/transam/xlogprefetcher.c
@@ -983,7 +983,7 @@ XLogPrefetcherBeginRead(XLogPrefetcher *prefetcher, XLogRecPtr recPtr)
* A wrapper for XLogReadRecord() that provides the same interface, but also
* tries to initiate I/O for blocks referenced in future WAL records.
*/
-XLogRecord *
+XLRHeaderData *
XLogPrefetcherReadRecord(XLogPrefetcher *prefetcher, char **errmsg)
{
DecodedXLogRecord *record;
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 9200d7f56c..56bf86cb27 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -49,7 +49,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr,
static void XLogReaderInvalReadState(XLogReaderState *state);
static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking);
static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr,
- XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess);
+ XLogRecPtr PrevRecPtr, XLogRecord *record,
+ bool randAccess);
static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record,
XLogRecPtr recptr);
static void ResetDecoder(XLogReaderState *state);
@@ -418,7 +419,7 @@ XLogNextRecord(XLogReaderState *state, char **errormsg)
* The returned pointer (or *errormsg) points to an internal buffer that's
* valid until the next call to XLogReadRecord.
*/
-XLogRecord *
+XLRHeaderData *
XLogReadRecord(XLogReaderState *state, char **errormsg)
{
DecodedXLogRecord *decoded;
@@ -531,6 +532,83 @@ XLogReadRecordAlloc(XLogReaderState *state, size_t xl_tot_len, bool allow_oversi
return NULL;
}
+/*
+ * Fills contdata with pointer to record continuation data of the next page,
+ * and len with the length of the data on that page.
+ */
+#define ReadPageData(dataNeeded, gotLen, expectLen, strict) \
+ do \
+ { \
+ /* Wait for the next page to become available */ \
+ readOff = ReadPageInternal(state, targetPagePtr, \
+ Min((dataNeeded) + SizeOfXLogShortPHD, \
+ XLOG_BLCKSZ)); \
+ \
+ if (readOff == XLREAD_WOULDBLOCK) \
+ return XLREAD_WOULDBLOCK; \
+ else if (readOff < 0) \
+ goto err; \
+ Assert(SizeOfXLogShortPHD <= readOff); \
+ pageHeader = (XLogPageHeader) state->readBuf; \
+ /*
+ * If we were expecting a continuation record and got an
+ * "overwrite contrecord" flag, that means the continuation record
+ * was overwritten with a different record. Restart the read by
+ * assuming the address to read is the location where we found
+ * this flag; but keep track of the LSN of the record we were
+ * reading, for later verification.
+ */ \
+ if (pageHeader->xlp_info & XLP_FIRST_IS_OVERWRITE_CONTRECORD) \
+ { \
+ state->overwrittenRecPtr = RecPtr; \
+ RecPtr = targetPagePtr; \
+ goto restart; \
+ } \
+ \
+ /* Check that the continuation on next page looks valid */ \
+ if (!(pageHeader->xlp_info & XLP_FIRST_IS_CONTRECORD)) \
+ { \
+ report_invalid_record(state, \
+ "there is no contrecord flag at %X/%X", \
+ LSN_FORMAT_ARGS(RecPtr)); \
+ goto err; \
+ } \
+ /*
+ * Cross-check that xlp_rem_len agrees with how much of the record
+ * we expect there to be left.
+ */ \
+ if (pageHeader->xlp_rem_len == 0 || (strict) ? ( \
+ (expectLen) != (pageHeader->xlp_rem_len + (gotLen)) \
+ ) : ( \
+ (expectLen) < (pageHeader->xlp_rem_len + (gotLen)) \
+ )) \
+ { \
+ report_invalid_record(state, \
+ "invalid contrecord length %u (expected %lld) at %X/%X", \
+ pageHeader->xlp_rem_len, \
+ ((long long) (expectLen)) - (gotLen), \
+ LSN_FORMAT_ARGS(RecPtr)); \
+ goto err; \
+ } \
+ /* Append the continuation from this page to the buffer */ \
+ pageHeaderSize = XLogPageHeaderSize(pageHeader); \
+ \
+ if (readOff < Min(pageHeaderSize + (dataNeeded), XLOG_BLCKSZ)) \
+ { \
+ readOff = ReadPageInternal(state, targetPagePtr, \
+ Min(pageHeaderSize + (dataNeeded), XLOG_BLCKSZ)); \
+ if (readOff == XLREAD_WOULDBLOCK) \
+ return XLREAD_WOULDBLOCK; \
+ else if (readOff < 0) \
+ goto err; \
+ } \
+ Assert(pageHeaderSize <= readOff); \
+ \
+ /* point to the right */\
+ contdata = (char *) state->readBuf + pageHeaderSize; \
+ len = readOff - pageHeaderSize; \
+ } while (false)
+
static XLogPageReadResult
XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking)
{
@@ -543,10 +621,16 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking)
uint32 targetRecOff;
uint32 pageHeaderSize;
bool assembled;
- bool gotheader;
int readOff;
DecodedXLogRecord *decoded;
char *errormsg; /* not used */
+ union {
+ char bytes[MaxXLogHeaderSize];
+ XLogRecord hdr;
+ } rechdr = {0}; /* record header buffer for cross-page header reads */
+ uint32 rechdrsize = 0;
+ XLogPageHeader pageHeader;
+ char *contdata;
/*
* randAccess indicates whether to verify the previous-record pointer of
@@ -602,7 +686,8 @@ restart:
* fits on the same page.
*/
readOff = ReadPageInternal(state, targetPagePtr,
- Min(targetRecOff + SizeOfXLogRecord, XLOG_BLCKSZ));
+ Min(targetRecOff + MAXALIGN(MinXLogHeaderSize),
+ XLOG_BLCKSZ));
if (readOff == XLREAD_WOULDBLOCK)
return XLREAD_WOULDBLOCK;
else if (readOff < 0)
@@ -639,47 +724,126 @@ restart:
/* ReadPageInternal has verified the page header */
Assert(pageHeaderSize <= readOff);
- /*
- * Read the record length.
- *
- * NB: Even though we use an XLogRecord pointer here, the whole record
- * header might not fit on this page. xl_tot_len is the first field of the
- * struct, so it must be on this page (the records are MAXALIGNed), but we
- * cannot access any other fields until we've verified that we got the
- * whole header.
- */
- record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ);
- total_len = record->xl_tot_len;
+ contdata = state->readBuf + targetRecOff;
+ len = state->readLen - targetRecOff;
+
+ rechdrsize = Min(len, MinXLogHeaderSize);
+
+ memcpy(&rechdr.bytes[0], contdata, rechdrsize);
+
+ contdata += rechdrsize;
+ len -= rechdrsize;
+
/*
- * If the whole record header is on this page, validate it immediately.
- * Otherwise do just a basic sanity check on xl_tot_len, and validate the
- * rest of the header after reading it from the next page. The xl_tot_len
- * check is necessary here to ensure that we enter the "Need to reassemble
- * record" code path below; otherwise we might fail to apply
- * ValidXLogRecordHeader at all.
+ * If we don't yet have the minimum required data of an XLogRecord,
+ * we have to read from the next page.
*/
- if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord)
+ if (rechdrsize < MinXLogHeaderSize)
{
- if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record,
- randAccess))
- goto err;
- gotheader = true;
+ int gotlen;
+
+ /* Calculate pointer to beginning of next page */
+ targetPagePtr += XLOG_BLCKSZ;
+ ReadPageData(MinXLogHeaderSize - rechdrsize,
+ rechdrsize,
+ MinXLogHeaderSize,
+ false);
+
+ gotlen = Min(len, MinXLogHeaderSize - rechdrsize);
+
+ memcpy(&rechdr.bytes[rechdrsize],
+ contdata,
+ gotlen);
+ rechdrsize += gotlen;
+ contdata += gotlen;
+ len -= gotlen;
}
- else
+
+ Assert(rechdrsize >= MinXLogHeaderSize);
+
+ /*
+ * We now have a minimal XLogHeader in buffers. This is enough
+ * to determine the size of all header data, but we don't yet have
+ * all of that.
+ */
+
+ total_len = XLRSizeOfHeader(rechdr.hdr.xl_info);
+
+ /*
+ * We now known the expected size of the xlog header structures
+ * (which includes the xl_rec_len field).
+ *
+ * If the full headers section requires more data than what we've loaded
+ * by now, we need to read that data.
+ */
+ if (total_len > rechdrsize)
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
+ /* Consume any bytes that are still in the buffer */
+ if (len > 0)
{
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
+ int consumed = Min(len, total_len - rechdrsize);
+
+ memcpy(&rechdr.bytes[rechdrsize],
+ contdata,
+ consumed);
+ contdata += consumed;
+ rechdrsize += consumed;
+ len -= consumed;
+ }
+
+ /* Continue reading from this page if the page wasn't fully read */
+ if (total_len > rechdrsize && state->readLen < XLOG_BLCKSZ)
+ {
+ int consumed;
+ int needed = Min(state->readLen + total_len - rechdrsize,
+ XLOG_BLCKSZ);
+
+ ReadPageData(needed, rechdrsize, total_len, false);
+ consumed = Min(total_len - rechdrsize, len);
+
+ memcpy(&rechdr.bytes[rechdrsize],
+ contdata,
+ consumed);
+
+ contdata += consumed;
+ rechdrsize += consumed;
+ len -= consumed;
+ }
+
+ /* If we're still not done, read from the next page */
+ if (total_len > rechdrsize && state->readLen >= XLOG_BLCKSZ)
+ {
+ int consumed,
+ needed = total_len - rechdrsize;
+ targetPagePtr += XLOG_BLCKSZ;
+
+ ReadPageData(needed, rechdrsize, total_len, false);
+ consumed = Min(total_len - rechdrsize, len);
+
+ memcpy(&rechdr.bytes[rechdrsize],
+ contdata,
+ consumed);
+
+ contdata += consumed;
+ rechdrsize += consumed;
+ len -= consumed;
}
- gotheader = false;
}
+ /*
+ * By now, the full record header is loaded; allowing us to read the
+ * record's length field.
+ */
+ total_len = XLogRecordGetLength(&rechdr.hdr);
+
+ /*
+ * Because the xlog header is loaded, we should validate it as well.
+ */
+ if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, &rechdr.hdr,
+ randAccess))
+ goto err;
+
/*
* Find space to decode this record. Don't allow oversized allocation if
* the caller requested nonblocking. Otherwise, we *have* to try to
@@ -705,12 +869,9 @@ restart:
goto err;
}
- len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
- if (total_len > len)
+ if (total_len > (XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ))
{
/* Need to reassemble record */
- char *contdata;
- XLogPageHeader pageHeader;
char *buffer;
uint32 gotlen;
@@ -728,105 +889,51 @@ restart:
goto err;
}
- /* Copy the first fragment of the record from the first page. */
- memcpy(state->readRecordBuf,
- state->readBuf + RecPtr % XLOG_BLCKSZ, len);
- buffer = state->readRecordBuf + len;
- gotlen = len;
-
- do
- {
- /* Calculate pointer to beginning of next page */
- targetPagePtr += XLOG_BLCKSZ;
-
- /* Wait for the next page to become available */
- readOff = ReadPageInternal(state, targetPagePtr,
- Min(total_len - gotlen + SizeOfXLogShortPHD,
- XLOG_BLCKSZ));
+ /* Copy the buffered record header fragment */
+ memcpy(state->readRecordBuf, &rechdr.bytes[0], rechdrsize);
- if (readOff == XLREAD_WOULDBLOCK)
- return XLREAD_WOULDBLOCK;
- else if (readOff < 0)
- goto err;
+ len = Min(len, total_len - rechdrsize);
- Assert(SizeOfXLogShortPHD <= readOff);
+ /* Copy the remaining bytes of the page */
+ memcpy(state->readRecordBuf + rechdrsize, contdata, len);
- pageHeader = (XLogPageHeader) state->readBuf;
+ buffer = state->readRecordBuf + rechdrsize + len;
+ gotlen = rechdrsize + len;
+
+ /* Finish the last bytes of this page when available */
+ if (state->readLen < XLOG_BLCKSZ)
+ {
+ int buffered = state->readLen;
+ ReadPageData((state->readLen - SizeOfXLogShortPHD) + (total_len - gotlen),
+ gotlen, total_len, true);
- /*
- * If we were expecting a continuation record and got an
- * "overwrite contrecord" flag, that means the continuation record
- * was overwritten with a different record. Restart the read by
- * assuming the address to read is the location where we found
- * this flag; but keep track of the LSN of the record we were
- * reading, for later verification.
- */
- if (pageHeader->xlp_info & XLP_FIRST_IS_OVERWRITE_CONTRECORD)
- {
- state->overwrittenRecPtr = RecPtr;
- RecPtr = targetPagePtr;
- goto restart;
- }
+ if (pageHeader->xlp_rem_len < len)
+ len = pageHeader->xlp_rem_len;
- /* Check that the continuation on next page looks valid */
- if (!(pageHeader->xlp_info & XLP_FIRST_IS_CONTRECORD))
- {
- report_invalid_record(state,
- "there is no contrecord flag at %X/%X",
- LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ buffer += buffered;
+ len -= buffered;
- /*
- * Cross-check that xlp_rem_len agrees with how much of the record
- * we expect there to be left.
- */
- if (pageHeader->xlp_rem_len == 0 ||
- total_len != (pageHeader->xlp_rem_len + gotlen))
- {
- report_invalid_record(state,
- "invalid contrecord length %u (expected %lld) at %X/%X",
- pageHeader->xlp_rem_len,
- ((long long) total_len) - gotlen,
- LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ memcpy(buffer, contdata, len);
- /* Append the continuation from this page to the buffer */
- pageHeaderSize = XLogPageHeaderSize(pageHeader);
+ buffer += len;
+ gotlen += len;
+ }
- if (readOff < pageHeaderSize)
- readOff = ReadPageInternal(state, targetPagePtr,
- pageHeaderSize);
+ do
+ {
+ /* Calculate pointer to beginning of next page */
+ targetPagePtr += XLOG_BLCKSZ;
- Assert(pageHeaderSize <= readOff);
+ ReadPageData(total_len - gotlen, gotlen, total_len, true);
- contdata = (char *) state->readBuf + pageHeaderSize;
- len = XLOG_BLCKSZ - pageHeaderSize;
if (pageHeader->xlp_rem_len < len)
len = pageHeader->xlp_rem_len;
- if (readOff < pageHeaderSize + len)
- readOff = ReadPageInternal(state, targetPagePtr,
- pageHeaderSize + len);
-
- memcpy(buffer, (char *) contdata, len);
+ memcpy(buffer, contdata, len);
buffer += len;
gotlen += len;
-
- /* If we just reassembled the record header, validate it. */
- if (!gotheader)
- {
- record = (XLogRecord *) state->readRecordBuf;
- if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr,
- record, randAccess))
- goto err;
- gotheader = true;
- }
} while (gotlen < total_len);
- Assert(gotheader);
-
record = (XLogRecord *) state->readRecordBuf;
if (!ValidXLogRecord(state, record, RecPtr))
goto err;
@@ -846,6 +953,8 @@ restart:
else if (readOff < 0)
goto err;
+ record = (XLogRecord *) (state->readBuf + targetRecOff);
+
/* Record does not cross a page boundary */
if (!ValidXLogRecord(state, record, RecPtr))
goto err;
@@ -859,7 +968,7 @@ restart:
* Special processing if it's an XLOG SWITCH record
*/
if (record->xl_rmid == RM_XLOG_ID &&
- record->xl_rminfo == XLOG_SWITCH)
+ XLogRecordGetRMInfo(record) == XLOG_SWITCH)
{
/* Pretend it extends to end of segment */
state->NextRecPtr += state->segcxt.ws_segsize - 1;
@@ -1116,12 +1225,13 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr,
XLogRecPtr PrevRecPtr, XLogRecord *record,
bool randAccess)
{
- if (record->xl_tot_len < SizeOfXLogRecord)
+ uint32 reclen = XLogRecordGetLength(record);
+ if (reclen < MinXLogHeaderSize)
{
report_invalid_record(state,
"invalid record length at %X/%X: wanted %u, got %u",
LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, record->xl_tot_len);
+ (uint32) MinXLogHeaderSize, reclen);
return false;
}
if (!RmgrIdIsValid(record->xl_rmid))
@@ -1184,16 +1294,28 @@ ValidXLogRecord(XLogReaderState *state, XLogRecord *record, XLogRecPtr recptr)
/* Calculate the CRC */
INIT_CRC32C(crc);
- COMP_CRC32C(crc, ((char *) record) + SizeOfXLogRecord, record->xl_tot_len - SizeOfXLogRecord);
- /* include the record header last */
+ COMP_CRC32C(crc,
+ ((char *) record) + offsetof(XLogRecord, xl_rmid),
+ XLogRecordGetLength(record) - offsetof(XLogRecord, xl_rmid));
+ /* include the xl_prev pointer last */
COMP_CRC32C(crc, (char *) record, offsetof(XLogRecord, xl_crc));
FIN_CRC32C(crc);
if (!EQ_CRC32C(record->xl_crc, crc))
{
report_invalid_record(state,
- "incorrect resource manager data checksum in record at %X/%X",
- LSN_FORMAT_ARGS(recptr));
+ "incorrect resource manager data checksum in record at %X/%X\n"
+ "xl_crc: record: %d; calculated: %d\n"
+ "xl_info: %x\n"
+ "xl_prev: %X/%X\n"
+ "xl_rmid: %X\n",
+ LSN_FORMAT_ARGS(recptr),
+ record->xl_crc,
+ crc,
+ record->xl_info,
+ LSN_FORMAT_ARGS(record->xl_prev),
+ record->xl_rmid);
+ Assert(false);
return false;
}
@@ -1657,11 +1779,11 @@ DecodeXLogRecord(XLogReaderState *state,
*/
#define COPY_HEADER_FIELD(_dst, _size) \
do { \
- if (remaining < _size) \
+ if (remaining < (_size)) \
goto shortdata_err; \
- memcpy(_dst, ptr, _size); \
- ptr += _size; \
- remaining -= _size; \
+ memcpy((_dst), ptr, (_size)); \
+ ptr += (_size); \
+ remaining -= (_size); \
} while(0)
char *ptr;
@@ -1671,7 +1793,7 @@ DecodeXLogRecord(XLogReaderState *state,
RelFileLocator *rlocator = NULL;
uint8 block_id;
- decoded->header = *record;
+ decoded->header = (XLRHeaderData) {0};
decoded->lsn = lsn;
decoded->next = NULL;
decoded->record_origin = InvalidRepOriginId;
@@ -1679,11 +1801,53 @@ DecodeXLogRecord(XLogReaderState *state,
decoded->main_data = NULL;
decoded->main_data_len = 0;
decoded->max_block_id = -1;
+
+ decoded->header.xl_prev = record->xl_prev;
+ decoded->header.xl_crc = record->xl_crc;
+ decoded->header.xl_rmid = record->xl_rmid;
+ decoded->header.xl_info = record->xl_info;
+ decoded->header.xl_tot_len = XLogRecordGetLength(record);
+
+ remaining = decoded->header.xl_tot_len - MinXLogHeaderSize;
ptr = (char *) record;
- ptr += SizeOfXLogRecord;
- remaining = record->xl_tot_len - SizeOfXLogRecord;
+ ptr += MinXLogHeaderSize;
+
+ switch (record->xl_info & XLR2_LEN_MASK)
+ {
+ case XLR2_LEN_ABSENT:
+ break;
+ case XLR2_LEN_1B:
+ ptr += 1;
+ remaining -= 1;
+ break;
+ case XLR2_LEN_2B:
+ ptr += 2;
+ remaining -= 2;
+ break;
+ case XLR2_LEN_4B:
+ ptr += 4;
+ remaining -= 4;
+ break;
+ default:
+ pg_unreachable();
+ }
+
+ if (record->xl_info & XLR_HAS_XID)
+ COPY_HEADER_FIELD(&decoded->header.xl_xid, sizeof(TransactionId));
+ else
+ decoded->header.xl_xid = InvalidTransactionId;
+
+ if (record->xl_info & XLR_HAS_CID)
+ COPY_HEADER_FIELD(&decoded->header.xl_cid, sizeof(CommandId));
+ else
+ decoded->header.xl_cid = InvalidCommandId;
+
+ if (record->xl_info & XLR_HAS_RMINFO)
+ COPY_HEADER_FIELD(&decoded->header.xl_rminfo, sizeof(uint8));
+ else
+ decoded->header.xl_rminfo = 0;
- /* Decode the headers */
+ /* Decode the block headers */
datatotal = 0;
while (remaining > datatotal)
{
@@ -1933,7 +2097,7 @@ DecodeXLogRecord(XLogReaderState *state,
/* Report the actual size we used. */
decoded->size = MAXALIGN(out - (char *) decoded);
- Assert(DecodeXLogRecordRequiredSpace(record->xl_tot_len) >=
+ Assert(DecodeXLogRecordRequiredSpace(decoded->header.xl_tot_len) >=
decoded->size);
return true;
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index 96d12994b5..d8c380f694 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -384,7 +384,7 @@ static char recoveryStopName[MAXFNAMELEN];
static bool recoveryStopAfter;
/* prototypes for local functions */
-static void ApplyWalRecord(XLogReaderState *xlogreader, XLogRecord *record, TimeLineID *replayTLI);
+static void ApplyWalRecord(XLogReaderState *xlogreader, XLRHeaderData *record, TimeLineID *replayTLI);
static void readRecoverySignalFile(void);
static void validateRecoveryParameters(void);
@@ -412,9 +412,9 @@ static void recoveryPausesHere(bool endOfRecovery);
static bool recoveryApplyDelay(XLogReaderState *record);
static void ConfirmRecoveryPaused(void);
-static XLogRecord *ReadRecord(XLogPrefetcher *xlogprefetcher,
- int emode, bool fetching_ckpt,
- TimeLineID replayTLI);
+static XLRHeaderData *ReadRecord(XLogPrefetcher *xlogprefetcher,
+ int emode, bool fetching_ckpt,
+ TimeLineID replayTLI);
static int XLogPageRead(XLogReaderState *xlogreader, XLogRecPtr targetPagePtr,
int reqLen, XLogRecPtr targetRecPtr, char *readBuf);
@@ -426,8 +426,8 @@ static XLogPageReadResult WaitForWALToBecomeAvailable(XLogRecPtr RecPtr,
XLogRecPtr replayLSN,
bool nonblocking);
static int emode_for_corrupt_record(int emode, XLogRecPtr RecPtr);
-static XLogRecord *ReadCheckpointRecord(XLogPrefetcher *xlogprefetcher,
- XLogRecPtr RecPtr, TimeLineID replayTLI);
+static XLRHeaderData *ReadCheckpointRecord(XLogPrefetcher *xlogprefetcher,
+ XLogRecPtr RecPtr, TimeLineID replayTLI);
static bool rescanLatestTimeLine(TimeLineID replayTLI, XLogRecPtr replayLSN);
static int XLogFileRead(XLogSegNo segno, int emode, TimeLineID tli,
XLogSource source, bool notfoundOk);
@@ -497,7 +497,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr,
XLogPageReadPrivate *private;
struct stat st;
bool wasShutdown;
- XLogRecord *record;
+ XLRHeaderData *record;
DBState dbstate_at_startup;
bool haveTblspcMap = false;
bool haveBackupLabel = false;
@@ -1566,7 +1566,7 @@ ShutdownWalRecovery(void)
void
PerformWalRecovery(void)
{
- XLogRecord *record;
+ XLRHeaderData *record;
bool reachedRecoveryTarget = false;
TimeLineID replayTLI;
@@ -1811,7 +1811,7 @@ PerformWalRecovery(void)
* Subroutine of PerformWalRecovery, to apply one WAL record.
*/
static void
-ApplyWalRecord(XLogReaderState *xlogreader, XLogRecord *record, TimeLineID *replayTLI)
+ApplyWalRecord(XLogReaderState *xlogreader, XLRHeaderData *record, TimeLineID *replayTLI)
{
ErrorContextCallback errcallback;
bool switchedTLI = false;
@@ -3004,11 +3004,11 @@ ConfirmRecoveryPaused(void)
* (emode must be either PANIC, LOG). In standby mode, retries until a valid
* record is available.
*/
-static XLogRecord *
+static XLRHeaderData *
ReadRecord(XLogPrefetcher *xlogprefetcher, int emode,
bool fetching_ckpt, TimeLineID replayTLI)
{
- XLogRecord *record;
+ XLRHeaderData *record;
XLogReaderState *xlogreader = XLogPrefetcherGetReader(xlogprefetcher);
XLogPageReadPrivate *private = (XLogPageReadPrivate *) xlogreader->private_data;
@@ -3912,11 +3912,11 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr)
/*
* Subroutine to try to fetch and validate a prior checkpoint record.
*/
-static XLogRecord *
+static XLRHeaderData *
ReadCheckpointRecord(XLogPrefetcher *xlogprefetcher, XLogRecPtr RecPtr,
TimeLineID replayTLI)
{
- XLogRecord *record;
+ XLRHeaderData *record;
uint8 rminfo;
Assert(xlogreader != NULL);
@@ -3951,7 +3951,10 @@ ReadCheckpointRecord(XLogPrefetcher *xlogprefetcher, XLogRecPtr RecPtr,
(errmsg("invalid xl_info in checkpoint record")));
return NULL;
}
- if (record->xl_tot_len != SizeOfXLogRecord + SizeOfXLogRecordDataHeaderShort + sizeof(CheckPoint))
+ if (record->xl_tot_len != (MinXLogHeaderSize
+ + sizeof(uint8) /* length field */
+ + SizeOfXLogRecordDataHeaderShort
+ + sizeof(CheckPoint)))
{
ereport(LOG,
(errmsg("invalid length of checkpoint record")));
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index 4ef46a1855..fa4be2c68b 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -194,7 +194,8 @@ log_smgrcreate(const RelFileLocator *rlocator, ForkNumber forkNum)
XLogBeginInsert();
XLogRegisterData((char *) &xlrec, sizeof(xlrec));
- XLogInsertExtended(RM_SMGR_ID, XLR_SPECIAL_REL_UPDATE, XLOG_SMGR_CREATE);
+ XLogInsertExtended(RM_SMGR_ID, XLR_SPECIAL_REL_UPDATE | XLR_HAS_XID,
+ XLOG_SMGR_CREATE, InvalidCommandId);
}
/*
@@ -376,8 +377,9 @@ RelationTruncate(Relation rel, BlockNumber nblocks)
XLogRegisterData((char *) &xlrec, sizeof(xlrec));
lsn = XLogInsertExtended(RM_SMGR_ID,
- XLR_SPECIAL_REL_UPDATE,
- XLOG_SMGR_TRUNCATE);
+ XLR_SPECIAL_REL_UPDATE | XLR_HAS_XID,
+ XLOG_SMGR_TRUNCATE,
+ InvalidCommandId);
/*
* Flush, because otherwise the truncation of the main relation might
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 25c4672917..fea070d768 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -624,9 +624,8 @@ CreateDatabaseUsingFileCopy(Oid src_dboid, Oid dst_dboid, Oid src_tsid,
XLogRegisterData((char *) &xlrec,
sizeof(xl_dbase_create_file_copy_rec));
- (void) XLogInsertExtended(RM_DBASE_ID,
- XLR_SPECIAL_REL_UPDATE,
- XLOG_DBASE_CREATE_FILE_COPY);
+ (void) XLogInsertExtended(RM_DBASE_ID, XLR_SPECIAL_REL_UPDATE,
+ XLOG_DBASE_CREATE_FILE_COPY, InvalidCommandId);
}
pfree(srcpath);
pfree(dstpath);
@@ -2022,9 +2021,8 @@ movedb(const char *dbname, const char *tblspcname)
XLogRegisterData((char *) &xlrec,
sizeof(xl_dbase_create_file_copy_rec));
- (void) XLogInsertExtended(RM_DBASE_ID,
- XLR_SPECIAL_REL_UPDATE,
- XLOG_DBASE_CREATE_FILE_COPY);
+ (void) XLogInsertExtended(RM_DBASE_ID, XLR_SPECIAL_REL_UPDATE,
+ XLOG_DBASE_CREATE_FILE_COPY, InvalidCommandId);
}
/*
@@ -2117,9 +2115,8 @@ movedb(const char *dbname, const char *tblspcname)
XLogRegisterData((char *) &xlrec, sizeof(xl_dbase_drop_rec));
XLogRegisterData((char *) &src_tblspcoid, sizeof(Oid));
- (void) XLogInsertExtended(RM_DBASE_ID,
- XLR_SPECIAL_REL_UPDATE,
- XLOG_DBASE_DROP);
+ (void) XLogInsertExtended(RM_DBASE_ID, XLR_SPECIAL_REL_UPDATE,
+ XLOG_DBASE_DROP, InvalidCommandId);
}
/* Now it's safe to release the database lock */
@@ -2837,9 +2834,8 @@ remove_dbtablespaces(Oid db_id)
XLogRegisterData((char *) &xlrec, MinSizeOfDbaseDropRec);
XLogRegisterData((char *) tablespace_ids, ntblspc * sizeof(Oid));
- (void) XLogInsertExtended(RM_DBASE_ID,
- XLR_SPECIAL_REL_UPDATE,
- XLOG_DBASE_DROP);
+ (void) XLogInsertExtended(RM_DBASE_ID, XLR_SPECIAL_REL_UPDATE,
+ XLOG_DBASE_DROP, InvalidCommandId);
}
list_free(ltblspc);
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index d6abeb9a9d..7cbbb9b73a 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -200,7 +200,10 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
ParseCommitRecord(XLogRecGetRmInfo(buf->record), xlrec, &parsed);
if (!TransactionIdIsValid(parsed.twophase_xid))
+ {
+ Assert(info == XLOG_XACT_COMMIT);
xid = XLogRecGetXid(r);
+ }
else
xid = parsed.twophase_xid;
@@ -228,7 +231,10 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
ParseAbortRecord(XLogRecGetRmInfo(buf->record), xlrec, &parsed);
if (!TransactionIdIsValid(parsed.twophase_xid))
+ {
+ Assert(info == XLOG_XACT_ABORT);
xid = XLogRecGetXid(r);
+ }
else
xid = parsed.twophase_xid;
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 625a7f4273..6faa519a35 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -600,7 +600,7 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
/* Wait for a consistent starting point */
for (;;)
{
- XLogRecord *record;
+ XLRHeaderData *record;
char *err = NULL;
/* the read_page callback waits for new WAL */
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index 7fa2b2cba7..a2a13c8ae4 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -256,7 +256,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
/* Decode until we run out of records */
while (ctx->reader->EndRecPtr < end_of_wal)
{
- XLogRecord *record;
+ XLRHeaderData *record;
char *errm = NULL;
record = XLogReadRecord(ctx->reader, &errm);
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ca945994ef..6ffecd0fcc 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -495,7 +495,7 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
while (ctx->reader->EndRecPtr < moveto)
{
char *errm = NULL;
- XLogRecord *record;
+ XLRHeaderData *record;
/*
* Read records. No changes are generated in fast_forward mode,
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index e9ba500a15..8129a2552b 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -3019,7 +3019,7 @@ retry:
static void
XLogSendLogical(void)
{
- XLogRecord *record;
+ XLRHeaderData *record;
char *errm;
/*
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index eb5782f82a..78f17b5d09 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -1632,6 +1632,7 @@ LogLogicalInvalidations(void)
ProcessMessageSubGroupMulti(group, RelCacheMsgs,
XLogRegisterData((char *) msgs,
n * sizeof(SharedInvalidationMessage)));
- XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+ XLogInsertExtended(RM_XACT_ID, XLR_HAS_XID,
+ XLOG_XACT_INVALIDATIONS, InvalidCommandId);
}
}
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 089063f471..380d0a260c 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -1046,6 +1046,9 @@ WriteEmptyXLOG(void)
int fd;
int nbytes;
char *recptr;
+ const uint32 rec_tot_len = MinXLogHeaderSize + sizeof(uint8) /* xl_tot_len */
+ + SizeOfXLogRecordDataHeaderShort + sizeof(CheckPoint);
+ const uint8 rec_len = rec_tot_len;
memset(buffer.data, 0, XLOG_BLCKSZ);
@@ -1060,23 +1063,33 @@ WriteEmptyXLOG(void)
longpage->xlp_seg_size = WalSegSz;
longpage->xlp_xlog_blcksz = XLOG_BLCKSZ;
+ /*
+ * Ensure that the length actually fits within the 8-bit length field in
+ * the header.
+ */
+ Assert(rec_tot_len < UINT8_MAX);
+
/* Insert the initial checkpoint record */
recptr = (char *) page + SizeOfXLogLongPHD;
record = (XLogRecord *) recptr;
record->xl_prev = 0;
- record->xl_xid = InvalidTransactionId;
- record->xl_tot_len = SizeOfXLogRecord + SizeOfXLogRecordDataHeaderShort + sizeof(CheckPoint);
- record->xl_info = XLOG_CHECKPOINT_SHUTDOWN;
+ record->xl_info = XLR2_LEN_1B;
record->xl_rmid = RM_XLOG_ID;
- recptr += SizeOfXLogRecord;
+ recptr += MinXLogHeaderSize;
+
+ memcpy(recptr, (char *) &rec_len, sizeof(uint8));
+ recptr += sizeof(uint8);
+
*(recptr++) = (char) XLR_BLOCK_ID_DATA_SHORT;
*(recptr++) = sizeof(CheckPoint);
memcpy(recptr, &ControlFile.checkPointCopy,
sizeof(CheckPoint));
INIT_CRC32C(crc);
- COMP_CRC32C(crc, ((char *) record) + SizeOfXLogRecord, record->xl_tot_len - SizeOfXLogRecord);
+ COMP_CRC32C(crc,
+ ((char *) record) + offsetof(XLogRecord, xl_rmid),
+ rec_len - offsetof(XLogRecord, xl_rmid));
COMP_CRC32C(crc, (char *) record, offsetof(XLogRecord, xl_crc));
FIN_CRC32C(crc);
record->xl_crc = crc;
diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c
index 132c4db65e..fc2dccb63c 100644
--- a/src/bin/pg_rewind/parsexlog.c
+++ b/src/bin/pg_rewind/parsexlog.c
@@ -66,7 +66,7 @@ void
extractPageMap(const char *datadir, XLogRecPtr startpoint, int tliIndex,
XLogRecPtr endpoint, const char *restoreCommand)
{
- XLogRecord *record;
+ XLRHeaderData *record;
XLogReaderState *xlogreader;
char *errormsg;
XLogPageReadPrivate private;
@@ -124,7 +124,7 @@ XLogRecPtr
readOneRecord(const char *datadir, XLogRecPtr ptr, int tliIndex,
const char *restoreCommand)
{
- XLogRecord *record;
+ XLRHeaderData *record;
XLogReaderState *xlogreader;
char *errormsg;
XLogPageReadPrivate private;
@@ -170,7 +170,7 @@ findLastCheckpoint(const char *datadir, XLogRecPtr forkptr, int tliIndex,
XLogRecPtr *lastchkptredo, const char *restoreCommand)
{
/* Walk backwards, starting from the given record */
- XLogRecord *record;
+ XLRHeaderData *record;
XLogRecPtr searchptr;
XLogReaderState *xlogreader;
char *errormsg;
diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c
index 70886beedd..a821d7b40a 100644
--- a/src/bin/pg_waldump/pg_waldump.c
+++ b/src/bin/pg_waldump/pg_waldump.c
@@ -696,7 +696,7 @@ main(int argc, char **argv)
XLogDumpPrivate private;
XLogDumpConfig config;
XLogStats stats;
- XLogRecord *record;
+ XLRHeaderData *record;
XLogRecPtr first_record;
char *waldir = NULL;
char *errormsg;
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index cfe53c7175..f41bca6cea 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -42,7 +42,7 @@
extern void XLogBeginInsert(void);
extern void XLogSetRecordFlags(uint8 flags);
extern XLogRecPtr XLogInsert(RmgrId rmid, uint8 rminfo);
-extern XLogRecPtr XLogInsertExtended(RmgrId rmid, uint8 info, uint8 rminfo);
+extern XLogRecPtr XLogInsertExtended(RmgrId rmid, uint8 info, uint8 rminfo, CommandId cid);
extern void XLogEnsureRecordSpace(int max_block_id, int ndatas);
extern void XLogRegisterData(char *data, uint32 len);
extern void XLogRegisterBuffer(uint8 block_id, Buffer buffer, uint8 flags);
diff --git a/src/include/access/xlogprefetcher.h b/src/include/access/xlogprefetcher.h
index fdd67fcedd..fbfd9e02a4 100644
--- a/src/include/access/xlogprefetcher.h
+++ b/src/include/access/xlogprefetcher.h
@@ -47,8 +47,8 @@ extern XLogReaderState *XLogPrefetcherGetReader(XLogPrefetcher *prefetcher);
extern void XLogPrefetcherBeginRead(XLogPrefetcher *prefetcher,
XLogRecPtr recPtr);
-extern XLogRecord *XLogPrefetcherReadRecord(XLogPrefetcher *prefetcher,
- char **errmsg);
+extern XLRHeaderData *XLogPrefetcherReadRecord(XLogPrefetcher *prefetcher,
+ char **errmsg);
extern void XLogPrefetcherComputeStats(XLogPrefetcher *prefetcher);
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index fb6cae08ad..9a31aec414 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -163,7 +163,7 @@ typedef struct DecodedXLogRecord
/* Public members. */
XLogRecPtr lsn; /* location */
XLogRecPtr next_lsn; /* location of next record */
- XLogRecord header; /* header */
+ XLRHeaderData header; /* header */
RepOriginId record_origin;
TransactionId toplevel_xid; /* XID of top-level transaction */
char *main_data; /* record's main data portion */
@@ -355,8 +355,8 @@ typedef enum XLogPageReadResult
} XLogPageReadResult;
/* Read the next XLog record. Returns NULL on end-of-WAL or failure */
-extern struct XLogRecord *XLogReadRecord(XLogReaderState *state,
- char **errormsg);
+extern struct XLRHeaderData *XLogReadRecord(XLogReaderState *state,
+ char **errormsg);
/* Consume the next record or error. */
extern DecodedXLogRecord *XLogNextRecord(XLogReaderState *state,
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 17093d93b6..94125f9878 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -19,7 +19,11 @@
/*
* The overall layout of an XLOG record is:
- * Fixed-size header (XLogRecord struct)
+ * Fixed-size header (XLogRecord struct) + variable-sized header data:
+ * - xl_cid (0 or 4 bytes)
+ * - xl_cid (0 or 4 bytes)
+ * - xl_rmgr_flags (0 or 1 byte)
+ * - xl_len (0, 1, 2 or 4 bytes)
* XLogRecordBlockHeader struct
* XLogRecordBlockHeader struct
* ...
@@ -37,23 +41,97 @@
* The XLogRecordBlockHeader, XLogRecordDataHeaderShort and
* XLogRecordDataHeaderLong structs all begin with a single 'id' byte. It's
* used to distinguish between block references, and the main data structs.
+ *
+ * The smallest size that XLogRecord header takes up is now 14 bytes: 8 bytes
+ * in xl_prev, 4 in checksum, and 1 in xl_rmid and xl_info each, while the
+ * max-sized xlog header now takes up 27 bytes; with 4 bytes each in
+ * xl_tot_len, xl_xid and xl_cid, plus one in xl_rminfo.
*/
-typedef struct XLogRecord
-{
- uint32 xl_tot_len; /* total len of entire record */
- TransactionId xl_xid; /* xact id */
- XLogRecPtr xl_prev; /* ptr to previous record in log */
- uint8 xl_info; /* flag bits, see below */
- RmgrId xl_rmid; /* resource manager for this record */
- uint8 xl_rminfo; /* flag bits for rmgr use */
- /* 1 byte of padding here, initialize to zero */
- pg_crc32c xl_crc; /* CRC for this record */
-
- /* XLogRecordBlockHeaders and XLogRecordDataHeader follow, no padding */
+typedef struct XLogRecord {
+ XLogRecPtr xl_prev;
+ pg_crc32c xl_crc;
+ RmgrId xl_rmid;
+ /* Flags for record handling and variable-length header fields */
+ uint8 xl_info;
+ /*
+ * Without padding:
+ * - depending on flags, length field follows (0, 1, 2 or 4 bytes)
+ * - if HAS_XID, TransactionId follows
+ * - if HAS_CID, CommandID follows
+ * - if HAS_RMINFO, uint8 with rminfo flags follows
+ * - XLogRecordBlockHeaders and XLogRecordDataHeader follow
+ */
} XLogRecord;
-#define SizeOfXLogRecord (offsetof(XLogRecord, xl_crc) + sizeof(pg_crc32c))
+/*
+ *
+ */
+typedef struct XLRHeaderData {
+ XLogRecPtr xl_prev;
+ pg_crc32c xl_crc;
+ RmgrId xl_rmid;
+ uint8 xl_info;
+ TransactionId xl_xid;
+ CommandId xl_cid;
+ uint8 xl_rminfo;
+ uint32 xl_tot_len;
+} XLRHeaderData;
+
+#define MinXLogHeaderSize ( \
+ offsetof(XLogRecord, xl_info) \
+ + sizeof(uint8) /* xl_info */ \
+)
+
+#define MaxXLogHeaderSize ( \
+ MinXLogHeaderSize \
+ + sizeof(TransactionId) /* xl_xid */ \
+ + sizeof(CommandId) /* xl_cid */ \
+ + sizeof(uint8) /* xl_rminfo */ \
+ + sizeof(uint32) /* xl_len */ \
+)
+
+/*
+ * Mask for getting the size of the length field
+ */
+#define XLR2_LEN_MASK (0x03)
+
+/*
+ * IFF the record does not contain any registered data, the length field will
+ * be absent (as the size of a plain record is knowable from just the
+ * fixed-size struct's flags)
+ */
+#define XLR2_LEN_ABSENT 0x00
+/*
+ * Size of the xlog record is <= 255 bytes
+ */
+#define XLR2_LEN_1B 0x01
+/*
+ * Size of the xlog record is <= (2^16 - 1)
+ */
+#define XLR2_LEN_2B 0x02
+/*
+ * Size of the xlog record is <= (2^32 - 1)
+ */
+#define XLR2_LEN_4B 0x03
+
+/*
+ * Does this record contain an XID? This must be included if the data has
+ * transactional visibility.
+ */
+#define XLR_HAS_XID 0x04
+
+/*
+ * Doest this record contain a CID? This must be included if the data has
+ * transactional visibility, and remote snapshot transfer support is enabled.
+ */
+#define XLR_HAS_CID 0x08
+
+/*
+ * If the redo manager needs non-zero bits in the header to discern different
+ * types of WAL records, this flag is set.
+ */
+#define XLR_HAS_RMINFO 0x10
/*
* If a WAL record modifies any relation files, in ways not covered by the
@@ -61,7 +139,7 @@ typedef struct XLogRecord
* by PostgreSQL itself, but it allows external tools that read WAL and keep
* track of modified blocks to recognize such special record types.
*/
-#define XLR_SPECIAL_REL_UPDATE 0x01
+#define XLR_SPECIAL_REL_UPDATE 0x20
/*
* Enforces consistency checks of replayed WAL at recovery. If enabled,
@@ -70,7 +148,68 @@ typedef struct XLogRecord
* of XLogInsert can use this value if necessary, but if
* wal_consistency_checking is enabled for a rmgr this is set unconditionally.
*/
-#define XLR_CHECK_CONSISTENCY 0x02
+#define XLR_CHECK_CONSISTENCY 0x40
+
+#define XLR_INFO_USERFLAGS ( \
+ XLR_HAS_XID \
+ | XLR_HAS_CID \
+ | XLR_SPECIAL_REL_UPDATE \
+ | XLR_CHECK_CONSISTENCY \
+)
+
+#define XLRSizeOfHeader(infomask) ( \
+ MinXLogHeaderSize \
+ + ((1 << ((infomask) & XLR2_LEN_MASK)) >> 1) \
+ + (((infomask) & XLR_HAS_XID) ? sizeof(TransactionId) : 0) \
+ + (((infomask) & XLR_HAS_CID) ? sizeof(CommandId) : 0) \
+ + (((infomask) & XLR_HAS_RMINFO) ? sizeof(uint8) : 0) \
+)
+
+static inline uint32
+XLogRecordGetLength(XLogRecord *record)
+{
+ char *lenptr = (char *) record;
+ uint8 len8;
+ uint16 len16;
+ uint32 len32;
+
+ lenptr += MinXLogHeaderSize;
+
+ switch ((record->xl_info) & XLR2_LEN_MASK) {
+ case XLR2_LEN_ABSENT:
+ return XLRSizeOfHeader(record->xl_info);
+ case XLR2_LEN_1B:
+ memcpy(&len8, lenptr, sizeof(uint8));
+ return (uint32) len8;
+ case XLR2_LEN_2B:
+ memcpy(&len16, lenptr, sizeof(uint16));
+ return (uint32) len16;
+ case XLR2_LEN_4B:
+ memcpy(&len32, lenptr, sizeof(uint32));
+ return (uint32) len32;
+ default:
+ pg_unreachable();
+ }
+}
+
+static inline int8
+XLogRecordGetRMInfo(XLogRecord *record)
+{
+ int infooff = MinXLogHeaderSize;
+
+ if (!(record->xl_info & XLR_HAS_RMINFO))
+ return 0;
+
+ infooff += (1 << (record->xl_info & XLR2_LEN_MASK)) >> 1;
+
+ if (record->xl_info & XLR_HAS_XID)
+ infooff += sizeof(TransactionId);
+
+ if (record->xl_info & XLR_HAS_CID)
+ infooff += sizeof(CommandId);
+
+ return *(((uint8 *) record) + infooff);
+}
/*
* Header info for block data appended to an XLOG record.
@@ -145,7 +284,7 @@ typedef struct XLogRecordBlockImageHeader
#define BKPIMAGE_COMPRESS_ZSTD 0x10
#define BKPIMAGE_COMPRESSED(info) \
- ((info & (BKPIMAGE_COMPRESS_PGLZ | BKPIMAGE_COMPRESS_LZ4 | \
+ (((info) & (BKPIMAGE_COMPRESS_PGLZ | BKPIMAGE_COMPRESS_LZ4 | \
BKPIMAGE_COMPRESS_ZSTD)) != 0)
/*
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 9f644a0c1b..dfafbc36df 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -69,7 +69,12 @@ ignore: random
# aggregates depends on create_aggregate
# join depends on create_misc
# ----------
-test: select_into select_distinct select_distinct_on select_implicit select_having subselect union case join aggregates transactions random portals arrays btree_index hash_index update delete namespace prepared_xacts
+test: select_into select_distinct select_distinct_on select_implicit select_having subselect union case join aggregates transactions random portals arrays btree_index hash_index update delete namespace
+
+# ----------
+# Another group of parallel tests
+# ----------
+# test: prepared_xacts
# ----------
# Another group of parallel tests
--
2.30.2
[application/x-patch] 0001-Move-rmgr-specific-flags-out-of-xl_info-into-new-uin.patch (110.5K, ../../CAEze2Wjd3jY_UhhOGdGGnC6NO=+NmtNOmd=JaYv-v-nwBAiXXA@mail.gmail.com/4-0001-Move-rmgr-specific-flags-out-of-xl_info-into-new-uin.patch)
download | inline diff:
From e8a87c56860e605cbef8cb5d712414441c300f5c Mon Sep 17 00:00:00 2001
From: Matthias van de Meent <[email protected]>
Date: Fri, 9 Sep 2022 15:38:55 +0200
Subject: [PATCH 1/3] Move rmgr-specific flags out of xl_info into new uint8
xl_rminfo field.
This field doens't use extra data as it is located in the previously 2-byte
alignment padding in the xlog header behind xl_rmid.
The field is used to allow redo managers more bits for their own use without
growing the xlog record size.
---
contrib/pg_walinspect/pg_walinspect.c | 4 +-
src/backend/access/brin/brin_pageops.c | 16 +++---
src/backend/access/brin/brin_xlog.c | 8 +--
src/backend/access/gin/ginxlog.c | 6 +--
src/backend/access/gist/gistxlog.c | 6 +--
src/backend/access/hash/hash_xlog.c | 6 +--
src/backend/access/heap/heapam.c | 40 +++++++--------
src/backend/access/nbtree/nbtinsert.c | 18 +++----
src/backend/access/nbtree/nbtpage.c | 8 +--
src/backend/access/nbtree/nbtxlog.c | 10 ++--
src/backend/access/rmgrdesc/brindesc.c | 20 ++++----
src/backend/access/rmgrdesc/clogdesc.c | 10 ++--
src/backend/access/rmgrdesc/committsdesc.c | 10 ++--
src/backend/access/rmgrdesc/dbasedesc.c | 12 ++---
src/backend/access/rmgrdesc/genericdesc.c | 2 +-
src/backend/access/rmgrdesc/gindesc.c | 8 +--
src/backend/access/rmgrdesc/gistdesc.c | 8 +--
src/backend/access/rmgrdesc/hashdesc.c | 8 +--
src/backend/access/rmgrdesc/heapdesc.c | 46 ++++++++---------
src/backend/access/rmgrdesc/logicalmsgdesc.c | 8 +--
src/backend/access/rmgrdesc/mxactdesc.c | 14 ++---
src/backend/access/rmgrdesc/nbtdesc.c | 8 +--
src/backend/access/rmgrdesc/relmapdesc.c | 8 +--
src/backend/access/rmgrdesc/replorigindesc.c | 8 +--
src/backend/access/rmgrdesc/seqdesc.c | 8 +--
src/backend/access/rmgrdesc/smgrdesc.c | 10 ++--
src/backend/access/rmgrdesc/spgdesc.c | 8 +--
src/backend/access/rmgrdesc/standbydesc.c | 12 ++---
src/backend/access/rmgrdesc/tblspcdesc.c | 10 ++--
src/backend/access/rmgrdesc/xactdesc.c | 34 ++++++------
src/backend/access/rmgrdesc/xlogdesc.c | 28 +++++-----
src/backend/access/spgist/spgxlog.c | 6 +--
src/backend/access/transam/clog.c | 8 +--
src/backend/access/transam/commit_ts.c | 8 +--
src/backend/access/transam/multixact.c | 18 +++----
src/backend/access/transam/twophase.c | 2 +-
src/backend/access/transam/xact.c | 36 +++++++------
src/backend/access/transam/xlog.c | 34 ++++++------
src/backend/access/transam/xloginsert.c | 31 ++++++++---
src/backend/access/transam/xlogprefetcher.c | 2 +-
src/backend/access/transam/xlogreader.c | 2 +-
src/backend/access/transam/xlogrecovery.c | 54 ++++++++++----------
src/backend/access/transam/xlogstats.c | 2 +-
src/backend/catalog/storage.c | 15 +++---
src/backend/commands/dbcommands.c | 30 ++++++-----
src/backend/commands/sequence.c | 6 +--
src/backend/commands/tablespace.c | 8 +--
src/backend/replication/logical/decode.c | 38 +++++++-------
src/backend/replication/logical/message.c | 6 +--
src/backend/replication/logical/origin.c | 6 +--
src/backend/storage/ipc/standby.c | 10 ++--
src/backend/utils/cache/relmapper.c | 6 +--
src/bin/pg_rewind/parsexlog.c | 10 ++--
src/bin/pg_waldump/pg_waldump.c | 6 +--
src/include/access/brin_xlog.h | 2 +-
src/include/access/clog.h | 2 +-
src/include/access/ginxlog.h | 2 +-
src/include/access/gistxlog.h | 2 +-
src/include/access/hash_xlog.h | 2 +-
src/include/access/heapam_xlog.h | 4 +-
src/include/access/multixact.h | 2 +-
src/include/access/nbtxlog.h | 2 +-
src/include/access/spgxlog.h | 2 +-
src/include/access/xact.h | 6 +--
src/include/access/xlog.h | 2 +-
src/include/access/xloginsert.h | 3 +-
src/include/access/xlogreader.h | 1 +
src/include/access/xlogrecord.h | 11 +---
src/include/access/xlogstats.h | 2 +-
src/include/catalog/storage_xlog.h | 2 +-
src/include/commands/dbcommands_xlog.h | 2 +-
src/include/commands/sequence.h | 2 +-
src/include/commands/tablespace.h | 2 +-
src/include/replication/message.h | 2 +-
src/include/storage/standbydefs.h | 2 +-
src/include/utils/relmapper.h | 2 +-
76 files changed, 411 insertions(+), 394 deletions(-)
diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c
index 38fb4106da..f4e3b40bed 100644
--- a/contrib/pg_walinspect/pg_walinspect.c
+++ b/contrib/pg_walinspect/pg_walinspect.c
@@ -188,10 +188,10 @@ GetWALRecordInfo(XLogReaderState *record, Datum *values,
int i = 0;
desc = GetRmgr(XLogRecGetRmid(record));
- id = desc.rm_identify(XLogRecGetInfo(record));
+ id = desc.rm_identify(XLogRecGetRmInfo(record));
if (id == NULL)
- id = psprintf("UNKNOWN (%x)", XLogRecGetInfo(record) & ~XLR_INFO_MASK);
+ id = psprintf("UNKNOWN (%x)", XLogRecGetRmInfo(record));
initStringInfo(&rec_desc);
desc.rm_desc(&rec_desc, record);
diff --git a/src/backend/access/brin/brin_pageops.c b/src/backend/access/brin/brin_pageops.c
index f17aad51b6..150d6f9793 100644
--- a/src/backend/access/brin/brin_pageops.c
+++ b/src/backend/access/brin/brin_pageops.c
@@ -186,7 +186,7 @@ brin_doupdate(Relation idxrel, BlockNumber pagesPerRange,
{
xl_brin_samepage_update xlrec;
XLogRecPtr recptr;
- uint8 info = XLOG_BRIN_SAMEPAGE_UPDATE;
+ uint8 rminfo = XLOG_BRIN_SAMEPAGE_UPDATE;
xlrec.offnum = oldoff;
@@ -196,7 +196,7 @@ brin_doupdate(Relation idxrel, BlockNumber pagesPerRange,
XLogRegisterBuffer(0, oldbuf, REGBUF_STANDARD);
XLogRegisterBufData(0, (char *) unconstify(BrinTuple *, newtup), newsz);
- recptr = XLogInsert(RM_BRIN_ID, info);
+ recptr = XLogInsert(RM_BRIN_ID, rminfo);
PageSetLSN(oldpage, recptr);
}
@@ -271,9 +271,9 @@ brin_doupdate(Relation idxrel, BlockNumber pagesPerRange,
{
xl_brin_update xlrec;
XLogRecPtr recptr;
- uint8 info;
+ uint8 rminfo;
- info = XLOG_BRIN_UPDATE | (extended ? XLOG_BRIN_INIT_PAGE : 0);
+ rminfo = XLOG_BRIN_UPDATE | (extended ? XLOG_BRIN_INIT_PAGE : 0);
xlrec.insert.offnum = newoff;
xlrec.insert.heapBlk = heapBlk;
@@ -294,7 +294,7 @@ brin_doupdate(Relation idxrel, BlockNumber pagesPerRange,
/* old page */
XLogRegisterBuffer(2, oldbuf, REGBUF_STANDARD);
- recptr = XLogInsert(RM_BRIN_ID, info);
+ recptr = XLogInsert(RM_BRIN_ID, rminfo);
PageSetLSN(oldpage, recptr);
PageSetLSN(newpage, recptr);
@@ -428,9 +428,9 @@ brin_doinsert(Relation idxrel, BlockNumber pagesPerRange,
{
xl_brin_insert xlrec;
XLogRecPtr recptr;
- uint8 info;
+ uint8 rminfo;
- info = XLOG_BRIN_INSERT | (extended ? XLOG_BRIN_INIT_PAGE : 0);
+ rminfo = XLOG_BRIN_INSERT | (extended ? XLOG_BRIN_INIT_PAGE : 0);
xlrec.heapBlk = heapBlk;
xlrec.pagesPerRange = pagesPerRange;
xlrec.offnum = off;
@@ -443,7 +443,7 @@ brin_doinsert(Relation idxrel, BlockNumber pagesPerRange,
XLogRegisterBuffer(1, revmapbuf, 0);
- recptr = XLogInsert(RM_BRIN_ID, info);
+ recptr = XLogInsert(RM_BRIN_ID, rminfo);
PageSetLSN(page, recptr);
PageSetLSN(BufferGetPage(revmapbuf), recptr);
diff --git a/src/backend/access/brin/brin_xlog.c b/src/backend/access/brin/brin_xlog.c
index af6949882a..c0a23bdca6 100644
--- a/src/backend/access/brin/brin_xlog.c
+++ b/src/backend/access/brin/brin_xlog.c
@@ -56,7 +56,7 @@ brin_xlog_insert_update(XLogReaderState *record,
* If we inserted the first and only tuple on the page, re-initialize the
* page from scratch.
*/
- if (XLogRecGetInfo(record) & XLOG_BRIN_INIT_PAGE)
+ if (XLogRecGetRmInfo(record) & XLOG_BRIN_INIT_PAGE)
{
buffer = XLogInitBufferForRedo(record, 0);
page = BufferGetPage(buffer);
@@ -308,9 +308,9 @@ brin_xlog_desummarize_page(XLogReaderState *record)
void
brin_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- switch (info & XLOG_BRIN_OPMASK)
+ switch (rminfo & XLOG_BRIN_OPMASK)
{
case XLOG_BRIN_CREATE_INDEX:
brin_xlog_createidx(record);
@@ -331,7 +331,7 @@ brin_redo(XLogReaderState *record)
brin_xlog_desummarize_page(record);
break;
default:
- elog(PANIC, "brin_redo: unknown op code %u", info);
+ elog(PANIC, "brin_redo: unknown op code %u", rminfo);
}
}
diff --git a/src/backend/access/gin/ginxlog.c b/src/backend/access/gin/ginxlog.c
index 41b92115bf..e6a28ef6ce 100644
--- a/src/backend/access/gin/ginxlog.c
+++ b/src/backend/access/gin/ginxlog.c
@@ -725,7 +725,7 @@ ginRedoDeleteListPages(XLogReaderState *record)
void
gin_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
MemoryContext oldCtx;
/*
@@ -735,7 +735,7 @@ gin_redo(XLogReaderState *record)
*/
oldCtx = MemoryContextSwitchTo(opCtx);
- switch (info)
+ switch (rminfo)
{
case XLOG_GIN_CREATE_PTREE:
ginRedoCreatePTree(record);
@@ -765,7 +765,7 @@ gin_redo(XLogReaderState *record)
ginRedoDeleteListPages(record);
break;
default:
- elog(PANIC, "gin_redo: unknown op code %u", info);
+ elog(PANIC, "gin_redo: unknown op code %u", rminfo);
}
MemoryContextSwitchTo(oldCtx);
MemoryContextReset(opCtx);
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index 998befd2cb..940d9cd90e 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -402,7 +402,7 @@ gistRedoPageReuse(XLogReaderState *record)
void
gist_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
MemoryContext oldCxt;
/*
@@ -412,7 +412,7 @@ gist_redo(XLogReaderState *record)
*/
oldCxt = MemoryContextSwitchTo(opCtx);
- switch (info)
+ switch (rminfo)
{
case XLOG_GIST_PAGE_UPDATE:
gistRedoPageUpdateRecord(record);
@@ -433,7 +433,7 @@ gist_redo(XLogReaderState *record)
/* nop. See gistGetFakeLSN(). */
break;
default:
- elog(PANIC, "gist_redo: unknown op code %u", info);
+ elog(PANIC, "gist_redo: unknown op code %u", rminfo);
}
MemoryContextSwitchTo(oldCxt);
diff --git a/src/backend/access/hash/hash_xlog.c b/src/backend/access/hash/hash_xlog.c
index e88213c742..9bc8f4edd4 100644
--- a/src/backend/access/hash/hash_xlog.c
+++ b/src/backend/access/hash/hash_xlog.c
@@ -1052,9 +1052,9 @@ hash_xlog_vacuum_one_page(XLogReaderState *record)
void
hash_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- switch (info)
+ switch (rminfo)
{
case XLOG_HASH_INIT_META_PAGE:
hash_xlog_init_meta_page(record);
@@ -1096,7 +1096,7 @@ hash_redo(XLogReaderState *record)
hash_xlog_vacuum_one_page(record);
break;
default:
- elog(PANIC, "hash_redo: unknown op code %u", info);
+ elog(PANIC, "hash_redo: unknown op code %u", rminfo);
}
}
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index bd4d85041d..94e702cdb2 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2104,7 +2104,7 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
xl_heap_header xlhdr;
XLogRecPtr recptr;
Page page = BufferGetPage(buffer);
- uint8 info = XLOG_HEAP_INSERT;
+ uint8 rminfo = XLOG_HEAP_INSERT;
int bufflags = 0;
/*
@@ -2122,7 +2122,7 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
if (ItemPointerGetOffsetNumber(&(heaptup->t_self)) == FirstOffsetNumber &&
PageGetMaxOffsetNumber(page) == FirstOffsetNumber)
{
- info |= XLOG_HEAP_INIT_PAGE;
+ rminfo |= XLOG_HEAP_INIT_PAGE;
bufflags |= REGBUF_WILL_INIT;
}
@@ -2171,7 +2171,7 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
/* filtering by origin on a row level is much more efficient */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- recptr = XLogInsert(RM_HEAP_ID, info);
+ recptr = XLogInsert(RM_HEAP_ID, rminfo);
PageSetLSN(page, recptr);
}
@@ -2415,7 +2415,7 @@ heap_multi_insert(Relation relation, TupleTableSlot **slots, int ntuples,
{
XLogRecPtr recptr;
xl_heap_multi_insert *xlrec;
- uint8 info = XLOG_HEAP2_MULTI_INSERT;
+ uint8 rminfo = XLOG_HEAP2_MULTI_INSERT;
char *tupledata;
int totaldatalen;
char *scratchptr = scratch.data;
@@ -2499,7 +2499,7 @@ heap_multi_insert(Relation relation, TupleTableSlot **slots, int ntuples,
if (init)
{
- info |= XLOG_HEAP_INIT_PAGE;
+ rminfo |= XLOG_HEAP_INIT_PAGE;
bufflags |= REGBUF_WILL_INIT;
}
@@ -2519,7 +2519,7 @@ heap_multi_insert(Relation relation, TupleTableSlot **slots, int ntuples,
/* filtering by origin on a row level is much more efficient */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- recptr = XLogInsert(RM_HEAP2_ID, info);
+ recptr = XLogInsert(RM_HEAP2_ID, rminfo);
PageSetLSN(page, recptr);
}
@@ -8237,7 +8237,7 @@ log_heap_update(Relation reln, Buffer oldbuf,
xl_heap_update xlrec;
xl_heap_header xlhdr;
xl_heap_header xlhdr_idx;
- uint8 info;
+ uint8 rminfo;
uint16 prefix_suffix[2];
uint16 prefixlen = 0,
suffixlen = 0;
@@ -8253,9 +8253,9 @@ log_heap_update(Relation reln, Buffer oldbuf,
XLogBeginInsert();
if (HeapTupleIsHeapOnly(newtup))
- info = XLOG_HEAP_HOT_UPDATE;
+ rminfo = XLOG_HEAP_HOT_UPDATE;
else
- info = XLOG_HEAP_UPDATE;
+ rminfo = XLOG_HEAP_UPDATE;
/*
* If the old and new tuple are on the same page, we only need to log the
@@ -8335,7 +8335,7 @@ log_heap_update(Relation reln, Buffer oldbuf,
if (ItemPointerGetOffsetNumber(&(newtup->t_self)) == FirstOffsetNumber &&
PageGetMaxOffsetNumber(page) == FirstOffsetNumber)
{
- info |= XLOG_HEAP_INIT_PAGE;
+ rminfo |= XLOG_HEAP_INIT_PAGE;
init = true;
}
else
@@ -8439,7 +8439,7 @@ log_heap_update(Relation reln, Buffer oldbuf,
/* filtering by origin on a row level is much more efficient */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- recptr = XLogInsert(RM_HEAP_ID, info);
+ recptr = XLogInsert(RM_HEAP_ID, rminfo);
return recptr;
}
@@ -9122,7 +9122,7 @@ heap_xlog_insert(XLogReaderState *record)
* If we inserted the first and only tuple on the page, re-initialize the
* page from scratch.
*/
- if (XLogRecGetInfo(record) & XLOG_HEAP_INIT_PAGE)
+ if (XLogRecGetRmInfo(record) & XLOG_HEAP_INIT_PAGE)
{
buffer = XLogInitBufferForRedo(record, 0);
page = BufferGetPage(buffer);
@@ -9216,7 +9216,7 @@ heap_xlog_multi_insert(XLogReaderState *record)
uint32 newlen;
Size freespace = 0;
int i;
- bool isinit = (XLogRecGetInfo(record) & XLOG_HEAP_INIT_PAGE) != 0;
+ bool isinit = (XLogRecGetRmInfo(record) & XLOG_HEAP_INIT_PAGE) != 0;
XLogRedoAction action;
/*
@@ -9464,7 +9464,7 @@ heap_xlog_update(XLogReaderState *record, bool hot_update)
nbuffer = obuffer;
newaction = oldaction;
}
- else if (XLogRecGetInfo(record) & XLOG_HEAP_INIT_PAGE)
+ else if (XLogRecGetRmInfo(record) & XLOG_HEAP_INIT_PAGE)
{
nbuffer = XLogInitBufferForRedo(record, 0);
page = (Page) BufferGetPage(nbuffer);
@@ -9828,14 +9828,14 @@ heap_xlog_inplace(XLogReaderState *record)
void
heap_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
/*
* These operations don't overwrite MVCC data so no conflict processing is
* required. The ones in heap2 rmgr do.
*/
- switch (info & XLOG_HEAP_OPMASK)
+ switch (rminfo & XLOG_HEAP_OPMASK)
{
case XLOG_HEAP_INSERT:
heap_xlog_insert(record);
@@ -9867,16 +9867,16 @@ heap_redo(XLogReaderState *record)
heap_xlog_inplace(record);
break;
default:
- elog(PANIC, "heap_redo: unknown op code %u", info);
+ elog(PANIC, "heap_redo: unknown op code %u", rminfo);
}
}
void
heap2_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- switch (info & XLOG_HEAP_OPMASK)
+ switch (rminfo & XLOG_HEAP_OPMASK)
{
case XLOG_HEAP2_PRUNE:
heap_xlog_prune(record);
@@ -9907,7 +9907,7 @@ heap2_redo(XLogReaderState *record)
heap_xlog_logical_rewrite(record);
break;
default:
- elog(PANIC, "heap2_redo: unknown op code %u", info);
+ elog(PANIC, "heap2_redo: unknown op code %u", rminfo);
}
}
diff --git a/src/backend/access/nbtree/nbtinsert.c b/src/backend/access/nbtree/nbtinsert.c
index f6f4af8bfe..90948bf819 100644
--- a/src/backend/access/nbtree/nbtinsert.c
+++ b/src/backend/access/nbtree/nbtinsert.c
@@ -1308,7 +1308,7 @@ _bt_insertonpg(Relation rel,
{
xl_btree_insert xlrec;
xl_btree_metadata xlmeta;
- uint8 xlinfo;
+ uint8 xlrminfo;
XLogRecPtr recptr;
uint16 upostingoff;
@@ -1320,7 +1320,7 @@ _bt_insertonpg(Relation rel,
if (isleaf && postingoff == 0)
{
/* Simple leaf insert */
- xlinfo = XLOG_BTREE_INSERT_LEAF;
+ xlrminfo = XLOG_BTREE_INSERT_LEAF;
}
else if (postingoff != 0)
{
@@ -1329,18 +1329,18 @@ _bt_insertonpg(Relation rel,
* postingoff field before newitem/orignewitem.
*/
Assert(isleaf);
- xlinfo = XLOG_BTREE_INSERT_POST;
+ xlrminfo = XLOG_BTREE_INSERT_POST;
}
else
{
/* Internal page insert, which finishes a split on cbuf */
- xlinfo = XLOG_BTREE_INSERT_UPPER;
+ xlrminfo = XLOG_BTREE_INSERT_UPPER;
XLogRegisterBuffer(1, cbuf, REGBUF_STANDARD);
if (BufferIsValid(metabuf))
{
/* Actually, it's an internal page insert + meta update */
- xlinfo = XLOG_BTREE_INSERT_META;
+ xlrminfo = XLOG_BTREE_INSERT_META;
Assert(metad->btm_version >= BTREE_NOVAC_VERSION);
xlmeta.version = metad->btm_version;
@@ -1381,7 +1381,7 @@ _bt_insertonpg(Relation rel,
IndexTupleSize(origitup));
}
- recptr = XLogInsert(RM_BTREE_ID, xlinfo);
+ recptr = XLogInsert(RM_BTREE_ID, xlrminfo);
if (BufferIsValid(metabuf))
PageSetLSN(metapg, recptr);
@@ -1962,7 +1962,7 @@ _bt_split(Relation rel, BTScanInsert itup_key, Buffer buf, Buffer cbuf,
if (RelationNeedsWAL(rel))
{
xl_btree_split xlrec;
- uint8 xlinfo;
+ uint8 xlrminfo;
XLogRecPtr recptr;
xlrec.level = ropaque->btpo_level;
@@ -2045,8 +2045,8 @@ _bt_split(Relation rel, BTScanInsert itup_key, Buffer buf, Buffer cbuf,
(char *) rightpage + ((PageHeader) rightpage)->pd_upper,
((PageHeader) rightpage)->pd_special - ((PageHeader) rightpage)->pd_upper);
- xlinfo = newitemonleft ? XLOG_BTREE_SPLIT_L : XLOG_BTREE_SPLIT_R;
- recptr = XLogInsert(RM_BTREE_ID, xlinfo);
+ xlrminfo = newitemonleft ? XLOG_BTREE_SPLIT_L : XLOG_BTREE_SPLIT_R;
+ recptr = XLogInsert(RM_BTREE_ID, xlrminfo);
PageSetLSN(origpage, recptr);
PageSetLSN(rightpage, recptr);
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 8b96708b3e..e802b71704 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2634,7 +2634,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
{
xl_btree_unlink_page xlrec;
xl_btree_metadata xlmeta;
- uint8 xlinfo;
+ uint8 xlrminfo;
XLogRecPtr recptr;
XLogBeginInsert();
@@ -2673,12 +2673,12 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
xlmeta.allequalimage = metad->btm_allequalimage;
XLogRegisterBufData(4, (char *) &xlmeta, sizeof(xl_btree_metadata));
- xlinfo = XLOG_BTREE_UNLINK_PAGE_META;
+ xlrminfo = XLOG_BTREE_UNLINK_PAGE_META;
}
else
- xlinfo = XLOG_BTREE_UNLINK_PAGE;
+ xlrminfo = XLOG_BTREE_UNLINK_PAGE;
- recptr = XLogInsert(RM_BTREE_ID, xlinfo);
+ recptr = XLogInsert(RM_BTREE_ID, xlrminfo);
if (BufferIsValid(metabuf))
{
diff --git a/src/backend/access/nbtree/nbtxlog.c b/src/backend/access/nbtree/nbtxlog.c
index ad489e33b3..6c085fd43e 100644
--- a/src/backend/access/nbtree/nbtxlog.c
+++ b/src/backend/access/nbtree/nbtxlog.c
@@ -1012,11 +1012,11 @@ btree_xlog_reuse_page(XLogReaderState *record)
void
btree_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
MemoryContext oldCtx;
oldCtx = MemoryContextSwitchTo(opCtx);
- switch (info)
+ switch (rminfo)
{
case XLOG_BTREE_INSERT_LEAF:
btree_xlog_insert(true, false, false, record);
@@ -1046,11 +1046,11 @@ btree_redo(XLogReaderState *record)
btree_xlog_delete(record);
break;
case XLOG_BTREE_MARK_PAGE_HALFDEAD:
- btree_xlog_mark_page_halfdead(info, record);
+ btree_xlog_mark_page_halfdead(rminfo, record);
break;
case XLOG_BTREE_UNLINK_PAGE:
case XLOG_BTREE_UNLINK_PAGE_META:
- btree_xlog_unlink_page(info, record);
+ btree_xlog_unlink_page(rminfo, record);
break;
case XLOG_BTREE_NEWROOT:
btree_xlog_newroot(record);
@@ -1062,7 +1062,7 @@ btree_redo(XLogReaderState *record)
_bt_restore_meta(record, 0);
break;
default:
- elog(PANIC, "btree_redo: unknown op code %u", info);
+ elog(PANIC, "btree_redo: unknown op code %u", rminfo);
}
MemoryContextSwitchTo(oldCtx);
MemoryContextReset(opCtx);
diff --git a/src/backend/access/rmgrdesc/brindesc.c b/src/backend/access/rmgrdesc/brindesc.c
index f05607e6c3..44395dd9a5 100644
--- a/src/backend/access/rmgrdesc/brindesc.c
+++ b/src/backend/access/rmgrdesc/brindesc.c
@@ -20,17 +20,17 @@ void
brin_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- info &= XLOG_BRIN_OPMASK;
- if (info == XLOG_BRIN_CREATE_INDEX)
+ rminfo &= XLOG_BRIN_OPMASK;
+ if (rminfo == XLOG_BRIN_CREATE_INDEX)
{
xl_brin_createidx *xlrec = (xl_brin_createidx *) rec;
appendStringInfo(buf, "v%d pagesPerRange %u",
xlrec->version, xlrec->pagesPerRange);
}
- else if (info == XLOG_BRIN_INSERT)
+ else if (rminfo == XLOG_BRIN_INSERT)
{
xl_brin_insert *xlrec = (xl_brin_insert *) rec;
@@ -39,7 +39,7 @@ brin_desc(StringInfo buf, XLogReaderState *record)
xlrec->pagesPerRange,
xlrec->offnum);
}
- else if (info == XLOG_BRIN_UPDATE)
+ else if (rminfo == XLOG_BRIN_UPDATE)
{
xl_brin_update *xlrec = (xl_brin_update *) rec;
@@ -49,19 +49,19 @@ brin_desc(StringInfo buf, XLogReaderState *record)
xlrec->oldOffnum,
xlrec->insert.offnum);
}
- else if (info == XLOG_BRIN_SAMEPAGE_UPDATE)
+ else if (rminfo == XLOG_BRIN_SAMEPAGE_UPDATE)
{
xl_brin_samepage_update *xlrec = (xl_brin_samepage_update *) rec;
appendStringInfo(buf, "offnum %u", xlrec->offnum);
}
- else if (info == XLOG_BRIN_REVMAP_EXTEND)
+ else if (rminfo == XLOG_BRIN_REVMAP_EXTEND)
{
xl_brin_revmap_extend *xlrec = (xl_brin_revmap_extend *) rec;
appendStringInfo(buf, "targetBlk %u", xlrec->targetBlk);
}
- else if (info == XLOG_BRIN_DESUMMARIZE)
+ else if (rminfo == XLOG_BRIN_DESUMMARIZE)
{
xl_brin_desummarize *xlrec = (xl_brin_desummarize *) rec;
@@ -71,11 +71,11 @@ brin_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-brin_identify(uint8 info)
+brin_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_BRIN_CREATE_INDEX:
id = "CREATE_INDEX";
diff --git a/src/backend/access/rmgrdesc/clogdesc.c b/src/backend/access/rmgrdesc/clogdesc.c
index 87513732be..1b0cd7ddbf 100644
--- a/src/backend/access/rmgrdesc/clogdesc.c
+++ b/src/backend/access/rmgrdesc/clogdesc.c
@@ -21,16 +21,16 @@ void
clog_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- if (info == CLOG_ZEROPAGE)
+ if (rminfo == CLOG_ZEROPAGE)
{
int pageno;
memcpy(&pageno, rec, sizeof(int));
appendStringInfo(buf, "page %d", pageno);
}
- else if (info == CLOG_TRUNCATE)
+ else if (rminfo == CLOG_TRUNCATE)
{
xl_clog_truncate xlrec;
@@ -41,11 +41,11 @@ clog_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-clog_identify(uint8 info)
+clog_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case CLOG_ZEROPAGE:
id = "ZEROPAGE";
diff --git a/src/backend/access/rmgrdesc/committsdesc.c b/src/backend/access/rmgrdesc/committsdesc.c
index 3a65538bb0..120562222f 100644
--- a/src/backend/access/rmgrdesc/committsdesc.c
+++ b/src/backend/access/rmgrdesc/committsdesc.c
@@ -22,16 +22,16 @@ void
commit_ts_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- if (info == COMMIT_TS_ZEROPAGE)
+ if (rminfo == COMMIT_TS_ZEROPAGE)
{
int pageno;
memcpy(&pageno, rec, sizeof(int));
appendStringInfo(buf, "%d", pageno);
}
- else if (info == COMMIT_TS_TRUNCATE)
+ else if (rminfo == COMMIT_TS_TRUNCATE)
{
xl_commit_ts_truncate *trunc = (xl_commit_ts_truncate *) rec;
@@ -41,9 +41,9 @@ commit_ts_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-commit_ts_identify(uint8 info)
+commit_ts_identify(uint8 rminfo)
{
- switch (info)
+ switch (rminfo)
{
case COMMIT_TS_ZEROPAGE:
return "ZEROPAGE";
diff --git a/src/backend/access/rmgrdesc/dbasedesc.c b/src/backend/access/rmgrdesc/dbasedesc.c
index 523d0b3c1d..f184fbb2ab 100644
--- a/src/backend/access/rmgrdesc/dbasedesc.c
+++ b/src/backend/access/rmgrdesc/dbasedesc.c
@@ -22,9 +22,9 @@ void
dbase_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- if (info == XLOG_DBASE_CREATE_FILE_COPY)
+ if (rminfo == XLOG_DBASE_CREATE_FILE_COPY)
{
xl_dbase_create_file_copy_rec *xlrec =
(xl_dbase_create_file_copy_rec *) rec;
@@ -33,7 +33,7 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
xlrec->src_tablespace_id, xlrec->src_db_id,
xlrec->tablespace_id, xlrec->db_id);
}
- else if (info == XLOG_DBASE_CREATE_WAL_LOG)
+ else if (rminfo == XLOG_DBASE_CREATE_WAL_LOG)
{
xl_dbase_create_wal_log_rec *xlrec =
(xl_dbase_create_wal_log_rec *) rec;
@@ -41,7 +41,7 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "create dir %u/%u",
xlrec->tablespace_id, xlrec->db_id);
}
- else if (info == XLOG_DBASE_DROP)
+ else if (rminfo == XLOG_DBASE_DROP)
{
xl_dbase_drop_rec *xlrec = (xl_dbase_drop_rec *) rec;
int i;
@@ -54,11 +54,11 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-dbase_identify(uint8 info)
+dbase_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_DBASE_CREATE_FILE_COPY:
id = "CREATE_FILE_COPY";
diff --git a/src/backend/access/rmgrdesc/genericdesc.c b/src/backend/access/rmgrdesc/genericdesc.c
index d8509b884b..e1aa356686 100644
--- a/src/backend/access/rmgrdesc/genericdesc.c
+++ b/src/backend/access/rmgrdesc/genericdesc.c
@@ -50,7 +50,7 @@ generic_desc(StringInfo buf, XLogReaderState *record)
* inside generic xlog records.
*/
const char *
-generic_identify(uint8 info)
+generic_identify(uint8 rminfo)
{
return "Generic";
}
diff --git a/src/backend/access/rmgrdesc/gindesc.c b/src/backend/access/rmgrdesc/gindesc.c
index 7d147cea97..76505f5f14 100644
--- a/src/backend/access/rmgrdesc/gindesc.c
+++ b/src/backend/access/rmgrdesc/gindesc.c
@@ -74,9 +74,9 @@ void
gin_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- switch (info)
+ switch (rminfo)
{
case XLOG_GIN_CREATE_PTREE:
/* no further information */
@@ -179,11 +179,11 @@ gin_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-gin_identify(uint8 info)
+gin_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_GIN_CREATE_PTREE:
id = "CREATE_PTREE";
diff --git a/src/backend/access/rmgrdesc/gistdesc.c b/src/backend/access/rmgrdesc/gistdesc.c
index 7dd3c1d500..d6b6404213 100644
--- a/src/backend/access/rmgrdesc/gistdesc.c
+++ b/src/backend/access/rmgrdesc/gistdesc.c
@@ -60,9 +60,9 @@ void
gist_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- switch (info)
+ switch (rminfo)
{
case XLOG_GIST_PAGE_UPDATE:
out_gistxlogPageUpdate(buf, (gistxlogPageUpdate *) rec);
@@ -86,11 +86,11 @@ gist_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-gist_identify(uint8 info)
+gist_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_GIST_PAGE_UPDATE:
id = "PAGE_UPDATE";
diff --git a/src/backend/access/rmgrdesc/hashdesc.c b/src/backend/access/rmgrdesc/hashdesc.c
index ef443bdb16..8ebfc7d16b 100644
--- a/src/backend/access/rmgrdesc/hashdesc.c
+++ b/src/backend/access/rmgrdesc/hashdesc.c
@@ -20,9 +20,9 @@ void
hash_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- switch (info)
+ switch (rminfo)
{
case XLOG_HASH_INIT_META_PAGE:
{
@@ -122,11 +122,11 @@ hash_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-hash_identify(uint8 info)
+hash_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_HASH_INIT_META_PAGE:
id = "INIT_META_PAGE";
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 923d3bc43d..c07f237365 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -35,17 +35,17 @@ void
heap_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- info &= XLOG_HEAP_OPMASK;
- if (info == XLOG_HEAP_INSERT)
+ rminfo &= XLOG_HEAP_OPMASK;
+ if (rminfo == XLOG_HEAP_INSERT)
{
xl_heap_insert *xlrec = (xl_heap_insert *) rec;
appendStringInfo(buf, "off %u flags 0x%02X", xlrec->offnum,
xlrec->flags);
}
- else if (info == XLOG_HEAP_DELETE)
+ else if (rminfo == XLOG_HEAP_DELETE)
{
xl_heap_delete *xlrec = (xl_heap_delete *) rec;
@@ -54,7 +54,7 @@ heap_desc(StringInfo buf, XLogReaderState *record)
xlrec->flags);
out_infobits(buf, xlrec->infobits_set);
}
- else if (info == XLOG_HEAP_UPDATE)
+ else if (rminfo == XLOG_HEAP_UPDATE)
{
xl_heap_update *xlrec = (xl_heap_update *) rec;
@@ -67,7 +67,7 @@ heap_desc(StringInfo buf, XLogReaderState *record)
xlrec->new_offnum,
xlrec->new_xmax);
}
- else if (info == XLOG_HEAP_HOT_UPDATE)
+ else if (rminfo == XLOG_HEAP_HOT_UPDATE)
{
xl_heap_update *xlrec = (xl_heap_update *) rec;
@@ -80,7 +80,7 @@ heap_desc(StringInfo buf, XLogReaderState *record)
xlrec->new_offnum,
xlrec->new_xmax);
}
- else if (info == XLOG_HEAP_TRUNCATE)
+ else if (rminfo == XLOG_HEAP_TRUNCATE)
{
xl_heap_truncate *xlrec = (xl_heap_truncate *) rec;
int i;
@@ -93,13 +93,13 @@ heap_desc(StringInfo buf, XLogReaderState *record)
for (i = 0; i < xlrec->nrelids; i++)
appendStringInfo(buf, " %u", xlrec->relids[i]);
}
- else if (info == XLOG_HEAP_CONFIRM)
+ else if (rminfo == XLOG_HEAP_CONFIRM)
{
xl_heap_confirm *xlrec = (xl_heap_confirm *) rec;
appendStringInfo(buf, "off %u", xlrec->offnum);
}
- else if (info == XLOG_HEAP_LOCK)
+ else if (rminfo == XLOG_HEAP_LOCK)
{
xl_heap_lock *xlrec = (xl_heap_lock *) rec;
@@ -107,7 +107,7 @@ heap_desc(StringInfo buf, XLogReaderState *record)
xlrec->offnum, xlrec->locking_xid, xlrec->flags);
out_infobits(buf, xlrec->infobits_set);
}
- else if (info == XLOG_HEAP_INPLACE)
+ else if (rminfo == XLOG_HEAP_INPLACE)
{
xl_heap_inplace *xlrec = (xl_heap_inplace *) rec;
@@ -118,10 +118,10 @@ void
heap2_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- info &= XLOG_HEAP_OPMASK;
- if (info == XLOG_HEAP2_PRUNE)
+ rminfo &= XLOG_HEAP_OPMASK;
+ if (rminfo == XLOG_HEAP2_PRUNE)
{
xl_heap_prune *xlrec = (xl_heap_prune *) rec;
@@ -130,34 +130,34 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xlrec->nredirected,
xlrec->ndead);
}
- else if (info == XLOG_HEAP2_VACUUM)
+ else if (rminfo == XLOG_HEAP2_VACUUM)
{
xl_heap_vacuum *xlrec = (xl_heap_vacuum *) rec;
appendStringInfo(buf, "nunused %u", xlrec->nunused);
}
- else if (info == XLOG_HEAP2_FREEZE_PAGE)
+ else if (rminfo == XLOG_HEAP2_FREEZE_PAGE)
{
xl_heap_freeze_page *xlrec = (xl_heap_freeze_page *) rec;
appendStringInfo(buf, "cutoff xid %u ntuples %u",
xlrec->cutoff_xid, xlrec->ntuples);
}
- else if (info == XLOG_HEAP2_VISIBLE)
+ else if (rminfo == XLOG_HEAP2_VISIBLE)
{
xl_heap_visible *xlrec = (xl_heap_visible *) rec;
appendStringInfo(buf, "cutoff xid %u flags 0x%02X",
xlrec->cutoff_xid, xlrec->flags);
}
- else if (info == XLOG_HEAP2_MULTI_INSERT)
+ else if (rminfo == XLOG_HEAP2_MULTI_INSERT)
{
xl_heap_multi_insert *xlrec = (xl_heap_multi_insert *) rec;
appendStringInfo(buf, "%d tuples flags 0x%02X", xlrec->ntuples,
xlrec->flags);
}
- else if (info == XLOG_HEAP2_LOCK_UPDATED)
+ else if (rminfo == XLOG_HEAP2_LOCK_UPDATED)
{
xl_heap_lock_updated *xlrec = (xl_heap_lock_updated *) rec;
@@ -165,7 +165,7 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xlrec->offnum, xlrec->xmax, xlrec->flags);
out_infobits(buf, xlrec->infobits_set);
}
- else if (info == XLOG_HEAP2_NEW_CID)
+ else if (rminfo == XLOG_HEAP2_NEW_CID)
{
xl_heap_new_cid *xlrec = (xl_heap_new_cid *) rec;
@@ -181,11 +181,11 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-heap_identify(uint8 info)
+heap_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_HEAP_INSERT:
id = "INSERT";
@@ -226,11 +226,11 @@ heap_identify(uint8 info)
}
const char *
-heap2_identify(uint8 info)
+heap2_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_HEAP2_PRUNE:
id = "PRUNE";
diff --git a/src/backend/access/rmgrdesc/logicalmsgdesc.c b/src/backend/access/rmgrdesc/logicalmsgdesc.c
index 08e03aa30d..03e653c0e8 100644
--- a/src/backend/access/rmgrdesc/logicalmsgdesc.c
+++ b/src/backend/access/rmgrdesc/logicalmsgdesc.c
@@ -19,9 +19,9 @@ void
logicalmsg_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- if (info == XLOG_LOGICAL_MESSAGE)
+ if (rminfo == XLOG_LOGICAL_MESSAGE)
{
xl_logical_message *xlrec = (xl_logical_message *) rec;
char *prefix = xlrec->message;
@@ -43,9 +43,9 @@ logicalmsg_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-logicalmsg_identify(uint8 info)
+logicalmsg_identify(uint8 rminfo)
{
- if ((info & ~XLR_INFO_MASK) == XLOG_LOGICAL_MESSAGE)
+ if ((rminfo) == XLOG_LOGICAL_MESSAGE)
return "MESSAGE";
return NULL;
diff --git a/src/backend/access/rmgrdesc/mxactdesc.c b/src/backend/access/rmgrdesc/mxactdesc.c
index 7076be2b3f..6e5a411d16 100644
--- a/src/backend/access/rmgrdesc/mxactdesc.c
+++ b/src/backend/access/rmgrdesc/mxactdesc.c
@@ -50,17 +50,17 @@ void
multixact_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- if (info == XLOG_MULTIXACT_ZERO_OFF_PAGE ||
- info == XLOG_MULTIXACT_ZERO_MEM_PAGE)
+ if (rminfo == XLOG_MULTIXACT_ZERO_OFF_PAGE ||
+ rminfo == XLOG_MULTIXACT_ZERO_MEM_PAGE)
{
int pageno;
memcpy(&pageno, rec, sizeof(int));
appendStringInfo(buf, "%d", pageno);
}
- else if (info == XLOG_MULTIXACT_CREATE_ID)
+ else if (rminfo == XLOG_MULTIXACT_CREATE_ID)
{
xl_multixact_create *xlrec = (xl_multixact_create *) rec;
int i;
@@ -70,7 +70,7 @@ multixact_desc(StringInfo buf, XLogReaderState *record)
for (i = 0; i < xlrec->nmembers; i++)
out_member(buf, &xlrec->members[i]);
}
- else if (info == XLOG_MULTIXACT_TRUNCATE_ID)
+ else if (rminfo == XLOG_MULTIXACT_TRUNCATE_ID)
{
xl_multixact_truncate *xlrec = (xl_multixact_truncate *) rec;
@@ -81,11 +81,11 @@ multixact_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-multixact_identify(uint8 info)
+multixact_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_MULTIXACT_ZERO_OFF_PAGE:
id = "ZERO_OFF_PAGE";
diff --git a/src/backend/access/rmgrdesc/nbtdesc.c b/src/backend/access/rmgrdesc/nbtdesc.c
index 4843cd530d..c5e92e9e6b 100644
--- a/src/backend/access/rmgrdesc/nbtdesc.c
+++ b/src/backend/access/rmgrdesc/nbtdesc.c
@@ -20,9 +20,9 @@ void
btree_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- switch (info)
+ switch (rminfo)
{
case XLOG_BTREE_INSERT_LEAF:
case XLOG_BTREE_INSERT_UPPER:
@@ -121,11 +121,11 @@ btree_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-btree_identify(uint8 info)
+btree_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_BTREE_INSERT_LEAF:
id = "INSERT_LEAF";
diff --git a/src/backend/access/rmgrdesc/relmapdesc.c b/src/backend/access/rmgrdesc/relmapdesc.c
index 43d63eb9a4..0f2d0d80fb 100644
--- a/src/backend/access/rmgrdesc/relmapdesc.c
+++ b/src/backend/access/rmgrdesc/relmapdesc.c
@@ -20,9 +20,9 @@ void
relmap_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- if (info == XLOG_RELMAP_UPDATE)
+ if (rminfo == XLOG_RELMAP_UPDATE)
{
xl_relmap_update *xlrec = (xl_relmap_update *) rec;
@@ -32,11 +32,11 @@ relmap_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-relmap_identify(uint8 info)
+relmap_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_RELMAP_UPDATE:
id = "UPDATE";
diff --git a/src/backend/access/rmgrdesc/replorigindesc.c b/src/backend/access/rmgrdesc/replorigindesc.c
index e3213b1016..3bf3fde05f 100644
--- a/src/backend/access/rmgrdesc/replorigindesc.c
+++ b/src/backend/access/rmgrdesc/replorigindesc.c
@@ -19,9 +19,9 @@ void
replorigin_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- switch (info)
+ switch (rminfo)
{
case XLOG_REPLORIGIN_SET:
{
@@ -48,9 +48,9 @@ replorigin_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-replorigin_identify(uint8 info)
+replorigin_identify(uint8 rminfo)
{
- switch (info)
+ switch (rminfo)
{
case XLOG_REPLORIGIN_SET:
return "SET";
diff --git a/src/backend/access/rmgrdesc/seqdesc.c b/src/backend/access/rmgrdesc/seqdesc.c
index b3845f93bf..2a4b8367ec 100644
--- a/src/backend/access/rmgrdesc/seqdesc.c
+++ b/src/backend/access/rmgrdesc/seqdesc.c
@@ -21,21 +21,21 @@ void
seq_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
xl_seq_rec *xlrec = (xl_seq_rec *) rec;
- if (info == XLOG_SEQ_LOG)
+ if (rminfo == XLOG_SEQ_LOG)
appendStringInfo(buf, "rel %u/%u/%u",
xlrec->locator.spcOid, xlrec->locator.dbOid,
xlrec->locator.relNumber);
}
const char *
-seq_identify(uint8 info)
+seq_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_SEQ_LOG:
id = "LOG";
diff --git a/src/backend/access/rmgrdesc/smgrdesc.c b/src/backend/access/rmgrdesc/smgrdesc.c
index e0ee8a078a..944f3650cf 100644
--- a/src/backend/access/rmgrdesc/smgrdesc.c
+++ b/src/backend/access/rmgrdesc/smgrdesc.c
@@ -21,9 +21,9 @@ void
smgr_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- if (info == XLOG_SMGR_CREATE)
+ if (rminfo == XLOG_SMGR_CREATE)
{
xl_smgr_create *xlrec = (xl_smgr_create *) rec;
char *path = relpathperm(xlrec->rlocator, xlrec->forkNum);
@@ -31,7 +31,7 @@ smgr_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, path);
pfree(path);
}
- else if (info == XLOG_SMGR_TRUNCATE)
+ else if (rminfo == XLOG_SMGR_TRUNCATE)
{
xl_smgr_truncate *xlrec = (xl_smgr_truncate *) rec;
char *path = relpathperm(xlrec->rlocator, MAIN_FORKNUM);
@@ -43,11 +43,11 @@ smgr_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-smgr_identify(uint8 info)
+smgr_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_SMGR_CREATE:
id = "CREATE";
diff --git a/src/backend/access/rmgrdesc/spgdesc.c b/src/backend/access/rmgrdesc/spgdesc.c
index d5d921a42a..2afb827de4 100644
--- a/src/backend/access/rmgrdesc/spgdesc.c
+++ b/src/backend/access/rmgrdesc/spgdesc.c
@@ -20,9 +20,9 @@ void
spg_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- switch (info)
+ switch (rminfo)
{
case XLOG_SPGIST_ADD_LEAF:
{
@@ -128,11 +128,11 @@ spg_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-spg_identify(uint8 info)
+spg_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_SPGIST_ADD_LEAF:
id = "ADD_LEAF";
diff --git a/src/backend/access/rmgrdesc/standbydesc.c b/src/backend/access/rmgrdesc/standbydesc.c
index 2dba39e349..5a2e6a323c 100644
--- a/src/backend/access/rmgrdesc/standbydesc.c
+++ b/src/backend/access/rmgrdesc/standbydesc.c
@@ -40,9 +40,9 @@ void
standby_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- if (info == XLOG_STANDBY_LOCK)
+ if (rminfo == XLOG_STANDBY_LOCK)
{
xl_standby_locks *xlrec = (xl_standby_locks *) rec;
int i;
@@ -52,13 +52,13 @@ standby_desc(StringInfo buf, XLogReaderState *record)
xlrec->locks[i].xid, xlrec->locks[i].dbOid,
xlrec->locks[i].relOid);
}
- else if (info == XLOG_RUNNING_XACTS)
+ else if (rminfo == XLOG_RUNNING_XACTS)
{
xl_running_xacts *xlrec = (xl_running_xacts *) rec;
standby_desc_running_xacts(buf, xlrec);
}
- else if (info == XLOG_INVALIDATIONS)
+ else if (rminfo == XLOG_INVALIDATIONS)
{
xl_invalidations *xlrec = (xl_invalidations *) rec;
@@ -69,11 +69,11 @@ standby_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-standby_identify(uint8 info)
+standby_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_STANDBY_LOCK:
id = "LOCK";
diff --git a/src/backend/access/rmgrdesc/tblspcdesc.c b/src/backend/access/rmgrdesc/tblspcdesc.c
index ed94b6e2dd..b8c050fcba 100644
--- a/src/backend/access/rmgrdesc/tblspcdesc.c
+++ b/src/backend/access/rmgrdesc/tblspcdesc.c
@@ -21,15 +21,15 @@ void
tblspc_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- if (info == XLOG_TBLSPC_CREATE)
+ if (rminfo == XLOG_TBLSPC_CREATE)
{
xl_tblspc_create_rec *xlrec = (xl_tblspc_create_rec *) rec;
appendStringInfo(buf, "%u \"%s\"", xlrec->ts_id, xlrec->ts_path);
}
- else if (info == XLOG_TBLSPC_DROP)
+ else if (rminfo == XLOG_TBLSPC_DROP)
{
xl_tblspc_drop_rec *xlrec = (xl_tblspc_drop_rec *) rec;
@@ -38,11 +38,11 @@ tblspc_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-tblspc_identify(uint8 info)
+tblspc_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_TBLSPC_CREATE:
id = "CREATE";
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 39752cf349..93a653c75a 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -32,7 +32,7 @@
*/
void
-ParseCommitRecord(uint8 info, xl_xact_commit *xlrec, xl_xact_parsed_commit *parsed)
+ParseCommitRecord(uint8 rminfo, xl_xact_commit *xlrec, xl_xact_parsed_commit *parsed)
{
char *data = ((char *) xlrec) + MinSizeOfXactCommit;
@@ -43,7 +43,7 @@ ParseCommitRecord(uint8 info, xl_xact_commit *xlrec, xl_xact_parsed_commit *pars
parsed->xact_time = xlrec->xact_time;
- if (info & XLOG_XACT_HAS_INFO)
+ if (rminfo & XLOG_XACT_HAS_INFO)
{
xl_xact_xinfo *xl_xinfo = (xl_xact_xinfo *) data;
@@ -138,7 +138,7 @@ ParseCommitRecord(uint8 info, xl_xact_commit *xlrec, xl_xact_parsed_commit *pars
}
void
-ParseAbortRecord(uint8 info, xl_xact_abort *xlrec, xl_xact_parsed_abort *parsed)
+ParseAbortRecord(uint8 rminfo, xl_xact_abort *xlrec, xl_xact_parsed_abort *parsed)
{
char *data = ((char *) xlrec) + MinSizeOfXactAbort;
@@ -149,7 +149,7 @@ ParseAbortRecord(uint8 info, xl_xact_abort *xlrec, xl_xact_parsed_abort *parsed)
parsed->xact_time = xlrec->xact_time;
- if (info & XLOG_XACT_HAS_INFO)
+ if (rminfo & XLOG_XACT_HAS_INFO)
{
xl_xact_xinfo *xl_xinfo = (xl_xact_xinfo *) data;
@@ -236,7 +236,7 @@ ParseAbortRecord(uint8 info, xl_xact_abort *xlrec, xl_xact_parsed_abort *parsed)
* ParsePrepareRecord
*/
void
-ParsePrepareRecord(uint8 info, xl_xact_prepare *xlrec, xl_xact_parsed_prepare *parsed)
+ParsePrepareRecord(uint8 rminfo, xl_xact_prepare *xlrec, xl_xact_parsed_prepare *parsed)
{
char *bufptr;
@@ -328,11 +328,11 @@ xact_desc_stats(StringInfo buf, const char *label,
}
static void
-xact_desc_commit(StringInfo buf, uint8 info, xl_xact_commit *xlrec, RepOriginId origin_id)
+xact_desc_commit(StringInfo buf, uint8 rminfo, xl_xact_commit *xlrec, RepOriginId origin_id)
{
xl_xact_parsed_commit parsed;
- ParseCommitRecord(info, xlrec, &parsed);
+ ParseCommitRecord(rminfo, xlrec, &parsed);
/* If this is a prepared xact, show the xid of the original xact */
if (TransactionIdIsValid(parsed.twophase_xid))
@@ -364,11 +364,11 @@ xact_desc_commit(StringInfo buf, uint8 info, xl_xact_commit *xlrec, RepOriginId
}
static void
-xact_desc_abort(StringInfo buf, uint8 info, xl_xact_abort *xlrec, RepOriginId origin_id)
+xact_desc_abort(StringInfo buf, uint8 rminfo, xl_xact_abort *xlrec, RepOriginId origin_id)
{
xl_xact_parsed_abort parsed;
- ParseAbortRecord(info, xlrec, &parsed);
+ ParseAbortRecord(rminfo, xlrec, &parsed);
/* If this is a prepared xact, show the xid of the original xact */
if (TransactionIdIsValid(parsed.twophase_xid))
@@ -391,11 +391,11 @@ xact_desc_abort(StringInfo buf, uint8 info, xl_xact_abort *xlrec, RepOriginId or
}
static void
-xact_desc_prepare(StringInfo buf, uint8 info, xl_xact_prepare *xlrec, RepOriginId origin_id)
+xact_desc_prepare(StringInfo buf, uint8 rminfo, xl_xact_prepare *xlrec, RepOriginId origin_id)
{
xl_xact_parsed_prepare parsed;
- ParsePrepareRecord(info, xlrec, &parsed);
+ ParsePrepareRecord(rminfo, xlrec, &parsed);
appendStringInfo(buf, "gid %s: ", parsed.twophase_gid);
appendStringInfoString(buf, timestamptz_to_str(parsed.xact_time));
@@ -436,27 +436,27 @@ void
xact_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & XLOG_XACT_OPMASK;
+ uint8 info = XLogRecGetRmInfo(record) & XLOG_XACT_OPMASK;
if (info == XLOG_XACT_COMMIT || info == XLOG_XACT_COMMIT_PREPARED)
{
xl_xact_commit *xlrec = (xl_xact_commit *) rec;
- xact_desc_commit(buf, XLogRecGetInfo(record), xlrec,
+ xact_desc_commit(buf, XLogRecGetRmInfo(record), xlrec,
XLogRecGetOrigin(record));
}
else if (info == XLOG_XACT_ABORT || info == XLOG_XACT_ABORT_PREPARED)
{
xl_xact_abort *xlrec = (xl_xact_abort *) rec;
- xact_desc_abort(buf, XLogRecGetInfo(record), xlrec,
+ xact_desc_abort(buf, XLogRecGetRmInfo(record), xlrec,
XLogRecGetOrigin(record));
}
else if (info == XLOG_XACT_PREPARE)
{
xl_xact_prepare *xlrec = (xl_xact_prepare *) rec;
- xact_desc_prepare(buf, XLogRecGetInfo(record), xlrec,
+ xact_desc_prepare(buf, XLogRecGetRmInfo(record), xlrec,
XLogRecGetOrigin(record));
}
else if (info == XLOG_XACT_ASSIGNMENT)
@@ -481,11 +481,11 @@ xact_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-xact_identify(uint8 info)
+xact_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & XLOG_XACT_OPMASK)
+ switch (rminfo & XLOG_XACT_OPMASK)
{
case XLOG_XACT_COMMIT:
id = "COMMIT";
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 3fd7185f21..65ac642908 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -37,10 +37,10 @@ void
xlog_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- if (info == XLOG_CHECKPOINT_SHUTDOWN ||
- info == XLOG_CHECKPOINT_ONLINE)
+ if (rminfo == XLOG_CHECKPOINT_SHUTDOWN ||
+ rminfo == XLOG_CHECKPOINT_ONLINE)
{
CheckPoint *checkpoint = (CheckPoint *) rec;
@@ -65,33 +65,33 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
checkpoint->oldestCommitTsXid,
checkpoint->newestCommitTsXid,
checkpoint->oldestActiveXid,
- (info == XLOG_CHECKPOINT_SHUTDOWN) ? "shutdown" : "online");
+ (rminfo == XLOG_CHECKPOINT_SHUTDOWN) ? "shutdown" : "online");
}
- else if (info == XLOG_NEXTOID)
+ else if (rminfo == XLOG_NEXTOID)
{
Oid nextOid;
memcpy(&nextOid, rec, sizeof(Oid));
appendStringInfo(buf, "%u", nextOid);
}
- else if (info == XLOG_RESTORE_POINT)
+ else if (rminfo == XLOG_RESTORE_POINT)
{
xl_restore_point *xlrec = (xl_restore_point *) rec;
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (rminfo == XLOG_FPI || rminfo == XLOG_FPI_FOR_HINT)
{
/* no further information to print */
}
- else if (info == XLOG_BACKUP_END)
+ else if (rminfo == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
memcpy(&startpoint, rec, sizeof(XLogRecPtr));
appendStringInfo(buf, "%X/%X", LSN_FORMAT_ARGS(startpoint));
}
- else if (info == XLOG_PARAMETER_CHANGE)
+ else if (rminfo == XLOG_PARAMETER_CHANGE)
{
xl_parameter_change xlrec;
const char *wal_level_str;
@@ -123,14 +123,14 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
xlrec.wal_log_hints ? "on" : "off",
xlrec.track_commit_timestamp ? "on" : "off");
}
- else if (info == XLOG_FPW_CHANGE)
+ else if (rminfo == XLOG_FPW_CHANGE)
{
bool fpw;
memcpy(&fpw, rec, sizeof(bool));
appendStringInfoString(buf, fpw ? "true" : "false");
}
- else if (info == XLOG_END_OF_RECOVERY)
+ else if (rminfo == XLOG_END_OF_RECOVERY)
{
xl_end_of_recovery xlrec;
@@ -139,7 +139,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
xlrec.ThisTimeLineID, xlrec.PrevTimeLineID,
timestamptz_to_str(xlrec.end_time));
}
- else if (info == XLOG_OVERWRITE_CONTRECORD)
+ else if (rminfo == XLOG_OVERWRITE_CONTRECORD)
{
xl_overwrite_contrecord xlrec;
@@ -151,11 +151,11 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
}
const char *
-xlog_identify(uint8 info)
+xlog_identify(uint8 rminfo)
{
const char *id = NULL;
- switch (info & ~XLR_INFO_MASK)
+ switch (rminfo)
{
case XLOG_CHECKPOINT_SHUTDOWN:
id = "CHECKPOINT_SHUTDOWN";
diff --git a/src/backend/access/spgist/spgxlog.c b/src/backend/access/spgist/spgxlog.c
index 4c9f4020ff..cb3b864227 100644
--- a/src/backend/access/spgist/spgxlog.c
+++ b/src/backend/access/spgist/spgxlog.c
@@ -938,11 +938,11 @@ spgRedoVacuumRedirect(XLogReaderState *record)
void
spg_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
MemoryContext oldCxt;
oldCxt = MemoryContextSwitchTo(opCtx);
- switch (info)
+ switch (rminfo)
{
case XLOG_SPGIST_ADD_LEAF:
spgRedoAddLeaf(record);
@@ -969,7 +969,7 @@ spg_redo(XLogReaderState *record)
spgRedoVacuumRedirect(record);
break;
default:
- elog(PANIC, "spg_redo: unknown op code %u", info);
+ elog(PANIC, "spg_redo: unknown op code %u", rminfo);
}
MemoryContextSwitchTo(oldCxt);
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index a7dfcfb4da..5badd53418 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -986,12 +986,12 @@ WriteTruncateXlogRec(int pageno, TransactionId oldestXact, Oid oldestXactDb)
void
clog_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
/* Backup blocks are not used in clog records */
Assert(!XLogRecHasAnyBlockRefs(record));
- if (info == CLOG_ZEROPAGE)
+ if (rminfo == CLOG_ZEROPAGE)
{
int pageno;
int slotno;
@@ -1006,7 +1006,7 @@ clog_redo(XLogReaderState *record)
LWLockRelease(XactSLRULock);
}
- else if (info == CLOG_TRUNCATE)
+ else if (rminfo == CLOG_TRUNCATE)
{
xl_clog_truncate xlrec;
@@ -1017,7 +1017,7 @@ clog_redo(XLogReaderState *record)
SimpleLruTruncate(XactCtl, xlrec.pageno);
}
else
- elog(PANIC, "clog_redo: unknown op code %u", info);
+ elog(PANIC, "clog_redo: unknown op code %u", rminfo);
}
/*
diff --git a/src/backend/access/transam/commit_ts.c b/src/backend/access/transam/commit_ts.c
index 9aa4675cb7..43129c7a6f 100644
--- a/src/backend/access/transam/commit_ts.c
+++ b/src/backend/access/transam/commit_ts.c
@@ -984,12 +984,12 @@ WriteTruncateXlogRec(int pageno, TransactionId oldestXid)
void
commit_ts_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
/* Backup blocks are not used in commit_ts records */
Assert(!XLogRecHasAnyBlockRefs(record));
- if (info == COMMIT_TS_ZEROPAGE)
+ if (rminfo == COMMIT_TS_ZEROPAGE)
{
int pageno;
int slotno;
@@ -1004,7 +1004,7 @@ commit_ts_redo(XLogReaderState *record)
LWLockRelease(CommitTsSLRULock);
}
- else if (info == COMMIT_TS_TRUNCATE)
+ else if (rminfo == COMMIT_TS_TRUNCATE)
{
xl_commit_ts_truncate *trunc = (xl_commit_ts_truncate *) XLogRecGetData(record);
@@ -1019,7 +1019,7 @@ commit_ts_redo(XLogReaderState *record)
SimpleLruTruncate(CommitTsCtl, trunc->pageno);
}
else
- elog(PANIC, "commit_ts_redo: unknown op code %u", info);
+ elog(PANIC, "commit_ts_redo: unknown op code %u", rminfo);
}
/*
diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.c
index a7383f553b..ca6e238542 100644
--- a/src/backend/access/transam/multixact.c
+++ b/src/backend/access/transam/multixact.c
@@ -365,7 +365,7 @@ static bool MultiXactOffsetWouldWrap(MultiXactOffset boundary,
MultiXactOffset start, uint32 distance);
static bool SetOffsetVacuumLimit(bool is_startup);
static bool find_multixact_start(MultiXactId multi, MultiXactOffset *result);
-static void WriteMZeroPageXlogRec(int pageno, uint8 info);
+static void WriteMZeroPageXlogRec(int pageno, uint8 rminfo);
static void WriteMTruncateXlogRec(Oid oldestMultiDB,
MultiXactId startTruncOff,
MultiXactId endTruncOff,
@@ -3193,11 +3193,11 @@ MultiXactOffsetPrecedes(MultiXactOffset offset1, MultiXactOffset offset2)
* OFFSETs page (info shows which)
*/
static void
-WriteMZeroPageXlogRec(int pageno, uint8 info)
+WriteMZeroPageXlogRec(int pageno, uint8 rminfo)
{
XLogBeginInsert();
XLogRegisterData((char *) (&pageno), sizeof(int));
- (void) XLogInsert(RM_MULTIXACT_ID, info);
+ (void) XLogInsert(RM_MULTIXACT_ID, rminfo);
}
/*
@@ -3234,12 +3234,12 @@ WriteMTruncateXlogRec(Oid oldestMultiDB,
void
multixact_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
/* Backup blocks are not used in multixact records */
Assert(!XLogRecHasAnyBlockRefs(record));
- if (info == XLOG_MULTIXACT_ZERO_OFF_PAGE)
+ if (rminfo == XLOG_MULTIXACT_ZERO_OFF_PAGE)
{
int pageno;
int slotno;
@@ -3254,7 +3254,7 @@ multixact_redo(XLogReaderState *record)
LWLockRelease(MultiXactOffsetSLRULock);
}
- else if (info == XLOG_MULTIXACT_ZERO_MEM_PAGE)
+ else if (rminfo == XLOG_MULTIXACT_ZERO_MEM_PAGE)
{
int pageno;
int slotno;
@@ -3269,7 +3269,7 @@ multixact_redo(XLogReaderState *record)
LWLockRelease(MultiXactMemberSLRULock);
}
- else if (info == XLOG_MULTIXACT_CREATE_ID)
+ else if (rminfo == XLOG_MULTIXACT_CREATE_ID)
{
xl_multixact_create *xlrec =
(xl_multixact_create *) XLogRecGetData(record);
@@ -3298,7 +3298,7 @@ multixact_redo(XLogReaderState *record)
AdvanceNextFullTransactionIdPastXid(max_xid);
}
- else if (info == XLOG_MULTIXACT_TRUNCATE_ID)
+ else if (rminfo == XLOG_MULTIXACT_TRUNCATE_ID)
{
xl_multixact_truncate xlrec;
int pageno;
@@ -3339,7 +3339,7 @@ multixact_redo(XLogReaderState *record)
LWLockRelease(MultiXactTruncationLock);
}
else
- elog(PANIC, "multixact_redo: unknown op code %u", info);
+ elog(PANIC, "multixact_redo: unknown op code %u", rminfo);
}
Datum
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 803d169f57..98c1ef7ec5 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -1429,7 +1429,7 @@ XlogReadTwoPhaseData(XLogRecPtr lsn, char **buf, int *len)
}
if (XLogRecGetRmid(xlogreader) != RM_XACT_ID ||
- (XLogRecGetInfo(xlogreader) & XLOG_XACT_OPMASK) != XLOG_XACT_PREPARE)
+ (XLogRecGetRmInfo(xlogreader) & XLOG_XACT_OPMASK) != XLOG_XACT_PREPARE)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("expected two-phase state data is not present in WAL at %X/%X",
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index c1ffbd89b8..8ce44abfcf 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -5624,7 +5624,8 @@ XactLogCommitRecord(TimestampTz commit_time,
xl_xact_invals xl_invals;
xl_xact_twophase xl_twophase;
xl_xact_origin xl_origin;
- uint8 info;
+ uint8 rminfo;
+ uint8 info = 0;
Assert(CritSectionCount > 0);
@@ -5632,9 +5633,9 @@ XactLogCommitRecord(TimestampTz commit_time,
/* decide between a plain and 2pc commit */
if (!TransactionIdIsValid(twophase_xid))
- info = XLOG_XACT_COMMIT;
+ rminfo = XLOG_XACT_COMMIT;
else
- info = XLOG_XACT_COMMIT_PREPARED;
+ rminfo = XLOG_XACT_COMMIT_PREPARED;
/* First figure out and collect all the information needed */
@@ -5675,7 +5676,7 @@ XactLogCommitRecord(TimestampTz commit_time,
{
xl_xinfo.xinfo |= XACT_XINFO_HAS_RELFILELOCATORS;
xl_relfilelocators.nrels = nrels;
- info |= XLR_SPECIAL_REL_UPDATE;
+ info = XLR_SPECIAL_REL_UPDATE;
}
if (ndroppedstats > 0)
@@ -5710,7 +5711,7 @@ XactLogCommitRecord(TimestampTz commit_time,
}
if (xl_xinfo.xinfo != 0)
- info |= XLOG_XACT_HAS_INFO;
+ rminfo |= XLOG_XACT_HAS_INFO;
/* Then include all the collected data into the commit record. */
@@ -5768,7 +5769,7 @@ XactLogCommitRecord(TimestampTz commit_time,
/* we allow filtering by xacts */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- return XLogInsert(RM_XACT_ID, info);
+ return XLogInsertExtended(RM_XACT_ID, info, rminfo);
}
/*
@@ -5794,7 +5795,8 @@ XactLogAbortRecord(TimestampTz abort_time,
xl_xact_dbinfo xl_dbinfo;
xl_xact_origin xl_origin;
- uint8 info;
+ uint8 rminfo;
+ uint8 info = 0;
Assert(CritSectionCount > 0);
@@ -5802,9 +5804,9 @@ XactLogAbortRecord(TimestampTz abort_time,
/* decide between a plain and 2pc abort */
if (!TransactionIdIsValid(twophase_xid))
- info = XLOG_XACT_ABORT;
+ rminfo = XLOG_XACT_ABORT;
else
- info = XLOG_XACT_ABORT_PREPARED;
+ rminfo = XLOG_XACT_ABORT_PREPARED;
/* First figure out and collect all the information needed */
@@ -5824,7 +5826,7 @@ XactLogAbortRecord(TimestampTz abort_time,
{
xl_xinfo.xinfo |= XACT_XINFO_HAS_RELFILELOCATORS;
xl_relfilelocators.nrels = nrels;
- info |= XLR_SPECIAL_REL_UPDATE;
+ info = XLR_SPECIAL_REL_UPDATE;
}
if (ndroppedstats > 0)
@@ -5864,7 +5866,7 @@ XactLogAbortRecord(TimestampTz abort_time,
}
if (xl_xinfo.xinfo != 0)
- info |= XLOG_XACT_HAS_INFO;
+ rminfo |= XLOG_XACT_HAS_INFO;
/* Then include all the collected data into the abort record. */
@@ -5915,7 +5917,7 @@ XactLogAbortRecord(TimestampTz abort_time,
if (TransactionIdIsValid(twophase_xid))
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- return XLogInsert(RM_XACT_ID, info);
+ return XLogInsertExtended(RM_XACT_ID, info, rminfo);
}
/*
@@ -6158,7 +6160,7 @@ xact_redo_abort(xl_xact_parsed_abort *parsed, TransactionId xid,
void
xact_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & XLOG_XACT_OPMASK;
+ uint8 info = XLogRecGetRmInfo(record) & XLOG_XACT_OPMASK;
/* Backup blocks are not used in xact records */
Assert(!XLogRecHasAnyBlockRefs(record));
@@ -6168,7 +6170,7 @@ xact_redo(XLogReaderState *record)
xl_xact_commit *xlrec = (xl_xact_commit *) XLogRecGetData(record);
xl_xact_parsed_commit parsed;
- ParseCommitRecord(XLogRecGetInfo(record), xlrec, &parsed);
+ ParseCommitRecord(XLogRecGetRmInfo(record), xlrec, &parsed);
xact_redo_commit(&parsed, XLogRecGetXid(record),
record->EndRecPtr, XLogRecGetOrigin(record));
}
@@ -6177,7 +6179,7 @@ xact_redo(XLogReaderState *record)
xl_xact_commit *xlrec = (xl_xact_commit *) XLogRecGetData(record);
xl_xact_parsed_commit parsed;
- ParseCommitRecord(XLogRecGetInfo(record), xlrec, &parsed);
+ ParseCommitRecord(XLogRecGetRmInfo(record), xlrec, &parsed);
xact_redo_commit(&parsed, parsed.twophase_xid,
record->EndRecPtr, XLogRecGetOrigin(record));
@@ -6191,7 +6193,7 @@ xact_redo(XLogReaderState *record)
xl_xact_abort *xlrec = (xl_xact_abort *) XLogRecGetData(record);
xl_xact_parsed_abort parsed;
- ParseAbortRecord(XLogRecGetInfo(record), xlrec, &parsed);
+ ParseAbortRecord(XLogRecGetRmInfo(record), xlrec, &parsed);
xact_redo_abort(&parsed, XLogRecGetXid(record),
record->EndRecPtr, XLogRecGetOrigin(record));
}
@@ -6200,7 +6202,7 @@ xact_redo(XLogReaderState *record)
xl_xact_abort *xlrec = (xl_xact_abort *) XLogRecGetData(record);
xl_xact_parsed_abort parsed;
- ParseAbortRecord(XLogRecGetInfo(record), xlrec, &parsed);
+ ParseAbortRecord(XLogRecGetRmInfo(record), xlrec, &parsed);
xact_redo_abort(&parsed, parsed.twophase_xid,
record->EndRecPtr, XLogRecGetOrigin(record));
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 27085b15a8..3058041683 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -742,9 +742,9 @@ XLogInsertRecord(XLogRecData *rdata,
pg_crc32c rdata_crc;
bool inserted;
XLogRecord *rechdr = (XLogRecord *) rdata->data;
- uint8 info = rechdr->xl_info & ~XLR_INFO_MASK;
+ uint8 rminfo = rechdr->xl_rminfo;
bool isLogSwitch = (rechdr->xl_rmid == RM_XLOG_ID &&
- info == XLOG_SWITCH);
+ rminfo == XLOG_SWITCH);
XLogRecPtr StartPos;
XLogRecPtr EndPos;
bool prevDoPageWrites = doPageWrites;
@@ -7728,17 +7728,17 @@ UpdateFullPageWrites(void)
void
xlog_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
XLogRecPtr lsn = record->EndRecPtr;
/*
* In XLOG rmgr, backup blocks are only used by XLOG_FPI and
* XLOG_FPI_FOR_HINT records.
*/
- Assert(info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ Assert(rminfo == XLOG_FPI || rminfo == XLOG_FPI_FOR_HINT ||
!XLogRecHasAnyBlockRefs(record));
- if (info == XLOG_NEXTOID)
+ if (rminfo == XLOG_NEXTOID)
{
Oid nextOid;
@@ -7755,7 +7755,7 @@ xlog_redo(XLogReaderState *record)
ShmemVariableCache->oidCount = 0;
LWLockRelease(OidGenLock);
}
- else if (info == XLOG_CHECKPOINT_SHUTDOWN)
+ else if (rminfo == XLOG_CHECKPOINT_SHUTDOWN)
{
CheckPoint checkPoint;
TimeLineID replayTLI;
@@ -7852,7 +7852,7 @@ xlog_redo(XLogReaderState *record)
RecoveryRestartPoint(&checkPoint, record);
}
- else if (info == XLOG_CHECKPOINT_ONLINE)
+ else if (rminfo == XLOG_CHECKPOINT_ONLINE)
{
CheckPoint checkPoint;
TimeLineID replayTLI;
@@ -7910,11 +7910,11 @@ xlog_redo(XLogReaderState *record)
RecoveryRestartPoint(&checkPoint, record);
}
- else if (info == XLOG_OVERWRITE_CONTRECORD)
+ else if (rminfo == XLOG_OVERWRITE_CONTRECORD)
{
/* nothing to do here, handled in xlogrecovery_redo() */
}
- else if (info == XLOG_END_OF_RECOVERY)
+ else if (rminfo == XLOG_END_OF_RECOVERY)
{
xl_end_of_recovery xlrec;
TimeLineID replayTLI;
@@ -7937,19 +7937,19 @@ xlog_redo(XLogReaderState *record)
(errmsg("unexpected timeline ID %u (should be %u) in end-of-recovery record",
xlrec.ThisTimeLineID, replayTLI)));
}
- else if (info == XLOG_NOOP)
+ else if (rminfo == XLOG_NOOP)
{
/* nothing to do here */
}
- else if (info == XLOG_SWITCH)
+ else if (rminfo == XLOG_SWITCH)
{
/* nothing to do here */
}
- else if (info == XLOG_RESTORE_POINT)
+ else if (rminfo == XLOG_RESTORE_POINT)
{
/* nothing to do here, handled in xlogrecovery.c */
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (rminfo == XLOG_FPI || rminfo == XLOG_FPI_FOR_HINT)
{
/*
* XLOG_FPI records contain nothing else but one or more block
@@ -7973,7 +7973,7 @@ xlog_redo(XLogReaderState *record)
if (!XLogRecHasBlockImage(record, block_id))
{
- if (info == XLOG_FPI)
+ if (rminfo == XLOG_FPI)
elog(ERROR, "XLOG_FPI record did not contain a full-page image");
continue;
}
@@ -7983,11 +7983,11 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
- else if (info == XLOG_BACKUP_END)
+ else if (rminfo == XLOG_BACKUP_END)
{
/* nothing to do here, handled in xlogrecovery_redo() */
}
- else if (info == XLOG_PARAMETER_CHANGE)
+ else if (rminfo == XLOG_PARAMETER_CHANGE)
{
xl_parameter_change xlrec;
@@ -8035,7 +8035,7 @@ xlog_redo(XLogReaderState *record)
/* Check to see if any parameter change gives a problem on recovery */
CheckRequiredParameterValues();
}
- else if (info == XLOG_FPW_CHANGE)
+ else if (rminfo == XLOG_FPW_CHANGE)
{
bool fpw;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 5ca15ebbf2..cc4a262d8e 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -135,7 +135,7 @@ static bool begininsert_called = false;
/* Memory context to hold the registered buffer and data references. */
static MemoryContext xloginsert_cxt;
-static XLogRecData *XLogRecordAssemble(RmgrId rmid, uint8 info,
+static XLogRecData *XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
XLogRecPtr RedoRecPtr, bool doPageWrites,
XLogRecPtr *fpw_lsn, int *num_fpi,
bool *topxid_included);
@@ -436,8 +436,9 @@ XLogSetRecordFlags(uint8 flags)
curinsert_flags |= flags;
}
+
/*
- * Insert an XLOG record having the specified RMID and info bytes, with the
+ * Insert an XLOG record having the specified RMID and rminfo bytes, with the
* body of the record being the data and buffer references registered earlier
* with XLogRegister* calls.
*
@@ -448,7 +449,21 @@ XLogSetRecordFlags(uint8 flags)
* WAL rule "write the log before the data".)
*/
XLogRecPtr
-XLogInsert(RmgrId rmid, uint8 info)
+XLogInsert(RmgrId rmid, uint8 rminfo)
+{
+ return XLogInsertExtended(rmid, 0, rminfo);
+}
+
+
+/*
+ * Insert an XLOG record having the specified RMID, info and rminfo bytes,
+ * with the body of the record being the data and buffer references
+ * registered earlier with XLogRegister* calls.
+ *
+ * See also XLogInsert above.
+ */
+XLogRecPtr
+XLogInsertExtended(RmgrId rmid, uint8 info, uint8 rminfo)
{
XLogRecPtr EndPos;
@@ -457,11 +472,10 @@ XLogInsert(RmgrId rmid, uint8 info)
elog(ERROR, "XLogBeginInsert was not called");
/*
- * The caller can set rmgr bits, XLR_SPECIAL_REL_UPDATE and
+ * The caller can set XLR_SPECIAL_REL_UPDATE and
* XLR_CHECK_CONSISTENCY; the rest are reserved for use by me.
*/
- if ((info & ~(XLR_RMGR_INFO_MASK |
- XLR_SPECIAL_REL_UPDATE |
+ if ((info & ~(XLR_SPECIAL_REL_UPDATE |
XLR_CHECK_CONSISTENCY)) != 0)
elog(PANIC, "invalid xlog info mask %02X", info);
@@ -494,7 +508,7 @@ XLogInsert(RmgrId rmid, uint8 info)
*/
GetFullPageWriteInfo(&RedoRecPtr, &doPageWrites);
- rdt = XLogRecordAssemble(rmid, info, RedoRecPtr, doPageWrites,
+ rdt = XLogRecordAssemble(rmid, info, rminfo, RedoRecPtr, doPageWrites,
&fpw_lsn, &num_fpi, &topxid_included);
EndPos = XLogInsertRecord(rdt, fpw_lsn, curinsert_flags, num_fpi,
@@ -522,7 +536,7 @@ XLogInsert(RmgrId rmid, uint8 info)
* current subtransaction.
*/
static XLogRecData *
-XLogRecordAssemble(RmgrId rmid, uint8 info,
+XLogRecordAssemble(RmgrId rmid, uint8 info, uint8 rminfo,
XLogRecPtr RedoRecPtr, bool doPageWrites,
XLogRecPtr *fpw_lsn, int *num_fpi, bool *topxid_included)
{
@@ -881,6 +895,7 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
rechdr->xl_tot_len = total_len;
rechdr->xl_info = info;
rechdr->xl_rmid = rmid;
+ rechdr->xl_rminfo = rminfo;
rechdr->xl_prev = InvalidXLogRecPtr;
rechdr->xl_crc = rdata_crc;
diff --git a/src/backend/access/transam/xlogprefetcher.c b/src/backend/access/transam/xlogprefetcher.c
index 1cbac4b7f6..e5d4f36182 100644
--- a/src/backend/access/transam/xlogprefetcher.c
+++ b/src/backend/access/transam/xlogprefetcher.c
@@ -536,7 +536,7 @@ XLogPrefetcherNextBlock(uintptr_t pgsr_private, XLogRecPtr *lsn)
if (replaying_lsn < record->lsn)
{
uint8 rmid = record->header.xl_rmid;
- uint8 record_type = record->header.xl_info & ~XLR_INFO_MASK;
+ uint8 record_type = record->header.xl_rminfo;
if (rmid == RM_XLOG_ID)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 5a8fe81f82..9200d7f56c 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -859,7 +859,7 @@ restart:
* Special processing if it's an XLOG SWITCH record
*/
if (record->xl_rmid == RM_XLOG_ID &&
- (record->xl_info & ~XLR_INFO_MASK) == XLOG_SWITCH)
+ record->xl_rminfo == XLOG_SWITCH)
{
/* Pretend it extends to end of segment */
state->NextRecPtr += state->segcxt.ws_segsize - 1;
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index cb07694aea..96d12994b5 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -614,7 +614,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr,
if (record != NULL)
{
memcpy(&checkPoint, XLogRecGetData(xlogreader), sizeof(CheckPoint));
- wasShutdown = ((record->xl_info & ~XLR_INFO_MASK) == XLOG_CHECKPOINT_SHUTDOWN);
+ wasShutdown = (record->xl_rminfo == XLOG_CHECKPOINT_SHUTDOWN);
ereport(DEBUG1,
(errmsg_internal("checkpoint record is at %X/%X",
LSN_FORMAT_ARGS(CheckPointLoc))));
@@ -768,7 +768,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr,
(errmsg("could not locate a valid checkpoint record")));
}
memcpy(&checkPoint, XLogRecGetData(xlogreader), sizeof(CheckPoint));
- wasShutdown = ((record->xl_info & ~XLR_INFO_MASK) == XLOG_CHECKPOINT_SHUTDOWN);
+ wasShutdown = (record->xl_rminfo == XLOG_CHECKPOINT_SHUTDOWN);
}
/*
@@ -1839,9 +1839,9 @@ ApplyWalRecord(XLogReaderState *xlogreader, XLogRecord *record, TimeLineID *repl
{
TimeLineID newReplayTLI = *replayTLI;
TimeLineID prevReplayTLI = *replayTLI;
- uint8 info = record->xl_info & ~XLR_INFO_MASK;
+ uint8 rminfo = record->xl_rminfo;
- if (info == XLOG_CHECKPOINT_SHUTDOWN)
+ if (rminfo == XLOG_CHECKPOINT_SHUTDOWN)
{
CheckPoint checkPoint;
@@ -1849,7 +1849,7 @@ ApplyWalRecord(XLogReaderState *xlogreader, XLogRecord *record, TimeLineID *repl
newReplayTLI = checkPoint.ThisTimeLineID;
prevReplayTLI = checkPoint.PrevTimeLineID;
}
- else if (info == XLOG_END_OF_RECOVERY)
+ else if (rminfo == XLOG_END_OF_RECOVERY)
{
xl_end_of_recovery xlrec;
@@ -1958,12 +1958,12 @@ ApplyWalRecord(XLogReaderState *xlogreader, XLogRecord *record, TimeLineID *repl
static void
xlogrecovery_redo(XLogReaderState *record, TimeLineID replayTLI)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
XLogRecPtr lsn = record->EndRecPtr;
Assert(XLogRecGetRmid(record) == RM_XLOG_ID);
- if (info == XLOG_OVERWRITE_CONTRECORD)
+ if (rminfo == XLOG_OVERWRITE_CONTRECORD)
{
/* Verify the payload of a XLOG_OVERWRITE_CONTRECORD record. */
xl_overwrite_contrecord xlrec;
@@ -1986,7 +1986,7 @@ xlogrecovery_redo(XLogReaderState *record, TimeLineID replayTLI)
/* Verifying the record should only happen once */
record->overwrittenRecPtr = InvalidXLogRecPtr;
}
- else if (info == XLOG_BACKUP_END)
+ else if (rminfo == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
@@ -2176,15 +2176,15 @@ void
xlog_outdesc(StringInfo buf, XLogReaderState *record)
{
RmgrData rmgr = GetRmgr(XLogRecGetRmid(record));
- uint8 info = XLogRecGetInfo(record);
+ uint8 rminfo = XLogRecGetRmInfo(record);
const char *id;
appendStringInfoString(buf, rmgr.rm_name);
appendStringInfoChar(buf, '/');
- id = rmgr.rm_identify(info);
+ id = rmgr.rm_identify(rminfo);
if (id == NULL)
- appendStringInfo(buf, "UNKNOWN (%X): ", info & ~XLR_INFO_MASK);
+ appendStringInfo(buf, "UNKNOWN (%X): ", rminfo);
else
appendStringInfo(buf, "%s: ", id);
@@ -2304,11 +2304,11 @@ checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI, TimeLineID prevTLI,
static bool
getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
- uint8 xact_info = info & XLOG_XACT_OPMASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
+ uint8 xact_info = rminfo & XLOG_XACT_OPMASK;
uint8 rmid = XLogRecGetRmid(record);
- if (rmid == RM_XLOG_ID && info == XLOG_RESTORE_POINT)
+ if (rmid == RM_XLOG_ID && rminfo == XLOG_RESTORE_POINT)
{
*recordXtime = ((xl_restore_point *) XLogRecGetData(record))->rp_time;
return true;
@@ -2498,7 +2498,7 @@ recoveryStopsBefore(XLogReaderState *record)
if (XLogRecGetRmid(record) != RM_XACT_ID)
return false;
- xact_info = XLogRecGetInfo(record) & XLOG_XACT_OPMASK;
+ xact_info = XLogRecGetRmInfo(record) & XLOG_XACT_OPMASK;
if (xact_info == XLOG_XACT_COMMIT)
{
@@ -2511,7 +2511,7 @@ recoveryStopsBefore(XLogReaderState *record)
xl_xact_parsed_commit parsed;
isCommit = true;
- ParseCommitRecord(XLogRecGetInfo(record),
+ ParseCommitRecord(XLogRecGetRmInfo(record),
xlrec,
&parsed);
recordXid = parsed.twophase_xid;
@@ -2527,7 +2527,7 @@ recoveryStopsBefore(XLogReaderState *record)
xl_xact_parsed_abort parsed;
isCommit = false;
- ParseAbortRecord(XLogRecGetInfo(record),
+ ParseAbortRecord(XLogRecGetRmInfo(record),
xlrec,
&parsed);
recordXid = parsed.twophase_xid;
@@ -2599,7 +2599,7 @@ recoveryStopsBefore(XLogReaderState *record)
static bool
recoveryStopsAfter(XLogReaderState *record)
{
- uint8 info;
+ uint8 rminfo;
uint8 xact_info;
uint8 rmid;
TimestampTz recordXtime;
@@ -2611,7 +2611,7 @@ recoveryStopsAfter(XLogReaderState *record)
if (!ArchiveRecoveryRequested)
return false;
- info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ rminfo = XLogRecGetRmInfo(record);
rmid = XLogRecGetRmid(record);
/*
@@ -2619,7 +2619,7 @@ recoveryStopsAfter(XLogReaderState *record)
* the first one.
*/
if (recoveryTarget == RECOVERY_TARGET_NAME &&
- rmid == RM_XLOG_ID && info == XLOG_RESTORE_POINT)
+ rmid == RM_XLOG_ID && rminfo == XLOG_RESTORE_POINT)
{
xl_restore_point *recordRestorePointData;
@@ -2660,7 +2660,7 @@ recoveryStopsAfter(XLogReaderState *record)
if (rmid != RM_XACT_ID)
return false;
- xact_info = info & XLOG_XACT_OPMASK;
+ xact_info = rminfo & XLOG_XACT_OPMASK;
if (xact_info == XLOG_XACT_COMMIT ||
xact_info == XLOG_XACT_COMMIT_PREPARED ||
@@ -2689,7 +2689,7 @@ recoveryStopsAfter(XLogReaderState *record)
xl_xact_abort *xlrec = (xl_xact_abort *) XLogRecGetData(record);
xl_xact_parsed_abort parsed;
- ParseAbortRecord(XLogRecGetInfo(record),
+ ParseAbortRecord(XLogRecGetRmInfo(record),
xlrec,
&parsed);
recordXid = parsed.twophase_xid;
@@ -2883,7 +2883,7 @@ recoveryApplyDelay(XLogReaderState *record)
if (XLogRecGetRmid(record) != RM_XACT_ID)
return false;
- xact_info = XLogRecGetInfo(record) & XLOG_XACT_OPMASK;
+ xact_info = XLogRecGetRmInfo(record) & XLOG_XACT_OPMASK;
if (xact_info != XLOG_XACT_COMMIT &&
xact_info != XLOG_XACT_COMMIT_PREPARED)
@@ -3917,7 +3917,7 @@ ReadCheckpointRecord(XLogPrefetcher *xlogprefetcher, XLogRecPtr RecPtr,
TimeLineID replayTLI)
{
XLogRecord *record;
- uint8 info;
+ uint8 rminfo;
Assert(xlogreader != NULL);
@@ -3943,9 +3943,9 @@ ReadCheckpointRecord(XLogPrefetcher *xlogprefetcher, XLogRecPtr RecPtr,
(errmsg("invalid resource manager ID in checkpoint record")));
return NULL;
}
- info = record->xl_info & ~XLR_INFO_MASK;
- if (info != XLOG_CHECKPOINT_SHUTDOWN &&
- info != XLOG_CHECKPOINT_ONLINE)
+ rminfo = record->xl_rminfo;
+ if (rminfo != XLOG_CHECKPOINT_SHUTDOWN &&
+ rminfo != XLOG_CHECKPOINT_ONLINE)
{
ereport(LOG,
(errmsg("invalid xl_info in checkpoint record")));
diff --git a/src/backend/access/transam/xlogstats.c b/src/backend/access/transam/xlogstats.c
index 514181792d..5ff1a58134 100644
--- a/src/backend/access/transam/xlogstats.c
+++ b/src/backend/access/transam/xlogstats.c
@@ -79,7 +79,7 @@ XLogRecStoreStats(XLogStats *stats, XLogReaderState *record)
* RmgrId).
*/
- recid = XLogRecGetInfo(record) >> 4;
+ recid = XLogRecGetRmInfo(record);
/*
* XACT records need to be handled differently. Those records use the
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index d708af19ed..4ef46a1855 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -194,7 +194,7 @@ log_smgrcreate(const RelFileLocator *rlocator, ForkNumber forkNum)
XLogBeginInsert();
XLogRegisterData((char *) &xlrec, sizeof(xlrec));
- XLogInsert(RM_SMGR_ID, XLOG_SMGR_CREATE | XLR_SPECIAL_REL_UPDATE);
+ XLogInsertExtended(RM_SMGR_ID, XLR_SPECIAL_REL_UPDATE, XLOG_SMGR_CREATE);
}
/*
@@ -375,8 +375,9 @@ RelationTruncate(Relation rel, BlockNumber nblocks)
XLogBeginInsert();
XLogRegisterData((char *) &xlrec, sizeof(xlrec));
- lsn = XLogInsert(RM_SMGR_ID,
- XLOG_SMGR_TRUNCATE | XLR_SPECIAL_REL_UPDATE);
+ lsn = XLogInsertExtended(RM_SMGR_ID,
+ XLR_SPECIAL_REL_UPDATE,
+ XLOG_SMGR_TRUNCATE);
/*
* Flush, because otherwise the truncation of the main relation might
@@ -958,12 +959,12 @@ void
smgr_redo(XLogReaderState *record)
{
XLogRecPtr lsn = record->EndRecPtr;
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
/* Backup blocks are not used in smgr records */
Assert(!XLogRecHasAnyBlockRefs(record));
- if (info == XLOG_SMGR_CREATE)
+ if (rminfo == XLOG_SMGR_CREATE)
{
xl_smgr_create *xlrec = (xl_smgr_create *) XLogRecGetData(record);
SMgrRelation reln;
@@ -971,7 +972,7 @@ smgr_redo(XLogReaderState *record)
reln = smgropen(xlrec->rlocator, InvalidBackendId);
smgrcreate(reln, xlrec->forkNum, true);
}
- else if (info == XLOG_SMGR_TRUNCATE)
+ else if (rminfo == XLOG_SMGR_TRUNCATE)
{
xl_smgr_truncate *xlrec = (xl_smgr_truncate *) XLogRecGetData(record);
SMgrRelation reln;
@@ -1060,5 +1061,5 @@ smgr_redo(XLogReaderState *record)
FreeFakeRelcacheEntry(rel);
}
else
- elog(PANIC, "smgr_redo: unknown op code %u", info);
+ elog(PANIC, "smgr_redo: unknown op code %u", rminfo);
}
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 96b46cbc02..25c4672917 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -624,8 +624,9 @@ CreateDatabaseUsingFileCopy(Oid src_dboid, Oid dst_dboid, Oid src_tsid,
XLogRegisterData((char *) &xlrec,
sizeof(xl_dbase_create_file_copy_rec));
- (void) XLogInsert(RM_DBASE_ID,
- XLOG_DBASE_CREATE_FILE_COPY | XLR_SPECIAL_REL_UPDATE);
+ (void) XLogInsertExtended(RM_DBASE_ID,
+ XLR_SPECIAL_REL_UPDATE,
+ XLOG_DBASE_CREATE_FILE_COPY);
}
pfree(srcpath);
pfree(dstpath);
@@ -2021,8 +2022,9 @@ movedb(const char *dbname, const char *tblspcname)
XLogRegisterData((char *) &xlrec,
sizeof(xl_dbase_create_file_copy_rec));
- (void) XLogInsert(RM_DBASE_ID,
- XLOG_DBASE_CREATE_FILE_COPY | XLR_SPECIAL_REL_UPDATE);
+ (void) XLogInsertExtended(RM_DBASE_ID,
+ XLR_SPECIAL_REL_UPDATE,
+ XLOG_DBASE_CREATE_FILE_COPY);
}
/*
@@ -2115,8 +2117,9 @@ movedb(const char *dbname, const char *tblspcname)
XLogRegisterData((char *) &xlrec, sizeof(xl_dbase_drop_rec));
XLogRegisterData((char *) &src_tblspcoid, sizeof(Oid));
- (void) XLogInsert(RM_DBASE_ID,
- XLOG_DBASE_DROP | XLR_SPECIAL_REL_UPDATE);
+ (void) XLogInsertExtended(RM_DBASE_ID,
+ XLR_SPECIAL_REL_UPDATE,
+ XLOG_DBASE_DROP);
}
/* Now it's safe to release the database lock */
@@ -2834,8 +2837,9 @@ remove_dbtablespaces(Oid db_id)
XLogRegisterData((char *) &xlrec, MinSizeOfDbaseDropRec);
XLogRegisterData((char *) tablespace_ids, ntblspc * sizeof(Oid));
- (void) XLogInsert(RM_DBASE_ID,
- XLOG_DBASE_DROP | XLR_SPECIAL_REL_UPDATE);
+ (void) XLogInsertExtended(RM_DBASE_ID,
+ XLR_SPECIAL_REL_UPDATE,
+ XLOG_DBASE_DROP);
}
list_free(ltblspc);
@@ -3040,12 +3044,12 @@ recovery_create_dbdir(char *path, bool only_tblspc)
void
dbase_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
/* Backup blocks are not used in dbase records */
Assert(!XLogRecHasAnyBlockRefs(record));
- if (info == XLOG_DBASE_CREATE_FILE_COPY)
+ if (rminfo == XLOG_DBASE_CREATE_FILE_COPY)
{
xl_dbase_create_file_copy_rec *xlrec =
(xl_dbase_create_file_copy_rec *) XLogRecGetData(record);
@@ -3117,7 +3121,7 @@ dbase_redo(XLogReaderState *record)
pfree(src_path);
pfree(dst_path);
}
- else if (info == XLOG_DBASE_CREATE_WAL_LOG)
+ else if (rminfo == XLOG_DBASE_CREATE_WAL_LOG)
{
xl_dbase_create_wal_log_rec *xlrec =
(xl_dbase_create_wal_log_rec *) XLogRecGetData(record);
@@ -3136,7 +3140,7 @@ dbase_redo(XLogReaderState *record)
true);
pfree(dbpath);
}
- else if (info == XLOG_DBASE_DROP)
+ else if (rminfo == XLOG_DBASE_DROP)
{
xl_dbase_drop_rec *xlrec = (xl_dbase_drop_rec *) XLogRecGetData(record);
char *dst_path;
@@ -3198,5 +3202,5 @@ dbase_redo(XLogReaderState *record)
}
}
else
- elog(PANIC, "dbase_redo: unknown op code %u", info);
+ elog(PANIC, "dbase_redo: unknown op code %u", rminfo);
}
diff --git a/src/backend/commands/sequence.c b/src/backend/commands/sequence.c
index 99c9f91cba..56221903b7 100644
--- a/src/backend/commands/sequence.c
+++ b/src/backend/commands/sequence.c
@@ -1845,7 +1845,7 @@ void
seq_redo(XLogReaderState *record)
{
XLogRecPtr lsn = record->EndRecPtr;
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
Buffer buffer;
Page page;
Page localpage;
@@ -1854,8 +1854,8 @@ seq_redo(XLogReaderState *record)
xl_seq_rec *xlrec = (xl_seq_rec *) XLogRecGetData(record);
sequence_magic *sm;
- if (info != XLOG_SEQ_LOG)
- elog(PANIC, "seq_redo: unknown op code %u", info);
+ if (rminfo != XLOG_SEQ_LOG)
+ elog(PANIC, "seq_redo: unknown op code %u", rminfo);
buffer = XLogInitBufferForRedo(record, 0);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index b69ff37dbb..d9c89bbbf9 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -1516,19 +1516,19 @@ get_tablespace_name(Oid spc_oid)
void
tblspc_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
/* Backup blocks are not used in tblspc records */
Assert(!XLogRecHasAnyBlockRefs(record));
- if (info == XLOG_TBLSPC_CREATE)
+ if (rminfo == XLOG_TBLSPC_CREATE)
{
xl_tblspc_create_rec *xlrec = (xl_tblspc_create_rec *) XLogRecGetData(record);
char *location = xlrec->ts_path;
create_tablespace_directories(location, xlrec->ts_id);
}
- else if (info == XLOG_TBLSPC_DROP)
+ else if (rminfo == XLOG_TBLSPC_DROP)
{
xl_tblspc_drop_rec *xlrec = (xl_tblspc_drop_rec *) XLogRecGetData(record);
@@ -1571,5 +1571,5 @@ tblspc_redo(XLogReaderState *record)
}
}
else
- elog(PANIC, "tblspc_redo: unknown op code %u", info);
+ elog(PANIC, "tblspc_redo: unknown op code %u", rminfo);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 2cc0ac9eb0..d6abeb9a9d 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -132,12 +132,12 @@ void
xlog_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
SnapBuild *builder = ctx->snapshot_builder;
- uint8 info = XLogRecGetInfo(buf->record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(buf->record);
ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(buf->record),
buf->origptr);
- switch (info)
+ switch (rminfo)
{
/* this is also used in END_OF_RECOVERY checkpoints */
case XLOG_CHECKPOINT_SHUTDOWN:
@@ -164,7 +164,7 @@ xlog_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_OVERWRITE_CONTRECORD:
break;
default:
- elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
+ elog(ERROR, "unexpected RM_XLOG_ID record type: %u", rminfo);
}
}
@@ -177,7 +177,7 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
SnapBuild *builder = ctx->snapshot_builder;
ReorderBuffer *reorder = ctx->reorder;
XLogReaderState *r = buf->record;
- uint8 info = XLogRecGetInfo(r) & XLOG_XACT_OPMASK;
+ uint8 info = XLogRecGetRmInfo(r) & XLOG_XACT_OPMASK;
/*
* If the snapshot isn't yet fully built, we cannot decode anything, so
@@ -197,7 +197,7 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
bool two_phase = false;
xlrec = (xl_xact_commit *) XLogRecGetData(r);
- ParseCommitRecord(XLogRecGetInfo(buf->record), xlrec, &parsed);
+ ParseCommitRecord(XLogRecGetRmInfo(buf->record), xlrec, &parsed);
if (!TransactionIdIsValid(parsed.twophase_xid))
xid = XLogRecGetXid(r);
@@ -225,7 +225,7 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
bool two_phase = false;
xlrec = (xl_xact_abort *) XLogRecGetData(r);
- ParseAbortRecord(XLogRecGetInfo(buf->record), xlrec, &parsed);
+ ParseAbortRecord(XLogRecGetRmInfo(buf->record), xlrec, &parsed);
if (!TransactionIdIsValid(parsed.twophase_xid))
xid = XLogRecGetXid(r);
@@ -288,7 +288,7 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
/* ok, parse it */
xlrec = (xl_xact_prepare *) XLogRecGetData(r);
- ParsePrepareRecord(XLogRecGetInfo(buf->record),
+ ParsePrepareRecord(XLogRecGetRmInfo(buf->record),
xlrec, &parsed);
/*
@@ -333,11 +333,11 @@ standby_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
SnapBuild *builder = ctx->snapshot_builder;
XLogReaderState *r = buf->record;
- uint8 info = XLogRecGetInfo(r) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(r);
ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(r), buf->origptr);
- switch (info)
+ switch (rminfo)
{
case XLOG_RUNNING_XACTS:
{
@@ -367,7 +367,7 @@ standby_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
*/
break;
default:
- elog(ERROR, "unexpected RM_STANDBY_ID record type: %u", info);
+ elog(ERROR, "unexpected RM_STANDBY_ID record type: %u", rminfo);
}
}
@@ -377,7 +377,7 @@ standby_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
void
heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
- uint8 info = XLogRecGetInfo(buf->record) & XLOG_HEAP_OPMASK;
+ uint8 rminfo = XLogRecGetRmInfo(buf->record) & XLOG_HEAP_OPMASK;
TransactionId xid = XLogRecGetXid(buf->record);
SnapBuild *builder = ctx->snapshot_builder;
@@ -391,7 +391,7 @@ heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
ctx->fast_forward)
return;
- switch (info)
+ switch (rminfo)
{
case XLOG_HEAP2_MULTI_INSERT:
if (!ctx->fast_forward &&
@@ -427,7 +427,7 @@ heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_HEAP2_LOCK_UPDATED:
break;
default:
- elog(ERROR, "unexpected RM_HEAP2_ID record type: %u", info);
+ elog(ERROR, "unexpected RM_HEAP2_ID record type: %u", rminfo);
}
}
@@ -437,7 +437,7 @@ heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
void
heap_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
- uint8 info = XLogRecGetInfo(buf->record) & XLOG_HEAP_OPMASK;
+ uint8 rminfo = XLogRecGetRmInfo(buf->record) & XLOG_HEAP_OPMASK;
TransactionId xid = XLogRecGetXid(buf->record);
SnapBuild *builder = ctx->snapshot_builder;
@@ -451,7 +451,7 @@ heap_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
ctx->fast_forward)
return;
- switch (info)
+ switch (rminfo)
{
case XLOG_HEAP_INSERT:
if (SnapBuildProcessChange(builder, xid, buf->origptr))
@@ -512,7 +512,7 @@ heap_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
default:
- elog(ERROR, "unexpected RM_HEAP_ID record type: %u", info);
+ elog(ERROR, "unexpected RM_HEAP_ID record type: %u", rminfo);
break;
}
}
@@ -562,13 +562,13 @@ logicalmsg_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
SnapBuild *builder = ctx->snapshot_builder;
XLogReaderState *r = buf->record;
TransactionId xid = XLogRecGetXid(r);
- uint8 info = XLogRecGetInfo(r) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(r);
RepOriginId origin_id = XLogRecGetOrigin(r);
Snapshot snapshot;
xl_logical_message *message;
- if (info != XLOG_LOGICAL_MESSAGE)
- elog(ERROR, "unexpected RM_LOGICALMSG_ID record type: %u", info);
+ if (rminfo != XLOG_LOGICAL_MESSAGE)
+ elog(ERROR, "unexpected RM_LOGICALMSG_ID record type: %u", rminfo);
ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(r), buf->origptr);
diff --git a/src/backend/replication/logical/message.c b/src/backend/replication/logical/message.c
index 1c34912610..564990b610 100644
--- a/src/backend/replication/logical/message.c
+++ b/src/backend/replication/logical/message.c
@@ -80,10 +80,10 @@ LogLogicalMessage(const char *prefix, const char *message, size_t size,
void
logicalmsg_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- if (info != XLOG_LOGICAL_MESSAGE)
- elog(PANIC, "logicalmsg_redo: unknown op code %u", info);
+ if (rminfo != XLOG_LOGICAL_MESSAGE)
+ elog(PANIC, "logicalmsg_redo: unknown op code %u", rminfo);
/* This is only interesting for logical decoding, see decode.c. */
}
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index f19b72ff35..cf6be4e28d 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -822,9 +822,9 @@ StartupReplicationOrigin(void)
void
replorigin_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
- switch (info)
+ switch (rminfo)
{
case XLOG_REPLORIGIN_SET:
{
@@ -861,7 +861,7 @@ replorigin_redo(XLogReaderState *record)
break;
}
default:
- elog(PANIC, "replorigin_redo: unknown op code %u", info);
+ elog(PANIC, "replorigin_redo: unknown op code %u", rminfo);
}
}
diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c
index 7db86f7885..74565e6cb0 100644
--- a/src/backend/storage/ipc/standby.c
+++ b/src/backend/storage/ipc/standby.c
@@ -1145,7 +1145,7 @@ StandbyReleaseOldLocks(TransactionId oldxid)
void
standby_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
/* Backup blocks are not used in standby records */
Assert(!XLogRecHasAnyBlockRefs(record));
@@ -1154,7 +1154,7 @@ standby_redo(XLogReaderState *record)
if (standbyState == STANDBY_DISABLED)
return;
- if (info == XLOG_STANDBY_LOCK)
+ if (rminfo == XLOG_STANDBY_LOCK)
{
xl_standby_locks *xlrec = (xl_standby_locks *) XLogRecGetData(record);
int i;
@@ -1164,7 +1164,7 @@ standby_redo(XLogReaderState *record)
xlrec->locks[i].dbOid,
xlrec->locks[i].relOid);
}
- else if (info == XLOG_RUNNING_XACTS)
+ else if (rminfo == XLOG_RUNNING_XACTS)
{
xl_running_xacts *xlrec = (xl_running_xacts *) XLogRecGetData(record);
RunningTransactionsData running;
@@ -1179,7 +1179,7 @@ standby_redo(XLogReaderState *record)
ProcArrayApplyRecoveryInfo(&running);
}
- else if (info == XLOG_INVALIDATIONS)
+ else if (rminfo == XLOG_INVALIDATIONS)
{
xl_invalidations *xlrec = (xl_invalidations *) XLogRecGetData(record);
@@ -1190,7 +1190,7 @@ standby_redo(XLogReaderState *record)
xlrec->tsId);
}
else
- elog(PANIC, "standby_redo: unknown op code %u", info);
+ elog(PANIC, "standby_redo: unknown op code %u", rminfo);
}
/*
diff --git a/src/backend/utils/cache/relmapper.c b/src/backend/utils/cache/relmapper.c
index 6266568605..c9bbd98598 100644
--- a/src/backend/utils/cache/relmapper.c
+++ b/src/backend/utils/cache/relmapper.c
@@ -1084,12 +1084,12 @@ perform_relmap_update(bool shared, const RelMapFile *updates)
void
relmap_redo(XLogReaderState *record)
{
- uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
/* Backup blocks are not used in relmap records */
Assert(!XLogRecHasAnyBlockRefs(record));
- if (info == XLOG_RELMAP_UPDATE)
+ if (rminfo == XLOG_RELMAP_UPDATE)
{
xl_relmap_update *xlrec = (xl_relmap_update *) XLogRecGetData(record);
RelMapFile newmap;
@@ -1126,5 +1126,5 @@ relmap_redo(XLogReaderState *record)
pfree(dbpath);
}
else
- elog(PANIC, "relmap_redo: unknown op code %u", info);
+ elog(PANIC, "relmap_redo: unknown op code %u", rminfo);
}
diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c
index 53f011a2fe..132c4db65e 100644
--- a/src/bin/pg_rewind/parsexlog.c
+++ b/src/bin/pg_rewind/parsexlog.c
@@ -201,7 +201,7 @@ findLastCheckpoint(const char *datadir, XLogRecPtr forkptr, int tliIndex,
searchptr = forkptr;
for (;;)
{
- uint8 info;
+ uint8 rminfo;
XLogBeginRead(xlogreader, searchptr);
record = XLogReadRecord(xlogreader, &errormsg);
@@ -222,11 +222,11 @@ findLastCheckpoint(const char *datadir, XLogRecPtr forkptr, int tliIndex,
* be the latest checkpoint before WAL forked and not the checkpoint
* where the primary has been stopped to be rewound.
*/
- info = XLogRecGetInfo(xlogreader) & ~XLR_INFO_MASK;
+ rminfo = XLogRecGetRmInfo(xlogreader);
if (searchptr < forkptr &&
XLogRecGetRmid(xlogreader) == RM_XLOG_ID &&
- (info == XLOG_CHECKPOINT_SHUTDOWN ||
- info == XLOG_CHECKPOINT_ONLINE))
+ (rminfo == XLOG_CHECKPOINT_SHUTDOWN ||
+ rminfo == XLOG_CHECKPOINT_ONLINE))
{
CheckPoint checkPoint;
@@ -370,7 +370,7 @@ extractPageInfo(XLogReaderState *record)
int block_id;
RmgrId rmid = XLogRecGetRmid(record);
uint8 info = XLogRecGetInfo(record);
- uint8 rminfo = info & ~XLR_INFO_MASK;
+ uint8 rminfo = XLogRecGetRmInfo(record);
/* Is this a special record type that I recognize? */
diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c
index 9993378ca5..70886beedd 100644
--- a/src/bin/pg_waldump/pg_waldump.c
+++ b/src/bin/pg_waldump/pg_waldump.c
@@ -449,7 +449,7 @@ XLogDumpDisplayRecord(XLogDumpConfig *config, XLogReaderState *record)
const RmgrDescData *desc = GetRmgrDesc(XLogRecGetRmid(record));
uint32 rec_len;
uint32 fpi_len;
- uint8 info = XLogRecGetInfo(record);
+ uint8 rminfo = XLogRecGetRmInfo(record);
XLogRecPtr xl_prev = XLogRecGetPrev(record);
StringInfoData s;
@@ -462,9 +462,9 @@ XLogDumpDisplayRecord(XLogDumpConfig *config, XLogReaderState *record)
LSN_FORMAT_ARGS(record->ReadRecPtr),
LSN_FORMAT_ARGS(xl_prev));
- id = desc->rm_identify(info);
+ id = desc->rm_identify(rminfo);
if (id == NULL)
- printf("desc: UNKNOWN (%x) ", info & ~XLR_INFO_MASK);
+ printf("desc: UNKNOWN (%x) ", rminfo);
else
printf("desc: %s ", id);
diff --git a/src/include/access/brin_xlog.h b/src/include/access/brin_xlog.h
index 012a9afdf4..9a4445c122 100644
--- a/src/include/access/brin_xlog.h
+++ b/src/include/access/brin_xlog.h
@@ -145,7 +145,7 @@ typedef struct xl_brin_desummarize
extern void brin_redo(XLogReaderState *record);
extern void brin_desc(StringInfo buf, XLogReaderState *record);
-extern const char *brin_identify(uint8 info);
+extern const char *brin_identify(uint8 rminfo);
extern void brin_mask(char *pagedata, BlockNumber blkno);
#endif /* BRIN_XLOG_H */
diff --git a/src/include/access/clog.h b/src/include/access/clog.h
index 543f2e2643..c8aa16da93 100644
--- a/src/include/access/clog.h
+++ b/src/include/access/clog.h
@@ -58,6 +58,6 @@ extern int clogsyncfiletag(const FileTag *ftag, char *path);
extern void clog_redo(XLogReaderState *record);
extern void clog_desc(StringInfo buf, XLogReaderState *record);
-extern const char *clog_identify(uint8 info);
+extern const char *clog_identify(uint8 rminfo);
#endif /* CLOG_H */
diff --git a/src/include/access/ginxlog.h b/src/include/access/ginxlog.h
index 7f985039bb..7b79f18ad7 100644
--- a/src/include/access/ginxlog.h
+++ b/src/include/access/ginxlog.h
@@ -208,7 +208,7 @@ typedef struct ginxlogDeleteListPages
extern void gin_redo(XLogReaderState *record);
extern void gin_desc(StringInfo buf, XLogReaderState *record);
-extern const char *gin_identify(uint8 info);
+extern const char *gin_identify(uint8 rminfo);
extern void gin_xlog_startup(void);
extern void gin_xlog_cleanup(void);
extern void gin_mask(char *pagedata, BlockNumber blkno);
diff --git a/src/include/access/gistxlog.h b/src/include/access/gistxlog.h
index 9bbe4c2622..9bb848a207 100644
--- a/src/include/access/gistxlog.h
+++ b/src/include/access/gistxlog.h
@@ -106,7 +106,7 @@ typedef struct gistxlogPageReuse
extern void gist_redo(XLogReaderState *record);
extern void gist_desc(StringInfo buf, XLogReaderState *record);
-extern const char *gist_identify(uint8 info);
+extern const char *gist_identify(uint8 rminfo);
extern void gist_xlog_startup(void);
extern void gist_xlog_cleanup(void);
extern void gist_mask(char *pagedata, BlockNumber blkno);
diff --git a/src/include/access/hash_xlog.h b/src/include/access/hash_xlog.h
index 59230706bb..e3200fa11c 100644
--- a/src/include/access/hash_xlog.h
+++ b/src/include/access/hash_xlog.h
@@ -261,7 +261,7 @@ typedef struct xl_hash_vacuum_one_page
extern void hash_redo(XLogReaderState *record);
extern void hash_desc(StringInfo buf, XLogReaderState *record);
-extern const char *hash_identify(uint8 info);
+extern const char *hash_identify(uint8 rminfo);
extern void hash_mask(char *pagedata, BlockNumber blkno);
#endif /* HASH_XLOG_H */
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 34220d93cf..306cc568eb 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -394,11 +394,11 @@ extern void HeapTupleHeaderAdvanceLatestRemovedXid(HeapTupleHeader tuple,
extern void heap_redo(XLogReaderState *record);
extern void heap_desc(StringInfo buf, XLogReaderState *record);
-extern const char *heap_identify(uint8 info);
+extern const char *heap_identify(uint8 rminfo);
extern void heap_mask(char *pagedata, BlockNumber blkno);
extern void heap2_redo(XLogReaderState *record);
extern void heap2_desc(StringInfo buf, XLogReaderState *record);
-extern const char *heap2_identify(uint8 info);
+extern const char *heap2_identify(uint8 rminfo);
extern void heap_xlog_logical_rewrite(XLogReaderState *r);
extern XLogRecPtr log_heap_freeze(Relation reln, Buffer buffer,
diff --git a/src/include/access/multixact.h b/src/include/access/multixact.h
index 4cbe17de7b..979f2fafe7 100644
--- a/src/include/access/multixact.h
+++ b/src/include/access/multixact.h
@@ -158,7 +158,7 @@ extern void multixact_twophase_postabort(TransactionId xid, uint16 info,
extern void multixact_redo(XLogReaderState *record);
extern void multixact_desc(StringInfo buf, XLogReaderState *record);
-extern const char *multixact_identify(uint8 info);
+extern const char *multixact_identify(uint8 rminfo);
extern char *mxid_to_string(MultiXactId multi, int nmembers,
MultiXactMember *members);
diff --git a/src/include/access/nbtxlog.h b/src/include/access/nbtxlog.h
index dd504d1885..0adecc3c05 100644
--- a/src/include/access/nbtxlog.h
+++ b/src/include/access/nbtxlog.h
@@ -350,6 +350,6 @@ extern void btree_mask(char *pagedata, BlockNumber blkno);
* prototypes for functions in nbtdesc.c
*/
extern void btree_desc(StringInfo buf, XLogReaderState *record);
-extern const char *btree_identify(uint8 info);
+extern const char *btree_identify(uint8 rminfo);
#endif /* NBTXLOG_H */
diff --git a/src/include/access/spgxlog.h b/src/include/access/spgxlog.h
index 930ffdd4f7..06a4a4e074 100644
--- a/src/include/access/spgxlog.h
+++ b/src/include/access/spgxlog.h
@@ -249,7 +249,7 @@ typedef struct spgxlogVacuumRedirect
extern void spg_redo(XLogReaderState *record);
extern void spg_desc(StringInfo buf, XLogReaderState *record);
-extern const char *spg_identify(uint8 info);
+extern const char *spg_identify(uint8 rminfo);
extern void spg_xlog_startup(void);
extern void spg_xlog_cleanup(void);
extern void spg_mask(char *pagedata, BlockNumber blkno);
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index c604ee11f8..31aabb96e3 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -512,9 +512,9 @@ extern void xact_desc(StringInfo buf, XLogReaderState *record);
extern const char *xact_identify(uint8 info);
/* also in xactdesc.c, so they can be shared between front/backend code */
-extern void ParseCommitRecord(uint8 info, xl_xact_commit *xlrec, xl_xact_parsed_commit *parsed);
-extern void ParseAbortRecord(uint8 info, xl_xact_abort *xlrec, xl_xact_parsed_abort *parsed);
-extern void ParsePrepareRecord(uint8 info, xl_xact_prepare *xlrec, xl_xact_parsed_prepare *parsed);
+extern void ParseCommitRecord(uint8 rminfo, xl_xact_commit *xlrec, xl_xact_parsed_commit *parsed);
+extern void ParseAbortRecord(uint8 rminfo, xl_xact_abort *xlrec, xl_xact_parsed_abort *parsed);
+extern void ParsePrepareRecord(uint8 rminfo, xl_xact_prepare *xlrec, xl_xact_parsed_prepare *parsed);
extern void EnterParallelMode(void);
extern void ExitParallelMode(void);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index dce265098e..cf8c72c134 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -211,7 +211,7 @@ extern void XLogSetReplicationSlotMinimumLSN(XLogRecPtr lsn);
extern void xlog_redo(XLogReaderState *record);
extern void xlog_desc(StringInfo buf, XLogReaderState *record);
-extern const char *xlog_identify(uint8 info);
+extern const char *xlog_identify(uint8 rminfo);
extern void issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index 001ff2f521..cfe53c7175 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -41,7 +41,8 @@
/* prototypes for public functions in xloginsert.c: */
extern void XLogBeginInsert(void);
extern void XLogSetRecordFlags(uint8 flags);
-extern XLogRecPtr XLogInsert(RmgrId rmid, uint8 info);
+extern XLogRecPtr XLogInsert(RmgrId rmid, uint8 rminfo);
+extern XLogRecPtr XLogInsertExtended(RmgrId rmid, uint8 info, uint8 rminfo);
extern void XLogEnsureRecordSpace(int max_block_id, int ndatas);
extern void XLogRegisterData(char *data, uint32 len);
extern void XLogRegisterBuffer(uint8 block_id, Buffer buffer, uint8 flags);
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index e87f91316a..fb6cae08ad 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -410,6 +410,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state,
#define XLogRecGetPrev(decoder) ((decoder)->record->header.xl_prev)
#define XLogRecGetInfo(decoder) ((decoder)->record->header.xl_info)
#define XLogRecGetRmid(decoder) ((decoder)->record->header.xl_rmid)
+#define XLogRecGetRmInfo(decoder) ((decoder)->record->header.xl_rminfo)
#define XLogRecGetXid(decoder) ((decoder)->record->header.xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record->record_origin)
#define XLogRecGetTopXid(decoder) ((decoder)->record->toplevel_xid)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 835151ec92..17093d93b6 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -45,7 +45,8 @@ typedef struct XLogRecord
XLogRecPtr xl_prev; /* ptr to previous record in log */
uint8 xl_info; /* flag bits, see below */
RmgrId xl_rmid; /* resource manager for this record */
- /* 2 bytes of padding here, initialize to zero */
+ uint8 xl_rminfo; /* flag bits for rmgr use */
+ /* 1 byte of padding here, initialize to zero */
pg_crc32c xl_crc; /* CRC for this record */
/* XLogRecordBlockHeaders and XLogRecordDataHeader follow, no padding */
@@ -54,14 +55,6 @@ typedef struct XLogRecord
#define SizeOfXLogRecord (offsetof(XLogRecord, xl_crc) + sizeof(pg_crc32c))
-/*
- * The high 4 bits in xl_info may be used freely by rmgr. The
- * XLR_SPECIAL_REL_UPDATE and XLR_CHECK_CONSISTENCY bits can be passed by
- * XLogInsert caller. The rest are set internally by XLogInsert.
- */
-#define XLR_INFO_MASK 0x0F
-#define XLR_RMGR_INFO_MASK 0xF0
-
/*
* If a WAL record modifies any relation files, in ways not covered by the
* usual block references, this flag is set. This is not used for anything
diff --git a/src/include/access/xlogstats.h b/src/include/access/xlogstats.h
index 7eb4370f2d..5df0f584c1 100644
--- a/src/include/access/xlogstats.h
+++ b/src/include/access/xlogstats.h
@@ -16,7 +16,7 @@
#include "access/rmgr.h"
#include "access/xlogreader.h"
-#define MAX_XLINFO_TYPES 16
+#define MAX_XLINFO_TYPES UINT8_MAX
typedef struct XLogRecStats
{
diff --git a/src/include/catalog/storage_xlog.h b/src/include/catalog/storage_xlog.h
index 44a5e2043b..4db1cdffad 100644
--- a/src/include/catalog/storage_xlog.h
+++ b/src/include/catalog/storage_xlog.h
@@ -54,6 +54,6 @@ extern void log_smgrcreate(const RelFileLocator *rlocator, ForkNumber forkNum);
extern void smgr_redo(XLogReaderState *record);
extern void smgr_desc(StringInfo buf, XLogReaderState *record);
-extern const char *smgr_identify(uint8 info);
+extern const char *smgr_identify(uint8 rminfo);
#endif /* STORAGE_XLOG_H */
diff --git a/src/include/commands/dbcommands_xlog.h b/src/include/commands/dbcommands_xlog.h
index 545e5430cc..109cd358d9 100644
--- a/src/include/commands/dbcommands_xlog.h
+++ b/src/include/commands/dbcommands_xlog.h
@@ -55,6 +55,6 @@ typedef struct xl_dbase_drop_rec
extern void dbase_redo(XLogReaderState *record);
extern void dbase_desc(StringInfo buf, XLogReaderState *record);
-extern const char *dbase_identify(uint8 info);
+extern const char *dbase_identify(uint8 rminfo);
#endif /* DBCOMMANDS_XLOG_H */
diff --git a/src/include/commands/sequence.h b/src/include/commands/sequence.h
index b3b04ccfa9..78c7b15cf2 100644
--- a/src/include/commands/sequence.h
+++ b/src/include/commands/sequence.h
@@ -64,7 +64,7 @@ extern void ResetSequenceCaches(void);
extern void seq_redo(XLogReaderState *record);
extern void seq_desc(StringInfo buf, XLogReaderState *record);
-extern const char *seq_identify(uint8 info);
+extern const char *seq_identify(uint8 rminfo);
extern void seq_mask(char *page, BlockNumber blkno);
#endif /* SEQUENCE_H */
diff --git a/src/include/commands/tablespace.h b/src/include/commands/tablespace.h
index a11c9e9473..749413fc07 100644
--- a/src/include/commands/tablespace.h
+++ b/src/include/commands/tablespace.h
@@ -64,6 +64,6 @@ extern void remove_tablespace_symlink(const char *linkloc);
extern void tblspc_redo(XLogReaderState *record);
extern void tblspc_desc(StringInfo buf, XLogReaderState *record);
-extern const char *tblspc_identify(uint8 info);
+extern const char *tblspc_identify(uint8 rminfo);
#endif /* TABLESPACE_H */
diff --git a/src/include/replication/message.h b/src/include/replication/message.h
index 0b396c5669..6d3a77ea36 100644
--- a/src/include/replication/message.h
+++ b/src/include/replication/message.h
@@ -36,6 +36,6 @@ extern XLogRecPtr LogLogicalMessage(const char *prefix, const char *message,
#define XLOG_LOGICAL_MESSAGE 0x00
extern void logicalmsg_redo(XLogReaderState *record);
extern void logicalmsg_desc(StringInfo buf, XLogReaderState *record);
-extern const char *logicalmsg_identify(uint8 info);
+extern const char *logicalmsg_identify(uint8 rminfo);
#endif /* PG_LOGICAL_MESSAGE_H */
diff --git a/src/include/storage/standbydefs.h b/src/include/storage/standbydefs.h
index c0234b6cf3..51625e45d9 100644
--- a/src/include/storage/standbydefs.h
+++ b/src/include/storage/standbydefs.h
@@ -22,7 +22,7 @@
/* Recovery handlers for the Standby Rmgr (RM_STANDBY_ID) */
extern void standby_redo(XLogReaderState *record);
extern void standby_desc(StringInfo buf, XLogReaderState *record);
-extern const char *standby_identify(uint8 info);
+extern const char *standby_identify(uint8 rminfo);
extern void standby_desc_invalidations(StringInfo buf,
int nmsgs, SharedInvalidationMessage *msgs,
Oid dbId, Oid tsId,
diff --git a/src/include/utils/relmapper.h b/src/include/utils/relmapper.h
index 92f1f779a4..6fd75b546f 100644
--- a/src/include/utils/relmapper.h
+++ b/src/include/utils/relmapper.h
@@ -68,6 +68,6 @@ extern void RestoreRelationMap(char *startAddress);
extern void relmap_redo(XLogReaderState *record);
extern void relmap_desc(StringInfo buf, XLogReaderState *record);
-extern const char *relmap_identify(uint8 info);
+extern const char *relmap_identify(uint8 rminfo);
#endif /* RELMAPPER_H */
--
2.30.2
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-12 21:13 Andres Freund <[email protected]>
parent: Matthias van de Meent <[email protected]>
0 siblings, 2 replies; 25+ messages in thread
From: Andres Freund @ 2022-10-12 21:13 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: Robert Haas <[email protected]>; Peter Geoghegan <[email protected]>; David Rowley <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Heikki Linnakangas <[email protected]>
Hi,
On 2022-10-12 22:05:30 +0200, Matthias van de Meent wrote:
> On Wed, 5 Oct 2022 at 01:50, Andres Freund <[email protected]> wrote:
> > On 2022-10-03 10:01:25 -0700, Andres Freund wrote:
> > > On 2022-10-03 08:12:39 -0400, Robert Haas wrote:
> > > > On Fri, Sep 30, 2022 at 8:20 PM Andres Freund <[email protected]> wrote:
> > > > I thought about trying to buy back some space elsewhere, and I think
> > > > that would be a reasonable approach to getting this committed if we
> > > > could find a way to do it. However, I don't see a terribly obvious way
> > > > of making it happen.
> > >
> > > I think there's plenty potential...
> >
> > I light dusted off my old varint implementation from [1] and converted the
> > RelFileLocator and BlockNumber from fixed width integers to varint ones. This
> > isn't meant as a serious patch, but an experiment to see if this is a path
> > worth pursuing.
> >
> > A run of installcheck in a cluster with autovacuum=off, full_page_writes=off
> > (for increased reproducability) shows a decent saving:
> >
> > master: 241106544 - 230 MB
> > varint: 227858640 - 217 MB
>
> I think a signficant part of this improvement comes from the premise
> of starting with a fresh database. tablespace OID will indeed most
> likely be low, but database OID may very well be linearly distributed
> if concurrent workloads in the cluster include updating (potentially
> unlogged) TOASTed columns and the databases are not created in one
> "big bang" but over the lifetime of the cluster. In that case DBOID
> will consume 5B for a significant fraction of databases (anything with
> OID >=2^28).
>
> My point being: I don't think that we should have different WAL
> performance in databases which is dependent on which OID was assigned
> to that database.
To me this is raising the bar to an absurd level. Some minor space usage
increase after oid wraparound and for very large block numbers isn't a huge
issue - if you're in that situation you already have a huge amount of wal.
> 0002 - Rework XLogRecord
> This makes many fields in the xlog header optional, reducing the size
> of many xlog records by several bytes. This implements the design I
> shared in my earlier message [1].
>
> 0003 - Rework XLogRecordBlockHeader.
> This patch could be applied on current head, and saves some bytes in
> per-block data. It potentially saves some bytes per registered
> block/buffer in the WAL record (max 2 bytes for the first block, after
> that up to 3). See the patch's commit message in the patch for
> detailed information.
The amount of complexity these two introduce seems quite substantial to
me. Both from an maintenance and a runtime perspective. I think we'd be better
off using building blocks like variable lengths encoded values than open
coding it in many places.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-13 22:53 Matthias van de Meent <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 0 replies; 25+ messages in thread
From: Matthias van de Meent @ 2022-10-13 22:53 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Peter Geoghegan <[email protected]>; David Rowley <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Heikki Linnakangas <[email protected]>
On Wed, 12 Oct 2022 at 23:13, Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2022-10-12 22:05:30 +0200, Matthias van de Meent wrote:
> > On Wed, 5 Oct 2022 at 01:50, Andres Freund <[email protected]> wrote:
> > > I light dusted off my old varint implementation from [1] and converted the
> > > RelFileLocator and BlockNumber from fixed width integers to varint ones. This
> > > isn't meant as a serious patch, but an experiment to see if this is a path
> > > worth pursuing.
> > >
> > > A run of installcheck in a cluster with autovacuum=off, full_page_writes=off
> > > (for increased reproducability) shows a decent saving:
> > >
> > > master: 241106544 - 230 MB
> > > varint: 227858640 - 217 MB
> >
> > I think a signficant part of this improvement comes from the premise
> > of starting with a fresh database. tablespace OID will indeed most
> > likely be low, but database OID may very well be linearly distributed
> > if concurrent workloads in the cluster include updating (potentially
> > unlogged) TOASTed columns and the databases are not created in one
> > "big bang" but over the lifetime of the cluster. In that case DBOID
> > will consume 5B for a significant fraction of databases (anything with
> > OID >=2^28).
> >
> > My point being: I don't think that we should have different WAL
> > performance in databases which is dependent on which OID was assigned
> > to that database.
>
> To me this is raising the bar to an absurd level. Some minor space usage
> increase after oid wraparound and for very large block numbers isn't a huge
> issue - if you're in that situation you already have a huge amount of wal.
I didn't want to block all varlen encoding, I just want to make clear
that I don't think it's great for performance testing and consistency
across installations if WAL size (and thus part of your performance)
is dependent on which actual database/relation/tablespace combination
you're running your workload in.
With the 56-bit relfilenode, the size of a block reference would
realistically differ between 7 bytes and 23 bytes:
- tblspc=0=1B
db=16386=3B
rel=797=2B (787 = 4 * default # of data relations in a fresh DB in PG14, + 1)
block=0=1B
vs
- tsp>=2^28 = 5B
dat>=2^28 =5B
rel>=2^49 =8B
block>=2^28 =5B
That's a difference of 16 bytes, of which only the block number can
realistically be directly influenced by the user ("just don't have
relations larger than X blocks").
If applied to Dilip's pgbench transaction data, that would imply a
minimum per transaction wal usage of 509 bytes, and a maximum per
transaction wal usage of 609 bytes. That is nearly a 20% difference in
WAL size based only on the location of your data, and I'm just not
comfortable with that. Users have little or zero control over the
internal IDs we assign to these fields, while it would affect
performance fairly significantly.
(difference % between min/max wal size is unchanged (within 0.1%)
after accounting for record alignment)
> > 0002 - Rework XLogRecord
> > This makes many fields in the xlog header optional, reducing the size
> > of many xlog records by several bytes. This implements the design I
> > shared in my earlier message [1].
> >
> > 0003 - Rework XLogRecordBlockHeader.
> > This patch could be applied on current head, and saves some bytes in
> > per-block data. It potentially saves some bytes per registered
> > block/buffer in the WAL record (max 2 bytes for the first block, after
> > that up to 3). See the patch's commit message in the patch for
> > detailed information.
>
> The amount of complexity these two introduce seems quite substantial to
> me. Both from an maintenance and a runtime perspective. I think we'd be better
> off using building blocks like variable lengths encoded values than open
> coding it in many places.
I guess that's true for length fields, but I don't think dynamic
header field presence (the 0002 rewrite, and the omission of
data_length in 0003) is that bad. We already have dynamic data
inclusion through block ids 25x; I'm not sure why we couldn't do that
more compactly with bitfields as indicators instead (hence the dynamic
header size).
As for complexity, I think my current patchset is mostly complex due
to a lack of tooling. Note that decoding makes common use of
COPY_HEADER_FIELD, which we don't really have an equivalent for in
XLogRecordAssemble. I think the code for 0002 would improve
significantly in readability if such construct would be available.
To reduce complexity in 0003, I could drop the 'repeat id'
optimization, as that reduces the complexity significantly, at the
cost of not saving that 1 byte per registered block after the first.
Kind regards,
Matthias van de Meent
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-17 21:14 Robert Haas <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 25+ messages in thread
From: Robert Haas @ 2022-10-17 21:14 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Peter Geoghegan <[email protected]>; David Rowley <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Heikki Linnakangas <[email protected]>
On Wed, Oct 12, 2022 at 5:13 PM Andres Freund <[email protected]> wrote:
> > I think a signficant part of this improvement comes from the premise
> > of starting with a fresh database. tablespace OID will indeed most
> > likely be low, but database OID may very well be linearly distributed
> > if concurrent workloads in the cluster include updating (potentially
> > unlogged) TOASTed columns and the databases are not created in one
> > "big bang" but over the lifetime of the cluster. In that case DBOID
> > will consume 5B for a significant fraction of databases (anything with
> > OID >=2^28).
> >
> > My point being: I don't think that we should have different WAL
> > performance in databases which is dependent on which OID was assigned
> > to that database.
>
> To me this is raising the bar to an absurd level. Some minor space usage
> increase after oid wraparound and for very large block numbers isn't a huge
> issue - if you're in that situation you already have a huge amount of wal.
I have to admit that I worried about the same thing that Matthias
raises, more or less. But I don't know whether I'm right to be
worried. A variable-length representation of any kind is essentially a
gamble that values requiring fewer bytes will be more common than
values requiring more bytes, and by enough to justify the overhead
that the method has. And, you want it to be more common for each
individual user, not just overall. For example, more people are going
to have small relations than large ones, but nobody wants performance
to drop off a cliff when the relation passes a certain size threshold.
Now, it wouldn't drop off a cliff here, but what about someone with a
really big, append-only relation? Won't they just end up writing more
to WAL than with the present system?
Maybe not. They might still have some writes to relations other than
the very large, append-only relation, and then they could still win.
Also, if we assume that the overhead of the variable-length
representation is never more than 1 byte beyond what is needed to
represent the underlying quantity in the minimal number of bytes, they
are only going to lose if their relation is already more than half the
maximum theoretical size, and if that is the case, they are in danger
of hitting the size limit anyway. You can argue that there's still a
risk here, but it doesn't seem like that bad of a risk.
But the same thing is not so obvious for, let's say, database OIDs.
What if you just have one or a few databases, but due to the previous
history of the cluster, their OIDs just happen to be big? Then you're
just behind where you would have been without the patch. Granted, if
this happens to you, you will be in the minority, because most users
are likely to have small database OIDs, but the fact that other people
are writing less WAL on average isn't going to make you happy about
writing more WAL on average. And even for a user for which that
doesn't happen, it's not at all unlikely that the gains they see will
be less than what we see on a freshly-initdb'd database.
So I don't really know what the answer is here. I don't think this
technique sucks, but I don't think it's necessarily a categorical win
for every case, either. And it even seems hard to reason about which
cases are likely to be wins and which cases are likely to be losses.
> > 0002 - Rework XLogRecord
> > This makes many fields in the xlog header optional, reducing the size
> > of many xlog records by several bytes. This implements the design I
> > shared in my earlier message [1].
> >
> > 0003 - Rework XLogRecordBlockHeader.
> > This patch could be applied on current head, and saves some bytes in
> > per-block data. It potentially saves some bytes per registered
> > block/buffer in the WAL record (max 2 bytes for the first block, after
> > that up to 3). See the patch's commit message in the patch for
> > detailed information.
>
> The amount of complexity these two introduce seems quite substantial to
> me. Both from a maintenance and a runtime perspective. I think we'd be better
> off using building blocks like variable lengths encoded values than open
> coding it in many places.
I agree that this looks pretty ornate as written, but I think there
might be some good ideas in here, too. It is also easy to reason about
this kind of thing at least in terms of space consumption. It's a bit
harder to know how things will play out in terms of CPU cycles and
code complexity.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-19 19:21 Andres Freund <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Andres Freund @ 2022-10-19 19:21 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Peter Geoghegan <[email protected]>; David Rowley <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Heikki Linnakangas <[email protected]>
Hi,
On 2022-10-17 17:14:21 -0400, Robert Haas wrote:
> I have to admit that I worried about the same thing that Matthias
> raises, more or less. But I don't know whether I'm right to be
> worried. A variable-length representation of any kind is essentially a
> gamble that values requiring fewer bytes will be more common than
> values requiring more bytes, and by enough to justify the overhead
> that the method has. And, you want it to be more common for each
> individual user, not just overall. For example, more people are going
> to have small relations than large ones, but nobody wants performance
> to drop off a cliff when the relation passes a certain size threshold.
> Now, it wouldn't drop off a cliff here, but what about someone with a
> really big, append-only relation? Won't they just end up writing more
> to WAL than with the present system?
Perhaps. But I suspect it'd be a very small increase because they'd be using
bulk-insert paths in all likelihood anyway, if they managed to get to a very
large relation. And even in that case, if we e.g. were to make the record size
variable length, they'd still pretty much never reach that and it'd be an
overall win.
The number of people with that large relations, leaving partitioning aside
which'd still benefit as each relation is smaller, strikes me as a very small
percentage. And as you say, it's not like there's a cliff where everything
starts to be horrible.
> Maybe not. They might still have some writes to relations other than
> the very large, append-only relation, and then they could still win.
> Also, if we assume that the overhead of the variable-length
> representation is never more than 1 byte beyond what is needed to
> represent the underlying quantity in the minimal number of bytes, they
> are only going to lose if their relation is already more than half the
> maximum theoretical size, and if that is the case, they are in danger
> of hitting the size limit anyway. You can argue that there's still a
> risk here, but it doesn't seem like that bad of a risk.
Another thing here is that I suspect we ought to increase our relation size
beyond 4 byte * blocksize at some point - and then we'll have to use variable
encodings... Admittedly the amount of work needed to get there is substantial.
Somewhat relatedly, I think we, very slowly, should move towards wider OIDs as
well. Not having to deal with oid wraparound will be a significant win
(particularly for toast), but to keep the overhead reasonable, we're going to
need variable encodings.
> But the same thing is not so obvious for, let's say, database OIDs.
> What if you just have one or a few databases, but due to the previous
> history of the cluster, their OIDs just happen to be big? Then you're
> just behind where you would have been without the patch. Granted, if
> this happens to you, you will be in the minority, because most users
> are likely to have small database OIDs, but the fact that other people
> are writing less WAL on average isn't going to make you happy about
> writing more WAL on average. And even for a user for which that
> doesn't happen, it's not at all unlikely that the gains they see will
> be less than what we see on a freshly-initdb'd database.
I agree that going for variable width encodings on the basis of the database
oid field alone would be an unconvincing proposition. But variably encoding
database oids when we already variably encode other fields seems like a decent
bet. If you e.g. think of the 56-bit relfilenode field itself - obviously what
I was thinking about in the first place - it's going to be a win much more
often.
To really loose you'd not just have to have a large database oid, but also a
large tablespace and relation oid and a huge block number...
> So I don't really know what the answer is here. I don't think this
> technique sucks, but I don't think it's necessarily a categorical win
> for every case, either. And it even seems hard to reason about which
> cases are likely to be wins and which cases are likely to be losses.
True. I'm far less concerned than you or Matthias about increasing the size in
rare cases as long as it wins in the majority of cases. But that doesn't mean
every case is easy to consider.
> > > 0002 - Rework XLogRecord
> > > This makes many fields in the xlog header optional, reducing the size
> > > of many xlog records by several bytes. This implements the design I
> > > shared in my earlier message [1].
> > >
> > > 0003 - Rework XLogRecordBlockHeader.
> > > This patch could be applied on current head, and saves some bytes in
> > > per-block data. It potentially saves some bytes per registered
> > > block/buffer in the WAL record (max 2 bytes for the first block, after
> > > that up to 3). See the patch's commit message in the patch for
> > > detailed information.
> >
> > The amount of complexity these two introduce seems quite substantial to
> > me. Both from a maintenance and a runtime perspective. I think we'd be better
> > off using building blocks like variable lengths encoded values than open
> > coding it in many places.
>
> I agree that this looks pretty ornate as written, but I think there
> might be some good ideas in here, too.
Agreed! Several of the ideas seem orthogonal to using variable encodings, so
this isn't really an either / or.
> It is also easy to reason about this kind of thing at least in terms of
> space consumption.
Hm, not for me, but...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: problems with making relfilenodes 56-bits
@ 2022-10-20 08:40 Dilip Kumar <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dilip Kumar @ 2022-10-20 08:40 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Matthias van de Meent <[email protected]>; Peter Geoghegan <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Heikki Linnakangas <[email protected]>
On Thu, Oct 20, 2022 at 12:51 AM Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2022-10-17 17:14:21 -0400, Robert Haas wrote:
> > I have to admit that I worried about the same thing that Matthias
> > raises, more or less. But I don't know whether I'm right to be
> > worried. A variable-length representation of any kind is essentially a
> > gamble that values requiring fewer bytes will be more common than
> > values requiring more bytes, and by enough to justify the overhead
> > that the method has. And, you want it to be more common for each
> > individual user, not just overall. For example, more people are going
> > to have small relations than large ones, but nobody wants performance
> > to drop off a cliff when the relation passes a certain size threshold.
> > Now, it wouldn't drop off a cliff here, but what about someone with a
> > really big, append-only relation? Won't they just end up writing more
> > to WAL than with the present system?
>
> Perhaps. But I suspect it'd be a very small increase because they'd be using
> bulk-insert paths in all likelihood anyway, if they managed to get to a very
> large relation. And even in that case, if we e.g. were to make the record size
> variable length, they'd still pretty much never reach that and it'd be an
> overall win.
I think the number of cases where we will reduce the WAL size will be
far more than the cases where it will slightly increase the size. And
also the number of bytes we save in winning cases is far bigger than
the number of bytes we increase. So IMHO it seems like an overall win
at least from the WAL size reduction pov. Do we have some number that
how much overhead it has for encoding/decoding?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH 5/6] Use background worker to do logical decoding.
@ 2026-01-08 16:47 Antonin Houska <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Antonin Houska @ 2026-01-08 16:47 UTC (permalink / raw)
If the backend performing REPACK (CONCURRENTLY) does both data copying and
logical decoding, it has to "travel in time" back and forth and therefore it
has to invalidate system caches quite a few times. (The copying and the
decoding work with different catalog snapshots.) As the decoding worker has
separate caches, the switching is not necessary.
Without the worker, it'd also be difficult to switch between potentiallly long
running tasks like index build and WAL decoding. (No decoding during that time
at all can suspend archiving / recycling of WAL segments for some time, which
in turn may result in full disk.)
Another problem is that, after having acquired AccessExclusiveLock (in order
to swap the files), the backend needs to both decode and apply the data
changes that took place while it was waiting for the lock. With the decoding
worker, the decoding runs all the time, so the backend only needs to apply the
changes. This can reduce the time the exclusive lock is held for.
Note that the code added in order to handle ERRORs in the background worker
almost duplicates the existing code that does the same for other types of
workers (See ProcessParallelMessages() and
ProcessParallelApplyMessages()). Refactoring of the existing code might be
useful, to reduce the duplication.
---
src/backend/access/heap/heapam_handler.c | 44 -
src/backend/commands/cluster.c | 1174 +++++++++++++----
src/backend/libpq/pqmq.c | 5 +
src/backend/postmaster/bgworker.c | 4 +
src/backend/replication/logical/logical.c | 6 +-
.../pgoutput_repack/pgoutput_repack.c | 54 +-
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/tcop/postgres.c | 4 +
.../utils/activity/wait_event_names.txt | 2 +
src/include/access/tableam.h | 7 +-
src/include/commands/cluster.h | 71 +-
src/include/storage/procsignal.h | 1 +
src/tools/pgindent/typedefs.list | 4 +-
13 files changed, 979 insertions(+), 401 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 3526b6adcb5..475c536ce43 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -33,7 +33,6 @@
#include "catalog/index.h"
#include "catalog/storage.h"
#include "catalog/storage_xlog.h"
-#include "commands/cluster.h"
#include "commands/progress.h"
#include "executor/executor.h"
#include "miscadmin.h"
@@ -688,7 +687,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
Relation OldIndex, bool use_sort,
TransactionId OldestXmin,
Snapshot snapshot,
- LogicalDecodingContext *decoding_ctx,
TransactionId *xid_cutoff,
MultiXactId *multi_cutoff,
double *num_tuples,
@@ -710,7 +708,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
BufferHeapTupleTableSlot *hslot;
BlockNumber prev_cblock = InvalidBlockNumber;
bool concurrent = snapshot != NULL;
- XLogRecPtr end_of_wal_prev = GetFlushRecPtr(NULL);
/* Remember if it's a system catalog */
is_system_catalog = IsSystemRelation(OldHeap);
@@ -957,31 +954,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
ct_val[1] = *num_tuples;
pgstat_progress_update_multi_param(2, ct_index, ct_val);
}
-
- /*
- * Process the WAL produced by the load, as well as by other
- * transactions, so that the replication slot can advance and WAL does
- * not pile up. Use wal_segment_size as a threshold so that we do not
- * introduce the decoding overhead too often.
- *
- * Of course, we must not apply the changes until the initial load has
- * completed.
- *
- * Note that our insertions into the new table should not be decoded
- * as we (intentionally) do not write the logical decoding specific
- * information to WAL.
- */
- if (concurrent)
- {
- XLogRecPtr end_of_wal;
-
- end_of_wal = GetFlushRecPtr(NULL);
- if ((end_of_wal - end_of_wal_prev) > wal_segment_size)
- {
- repack_decode_concurrent_changes(decoding_ctx, end_of_wal);
- end_of_wal_prev = end_of_wal;
- }
- }
}
if (indexScan != NULL)
@@ -1027,22 +999,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
/* Report n_tuples */
pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED,
n_tuples);
-
- /*
- * Try to keep the amount of not-yet-decoded WAL small, like
- * above.
- */
- if (concurrent)
- {
- XLogRecPtr end_of_wal;
-
- end_of_wal = GetFlushRecPtr(NULL);
- if ((end_of_wal - end_of_wal_prev) > wal_segment_size)
- {
- repack_decode_concurrent_changes(decoding_ctx, end_of_wal);
- end_of_wal_prev = end_of_wal;
- }
- }
}
tuplesort_end(tuplesort);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index c3feb0c3de4..5232fbfb57d 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -12,12 +12,13 @@
* In concurrent mode, we lock the table with only ShareUpdateExclusiveLock,
* then do an initial copy as above. However, while the tuples are being
* copied, concurrent transactions could modify the table. To cope with those
- * changes, we rely on logical decoding to obtain them from WAL. The changes
- * are accumulated in a tuplestore. Once the initial copy is complete, we
- * read the changes from the tuplestore and re-apply them on the new heap.
- * Then we upgrade our ShareUpdateExclusiveLock to AccessExclusiveLock and
- * swap the relfilenodes. This way, the time we hold a strong lock on the
- * table is much reduced, and the bloat is eliminated.
+ * changes, we rely on logical decoding to obtain them from WAL. A bgworker
+ * consumes WAL while the initial copy is ongoing (to prevent excessive WAL
+ * from being reserved), and accumulates the changes in a file. Once the
+ * initial copy is complete, we read the changes from the file and re-apply
+ * them on the new heap. Then we upgrade our ShareUpdateExclusiveLock to
+ * AccessExclusiveLock and swap the relfilenodes. This way, the time we hold
+ * a strong lock on the table is much reduced, and the bloat is eliminated.
*
* There is hardly anything left of Paul Brown's original implementation...
*
@@ -45,6 +46,7 @@
#include "access/xlog_internal.h"
#include "access/xloginsert.h"
#include "access/xlogutils.h"
+#include "access/xlogwait.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
@@ -61,6 +63,8 @@
#include "commands/tablecmds.h"
#include "commands/vacuum.h"
#include "executor/executor.h"
+#include "libpq/pqformat.h"
+#include "libpq/pqmq.h"
#include "miscadmin.h"
#include "optimizer/optimizer.h"
#include "pgstat.h"
@@ -71,6 +75,8 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/predicate.h"
+#include "storage/procsignal.h"
+#include "tcop/tcopprot.h"
#include "utils/acl.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -117,6 +123,12 @@ typedef struct IndexInsertState
/* The WAL segment being decoded. */
static XLogSegNo repack_current_segment = 0;
+/*
+ * The first file exported by the decoding worker must contain a snapshot, the
+ * following ones contain the data changes.
+ */
+#define WORKER_FILE_SNAPSHOT 0
+
/*
* Information needed to apply concurrent data changes.
*/
@@ -136,8 +148,113 @@ typedef struct ChangeDest
/* Needed to update indexes of rel_dst. */
IndexInsertState *iistate;
+
+ /*
+ * Sequential number of the file containing the changes.
+ *
+ * TODO This field makes the structure name less descriptive. Should we
+ * rename it, e.g. to ChangeApplyInfo?
+ */
+ int file_seq;
} ChangeDest;
+/*
+ * Layout of shared memory used for communication between backend and the
+ * worker that performs logical decoding of data changes
+ */
+typedef struct DecodingWorkerShared
+{
+ /* Is the decoding initialized? */
+ bool initialized;
+
+ /*
+ * Once the worker has reached this LSN, it should close the current
+ * output file and either create a new one or exit, according to the field
+ * 'done'. If the value is InvalidXLogRecPtr, the worker should decode all
+ * the WAL available and keep checking this field. It is ok if the worker
+ * had already decoded records whose LSN is >= lsn_upto before this field
+ * has been set.
+ */
+ XLogRecPtr lsn_upto;
+
+ /* Exit after closing the current file? */
+ bool done;
+
+ /* The output is stored here. */
+ SharedFileSet sfs;
+
+ /* Number of the last file exported by the worker. */
+ int last_exported;
+
+ /* Synchronize access to the fields above. */
+ slock_t mutex;
+
+ /* Database to connect to. */
+ Oid dbid;
+
+ /* Role to connect as. */
+ Oid roleid;
+
+ /* Decode data changes of this relation. */
+ Oid relid;
+
+ /* The backend uses this to wait for the worker. */
+ ConditionVariable cv;
+
+ /* Info to signal the backend. */
+ PGPROC *backend_proc;
+ pid_t backend_pid;
+ ProcNumber backend_proc_number;
+
+ /* Error queue. */
+ shm_mq *error_mq;
+
+ /*
+ * Memory the queue is located int.
+ *
+ * For considerations on the value see the comments of
+ * PARALLEL_ERROR_QUEUE_SIZE.
+ */
+#define REPACK_ERROR_QUEUE_SIZE 16384
+ char error_queue[FLEXIBLE_ARRAY_MEMBER];
+} DecodingWorkerShared;
+
+/*
+ * Generate worker's output file name. If relations of the same 'relid' happen
+ * to be processed at the same time, they must be from different databases and
+ * therefore different backends must be involved. (PID is already present in
+ * the fileset name.)
+ */
+static inline void
+DecodingWorkerFileName(char *fname, Oid relid, uint32 seq)
+{
+ snprintf(fname, MAXPGPATH, "%u-%u", relid, seq);
+}
+
+/*
+ * Backend-local information to control the decoding worker.
+ */
+typedef struct DecodingWorker
+{
+ /* The worker. */
+ BackgroundWorkerHandle *handle;
+
+ /* DecodingWorkerShared is in this segment. */
+ dsm_segment *seg;
+
+ /* Handle of the error queue. */
+ shm_mq_handle *error_mqh;
+} DecodingWorker;
+
+/* Pointer to currently running decoding worker. */
+static DecodingWorker *decoding_worker = NULL;
+
+/*
+ * Is there a message sent by a repack worker that the backend needs to
+ * receive?
+ */
+volatile sig_atomic_t RepackMessagePending = false;
+
static bool cluster_rel_recheck(RepackCommand cmd, Relation OldHeap,
Oid indexOid, Oid userid, LOCKMODE lmode,
int options);
@@ -145,7 +262,7 @@ static void check_repack_concurrently_requirements(Relation rel);
static void rebuild_relation(Relation OldHeap, Relation index, bool verbose,
bool concurrent);
static void copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex,
- Snapshot snapshot, LogicalDecodingContext *decoding_ctx,
+ Snapshot snapshot,
bool verbose,
bool *pSwapToastByContent,
TransactionId *pFreezeXid,
@@ -158,12 +275,10 @@ static List *get_tables_to_repack_partitioned(RepackCommand cmd,
static bool cluster_is_permitted_for_relation(RepackCommand cmd,
Oid relid, Oid userid);
-static void begin_concurrent_repack(Relation rel);
-static void end_concurrent_repack(void);
static LogicalDecodingContext *setup_logical_decoding(Oid relid);
-static HeapTuple get_changed_tuple(char *change);
-static void apply_concurrent_changes(RepackDecodingState *dstate,
- ChangeDest *dest);
+static bool decode_concurrent_changes(LogicalDecodingContext *ctx,
+ DecodingWorkerShared *shared);
+static void apply_concurrent_changes(BufFile *file, ChangeDest *dest);
static void apply_concurrent_insert(Relation rel, HeapTuple tup,
IndexInsertState *iistate,
TupleTableSlot *index_slot);
@@ -175,9 +290,9 @@ static void apply_concurrent_delete(Relation rel, HeapTuple tup_target);
static HeapTuple find_target_tuple(Relation rel, ChangeDest *dest,
HeapTuple tup_key,
TupleTableSlot *ident_slot);
-static void process_concurrent_changes(LogicalDecodingContext *decoding_ctx,
- XLogRecPtr end_of_wal,
- ChangeDest *dest);
+static void process_concurrent_changes(XLogRecPtr end_of_wal,
+ ChangeDest *dest,
+ bool done);
static IndexInsertState *get_index_insert_state(Relation relation,
Oid ident_index_id,
Relation *ident_index_p);
@@ -187,7 +302,6 @@ static void free_index_insert_state(IndexInsertState *iistate);
static void cleanup_logical_decoding(LogicalDecodingContext *ctx);
static void rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
Relation cl_index,
- LogicalDecodingContext *decoding_ctx,
TransactionId frozenXid,
MultiXactId cutoffMulti);
static List *build_new_indexes(Relation NewHeap, Relation OldHeap, List *OldIndexes);
@@ -197,6 +311,13 @@ static Relation process_single_relation(RepackStmt *stmt,
ClusterParams *params);
static Oid determine_clustered_index(Relation rel, bool usingindex,
const char *indexname);
+static void start_decoding_worker(Oid relid);
+static void stop_decoding_worker(void);
+static void repack_worker_internal(dsm_segment *seg);
+static void export_initial_snapshot(Snapshot snapshot,
+ DecodingWorkerShared *shared);
+static Snapshot get_initial_snapshot(DecodingWorker *worker);
+static void ProcessRepackMessage(StringInfo msg);
static const char *RepackCommandAsString(RepackCommand cmd);
@@ -619,20 +740,20 @@ cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid,
/* rebuild_relation does all the dirty work */
PG_TRY();
{
- /*
- * For concurrent processing, make sure that our logical decoding
- * ignores data changes of other tables than the one we are
- * processing.
- */
- if (concurrent)
- begin_concurrent_repack(OldHeap);
-
rebuild_relation(OldHeap, index, verbose, concurrent);
}
PG_FINALLY();
{
if (concurrent)
- end_concurrent_repack();
+ {
+ /*
+ * Since during normal operation the worker was already asked to
+ * exit, stopping it explicitly is especially important on ERROR.
+ * However it still seems a good practice to make sure that the
+ * worker never survives the REPACK command.
+ */
+ stop_decoding_worker();
+ }
}
PG_END_TRY();
@@ -929,7 +1050,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, bool concurrent
bool swap_toast_by_content;
TransactionId frozenXid;
MultiXactId cutoffMulti;
- LogicalDecodingContext *decoding_ctx = NULL;
Snapshot snapshot = NULL;
#if USE_ASSERT_CHECKING
LOCKMODE lmode;
@@ -943,19 +1063,36 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, bool concurrent
if (concurrent)
{
/*
- * Prepare to capture the concurrent data changes.
+ * The worker needs to be member of the locking group we're the leader
+ * of. We ought to become the leader before the worker starts. The
+ * worker will join the group as soon as it starts.
*
- * Note that this call waits for all transactions with XID already
- * assigned to finish. If some of those transactions is waiting for a
- * lock conflicting with ShareUpdateExclusiveLock on our table (e.g.
- * it runs CREATE INDEX), we can end up in a deadlock. Not sure this
- * risk is worth unlocking/locking the table (and its clustering
- * index) and checking again if its still eligible for REPACK
- * CONCURRENTLY.
+ * This is to make sure that the deadlock described below is
+ * detectable by deadlock.c: if the worker waits for a transaction to
+ * complete and we are waiting for the worker output, then effectively
+ * we (i.e. this backend) are waiting for that transaction.
*/
- decoding_ctx = setup_logical_decoding(tableOid);
+ BecomeLockGroupLeader();
+
+ /*
+ * Start the worker that decodes data changes applied while we're
+ * copying the table contents.
+ *
+ * Note that the worker has to wait for all transactions with XID
+ * already assigned to finish. If some of those transactions is
+ * waiting for a lock conflicting with ShareUpdateExclusiveLock on our
+ * table (e.g. it runs CREATE INDEX), we can end up in a deadlock.
+ * Not sure this risk is worth unlocking/locking the table (and its
+ * clustering index) and checking again if its still eligible for
+ * REPACK CONCURRENTLY.
+ */
+ start_decoding_worker(tableOid);
+
+ /*
+ * Wait until the worker has the initial snapshot and retrieve it.
+ */
+ snapshot = get_initial_snapshot(decoding_worker);
- snapshot = SnapBuildInitialSnapshotForRepack(decoding_ctx->snapshot_builder);
PushActiveSnapshot(snapshot);
}
@@ -980,7 +1117,7 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, bool concurrent
NewHeap = table_open(OIDNewHeap, NoLock);
/* Copy the heap data into the new table in the desired order */
- copy_table_data(NewHeap, OldHeap, index, snapshot, decoding_ctx, verbose,
+ copy_table_data(NewHeap, OldHeap, index, snapshot, verbose,
&swap_toast_by_content, &frozenXid, &cutoffMulti);
/* The historic snapshot won't be needed anymore. */
@@ -994,14 +1131,10 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, bool concurrent
{
Assert(!swap_toast_by_content);
rebuild_relation_finish_concurrent(NewHeap, OldHeap, index,
- decoding_ctx,
frozenXid, cutoffMulti);
pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
PROGRESS_REPACK_PHASE_FINAL_CLEANUP);
-
- /* Done with decoding. */
- cleanup_logical_decoding(decoding_ctx);
}
else
{
@@ -1172,8 +1305,7 @@ make_new_heap(Oid OIDOldHeap, Oid NewTableSpace, Oid NewAccessMethod,
*/
static void
copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex,
- Snapshot snapshot, LogicalDecodingContext *decoding_ctx,
- bool verbose, bool *pSwapToastByContent,
+ Snapshot snapshot, bool verbose, bool *pSwapToastByContent,
TransactionId *pFreezeXid, MultiXactId *pCutoffMulti)
{
Relation relRelation;
@@ -1334,7 +1466,6 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex,
*/
table_relation_copy_for_cluster(OldHeap, NewHeap, OldIndex, use_sort,
cutoffs.OldestXmin, snapshot,
- decoding_ctx,
&cutoffs.FreezeLimit,
&cutoffs.MultiXactCutoff,
&num_tuples, &tups_vacuumed,
@@ -2367,59 +2498,6 @@ RepackCommandAsString(RepackCommand cmd)
return "???";
}
-
-/*
- * Call this function before REPACK CONCURRENTLY starts to setup logical
- * decoding. It makes sure that other users of the table put enough
- * information into WAL.
- *
- * The point is that at various places we expect that the table we're
- * processing is treated like a system catalog. For example, we need to be
- * able to scan it using a "historic snapshot" anytime during the processing
- * (as opposed to scanning only at the start point of the decoding, as logical
- * replication does during initial table synchronization), in order to apply
- * concurrent UPDATE / DELETE commands.
- *
- * Note that TOAST table needs no attention here as it's not scanned using
- * historic snapshot.
- */
-static void
-begin_concurrent_repack(Relation rel)
-{
- Oid toastrelid;
-
- /*
- * Avoid logical decoding of other relations by this backend. The lock we
- * have guarantees that the actual locator cannot be changed concurrently:
- * TRUNCATE needs AccessExclusiveLock.
- */
- Assert(CheckRelationLockedByMe(rel, ShareUpdateExclusiveLock, false));
- repacked_rel_locator = rel->rd_locator;
- toastrelid = rel->rd_rel->reltoastrelid;
- if (OidIsValid(toastrelid))
- {
- Relation toastrel;
-
- /* Avoid logical decoding of other TOAST relations. */
- toastrel = table_open(toastrelid, AccessShareLock);
- repacked_rel_toast_locator = toastrel->rd_locator;
- table_close(toastrel, AccessShareLock);
- }
-}
-
-/*
- * Call this when done with REPACK CONCURRENTLY.
- */
-static void
-end_concurrent_repack(void)
-{
- /*
- * Restore normal function of (future) logical decoding for this backend.
- */
- repacked_rel_locator.relNumber = InvalidOid;
- repacked_rel_toast_locator.relNumber = InvalidOid;
-}
-
/*
* Is this backend performing logical decoding on behalf of REPACK
* (CONCURRENTLY) ?
@@ -2484,9 +2562,10 @@ static LogicalDecodingContext *
setup_logical_decoding(Oid relid)
{
Relation rel;
- TupleDesc tupdesc;
+ Oid toastrelid;
LogicalDecodingContext *ctx;
- RepackDecodingState *dstate = palloc0_object(RepackDecodingState);
+ NameData slotname;
+ RepackDecodingState *dstate;
/*
* REPACK CONCURRENTLY is not allowed in a transaction block, so this
@@ -2494,21 +2573,21 @@ setup_logical_decoding(Oid relid)
*/
Assert(!TransactionIdIsValid(GetTopTransactionIdIfAny()));
- /*
- * A single backend should not execute multiple REPACK commands at a time,
- * so use PID to make the slot unique.
- */
- snprintf(NameStr(dstate->slotname), NAMEDATALEN, "repack_%d", MyProcPid);
-
/*
* Check if we can use logical decoding.
*/
CheckSlotPermissions();
CheckLogicalDecodingRequirements();
- /* RS_TEMPORARY so that the slot gets cleaned up on ERROR. */
- ReplicationSlotCreate(NameStr(dstate->slotname), true, RS_TEMPORARY,
- false, false, false);
+ /*
+ * A single backend should not execute multiple REPACK commands at a time,
+ * so use PID to make the slot unique.
+ *
+ * RS_TEMPORARY so that the slot gets cleaned up on ERROR.
+ */
+ snprintf(NameStr(slotname), NAMEDATALEN, "repack_%d", MyProcPid);
+ ReplicationSlotCreate(NameStr(slotname), true, RS_TEMPORARY, false, false,
+ false);
/*
* Neither prepare_write nor do_write callback nor update_progress is
@@ -2530,104 +2609,109 @@ setup_logical_decoding(Oid relid)
DecodingContextFindStartpoint(ctx);
+ /*
+ * decode_concurrent_changes() needs non-blocking callback.
+ */
+ ctx->reader->routine.page_read = read_local_xlog_page_no_wait;
+
+ /*
+ * read_local_xlog_page_no_wait() needs to be able to indicate the end of
+ * WAL.
+ */
+ ctx->reader->private_data = MemoryContextAllocZero(ctx->context,
+ sizeof(ReadLocalXLogPageNoWaitPrivate));
+
+
/* Some WAL records should have been read. */
Assert(ctx->reader->EndRecPtr != InvalidXLogRecPtr);
+ /*
+ * Initialize repack_current_segment so that we can notice WAL segment
+ * boundaries.
+ */
XLByteToSeg(ctx->reader->EndRecPtr, repack_current_segment,
wal_segment_size);
- /*
- * Setup structures to store decoded changes.
- */
+ dstate = palloc0_object(RepackDecodingState);
dstate->relid = relid;
- dstate->tstore = tuplestore_begin_heap(false, false,
- maintenance_work_mem);
- /* Caller should already have the table locked. */
- rel = table_open(relid, NoLock);
- tupdesc = CreateTupleDescCopy(RelationGetDescr(rel));
- dstate->tupdesc = tupdesc;
- table_close(rel, NoLock);
+ /*
+ * Tuple descriptor may be needed to flatten a tuple before we write it to
+ * a file. A copy is needed because the decoding worker invalidates system
+ * caches before it starts to do the actual work.
+ */
+ rel = table_open(relid, AccessShareLock);
+ dstate->tupdesc = CreateTupleDescCopy(RelationGetDescr(rel));
- /* Initialize the descriptor to store the changes ... */
- dstate->tupdesc_change = CreateTemplateTupleDesc(1);
+ /* Avoid logical decoding of other relations. */
+ repacked_rel_locator = rel->rd_locator;
+ toastrelid = rel->rd_rel->reltoastrelid;
+ if (OidIsValid(toastrelid))
+ {
+ Relation toastrel;
- TupleDescInitEntry(dstate->tupdesc_change, 1, NULL, BYTEAOID, -1, 0);
- /* ... as well as the corresponding slot. */
- dstate->tsslot = MakeSingleTupleTableSlot(dstate->tupdesc_change,
- &TTSOpsMinimalTuple);
+ /* Avoid logical decoding of other TOAST relations. */
+ toastrel = table_open(toastrelid, AccessShareLock);
+ repacked_rel_toast_locator = toastrel->rd_locator;
+ table_close(toastrel, AccessShareLock);
+ }
+ table_close(rel, AccessShareLock);
- dstate->resowner = ResourceOwnerCreate(CurrentResourceOwner,
- "logical decoding");
+ /* The file will be set as soon as we have it opened. */
+ dstate->file = NULL;
ctx->output_writer_private = dstate;
+
return ctx;
}
/*
- * Retrieve tuple from ConcurrentChange structure.
+ * Decode logical changes from the WAL sequence and store them to a file.
*
- * The input data starts with the structure but it might not be appropriately
- * aligned.
- */
-static HeapTuple
-get_changed_tuple(char *change)
-{
- HeapTupleData tup_data;
- HeapTuple result;
- char *src;
-
- /*
- * Ensure alignment before accessing the fields. (This is why we can't use
- * heap_copytuple() instead of this function.)
- */
- src = change + offsetof(ConcurrentChange, tup_data);
- memcpy(&tup_data, src, sizeof(HeapTupleData));
-
- result = (HeapTuple) palloc(HEAPTUPLESIZE + tup_data.t_len);
- memcpy(result, &tup_data, sizeof(HeapTupleData));
- result->t_data = (HeapTupleHeader) ((char *) result + HEAPTUPLESIZE);
- src = change + SizeOfConcurrentChange;
- memcpy(result->t_data, src, result->t_len);
-
- return result;
-}
-
-/*
- * Decode logical changes from the WAL sequence up to end_of_wal.
+ * If true is returned, there is no more work for the worker.
*/
-void
-repack_decode_concurrent_changes(LogicalDecodingContext *ctx,
- XLogRecPtr end_of_wal)
+static bool
+decode_concurrent_changes(LogicalDecodingContext *ctx,
+ DecodingWorkerShared *shared)
{
RepackDecodingState *dstate;
- ResourceOwner resowner_old;
+ XLogRecPtr lsn_upto;
+ bool done;
+ char fname[MAXPGPATH];
dstate = (RepackDecodingState *) ctx->output_writer_private;
- resowner_old = CurrentResourceOwner;
- CurrentResourceOwner = dstate->resowner;
- PG_TRY();
+ /* Open the output file. */
+ DecodingWorkerFileName(fname, shared->relid, shared->last_exported + 1);
+ dstate->file = BufFileCreateFileSet(&shared->sfs.fs, fname);
+
+ SpinLockAcquire(&shared->mutex);
+ lsn_upto = shared->lsn_upto;
+ done = shared->done;
+ SpinLockRelease(&shared->mutex);
+
+ while (true)
{
- while (ctx->reader->EndRecPtr < end_of_wal)
- {
- XLogRecord *record;
- XLogSegNo segno_new;
- char *errm = NULL;
- XLogRecPtr end_lsn;
+ XLogRecord *record;
+ XLogSegNo segno_new;
+ char *errm = NULL;
+ XLogRecPtr end_lsn;
- record = XLogReadRecord(ctx->reader, &errm);
- if (errm)
- elog(ERROR, "%s", errm);
+ CHECK_FOR_INTERRUPTS();
- if (record != NULL)
- LogicalDecodingProcessRecord(ctx, ctx->reader);
+ record = XLogReadRecord(ctx->reader, &errm);
+ if (record)
+ {
+ LogicalDecodingProcessRecord(ctx, ctx->reader);
/*
* If WAL segment boundary has been crossed, inform the decoding
- * system that the catalog_xmin can advance. (We can confirm more
- * often, but a filling a single WAL segment should not take much
- * time.)
+ * system that the catalog_xmin can advance.
+ *
+ * TODO Does it make sense to confirm more often? Segment size
+ * seems appropriate for restart_lsn (because less than a segment
+ * cannot be recycled anyway), however more frequent checks might
+ * be beneficial for catalog_xmin.
*/
end_lsn = ctx->reader->EndRecPtr;
XLByteToSeg(end_lsn, segno_new, wal_segment_size);
@@ -2638,80 +2722,137 @@ repack_decode_concurrent_changes(LogicalDecodingContext *ctx,
(uint32) (end_lsn >> 32), (uint32) end_lsn);
repack_current_segment = segno_new;
}
+ }
+ else
+ {
+ ReadLocalXLogPageNoWaitPrivate *priv;
- CHECK_FOR_INTERRUPTS();
+ if (errm)
+ ereport(ERROR, (errmsg("%s", errm)));
+
+ /*
+ * In the decoding loop we do not want to get blocked when there
+ * is no more WAL available, otherwise the loop would become
+ * uninterruptible.
+ */
+ priv = (ReadLocalXLogPageNoWaitPrivate *)
+ ctx->reader->private_data;
+ if (priv->end_of_wal)
+ /* Do not miss the end of WAL condition next time. */
+ priv->end_of_wal = false;
+ else
+ ereport(ERROR, (errmsg("could not read WAL record")));
+ }
+
+ /*
+ * Whether we could read new record or not, keep checking if
+ * 'lsn_upto' was specified.
+ */
+ if (XLogRecPtrIsInvalid(lsn_upto))
+ {
+ SpinLockAcquire(&shared->mutex);
+ lsn_upto = shared->lsn_upto;
+ /* 'done' should be set at the same time as 'lsn_upto' */
+ done = shared->done;
+ SpinLockRelease(&shared->mutex);
+ }
+ if (!XLogRecPtrIsInvalid(lsn_upto) &&
+ ctx->reader->EndRecPtr >= lsn_upto)
+ break;
+
+ if (record == NULL)
+ {
+ int64 timeout = 0;
+ WaitLSNResult res;
+
+ /*
+ * Before we retry reading, wait until new WAL is flushed.
+ *
+ * There is a race condition such that the backend executing
+ * REPACK determines 'lsn_upto', but before it sets the shared
+ * variable, we reach the end of WAL. In that case we'd need to
+ * wait until the next WAL flush (unrelated to REPACK). Although
+ * that should not be a problem in a busy system, it might be
+ * noticeable in other cases, including regression tests (which
+ * are not necessarily executed in parallel). Therefore it makes
+ * sense to use timeout.
+ *
+ * If lsn_upto is valid, WAL records having LSN lower than that
+ * should already have been flushed to disk.
+ */
+ if (XLogRecPtrIsInvalid(lsn_upto))
+ timeout = 100L;
+ res = WaitForLSN(WAIT_LSN_TYPE_PRIMARY_FLUSH,
+ ctx->reader->EndRecPtr + 1,
+ timeout);
+ if (res != WAIT_LSN_RESULT_SUCCESS &&
+ res != WAIT_LSN_RESULT_TIMEOUT)
+ ereport(ERROR, (errmsg("waiting for WAL failed")));
}
- InvalidateSystemCaches();
- CurrentResourceOwner = resowner_old;
- }
- PG_CATCH();
- {
- /* clear all timetravel entries */
- InvalidateSystemCaches();
- CurrentResourceOwner = resowner_old;
- PG_RE_THROW();
}
- PG_END_TRY();
+
+ /*
+ * Close the file so we can make it available to the backend.
+ */
+ BufFileClose(dstate->file);
+ dstate->file = NULL;
+ SpinLockAcquire(&shared->mutex);
+ shared->lsn_upto = InvalidXLogRecPtr;
+ shared->last_exported++;
+ SpinLockRelease(&shared->mutex);
+ ConditionVariableSignal(&shared->cv);
+
+ return done;
}
/*
* Apply changes stored in 'file'.
*/
static void
-apply_concurrent_changes(RepackDecodingState *dstate, ChangeDest *dest)
+apply_concurrent_changes(BufFile *file, ChangeDest *dest)
{
+ char kind;
+ uint32 t_len;
Relation rel = dest->rel;
TupleTableSlot *index_slot,
*ident_slot;
HeapTuple tup_old = NULL;
- if (dstate->nchanges == 0)
- return;
-
/* TupleTableSlot is needed to pass the tuple to ExecInsertIndexTuples(). */
- index_slot = MakeSingleTupleTableSlot(dstate->tupdesc, &TTSOpsHeapTuple);
+ index_slot = MakeSingleTupleTableSlot(RelationGetDescr(rel),
+ &TTSOpsHeapTuple);
/* A slot to fetch tuples from identity index. */
ident_slot = table_slot_create(rel, NULL);
- while (tuplestore_gettupleslot(dstate->tstore, true, false,
- dstate->tsslot))
+ while (true)
{
- bool shouldFree;
- HeapTuple tup_change,
- tup,
+ size_t nread;
+ HeapTuple tup,
tup_exist;
- char *change_raw,
- *src;
- ConcurrentChange change;
- bool isnull[1];
- Datum values[1];
CHECK_FOR_INTERRUPTS();
- /* Get the change from the single-column tuple. */
- tup_change = ExecFetchSlotHeapTuple(dstate->tsslot, false, &shouldFree);
- heap_deform_tuple(tup_change, dstate->tupdesc_change, values, isnull);
- Assert(!isnull[0]);
-
- /* Make sure we access aligned data. */
- change_raw = (char *) DatumGetByteaP(values[0]);
- src = (char *) VARDATA(change_raw);
- memcpy(&change, src, SizeOfConcurrentChange);
+ nread = BufFileReadMaybeEOF(file, &kind, 1, true);
+ /* Are we done with the file? */
+ if (nread == 0)
+ break;
- /*
- * Extract the tuple from the change. The tuple is copied here because
- * it might be assigned to 'tup_old', in which case it needs to
- * survive into the next iteration.
- */
- tup = get_changed_tuple(src);
+ /* Read the tuple. */
+ BufFileReadExact(file, &t_len, sizeof(t_len));
+ tup = (HeapTuple) palloc(HEAPTUPLESIZE + t_len);
+ tup->t_data = (HeapTupleHeader) ((char *) tup + HEAPTUPLESIZE);
+ BufFileReadExact(file, tup->t_data, t_len);
+ tup->t_len = t_len;
+ ItemPointerSetInvalid(&tup->t_self);
+ tup->t_tableOid = RelationGetRelid(dest->rel);
- if (change.kind == CHANGE_UPDATE_OLD)
+ if (kind == CHANGE_UPDATE_OLD)
{
Assert(tup_old == NULL);
tup_old = tup;
}
- else if (change.kind == CHANGE_INSERT)
+ else if (kind == CHANGE_INSERT)
{
Assert(tup_old == NULL);
@@ -2719,12 +2860,11 @@ apply_concurrent_changes(RepackDecodingState *dstate, ChangeDest *dest)
pfree(tup);
}
- else if (change.kind == CHANGE_UPDATE_NEW ||
- change.kind == CHANGE_DELETE)
+ else if (kind == CHANGE_UPDATE_NEW || kind == CHANGE_DELETE)
{
HeapTuple tup_key;
- if (change.kind == CHANGE_UPDATE_NEW)
+ if (kind == CHANGE_UPDATE_NEW)
{
tup_key = tup_old != NULL ? tup_old : tup;
}
@@ -2741,7 +2881,7 @@ apply_concurrent_changes(RepackDecodingState *dstate, ChangeDest *dest)
if (tup_exist == NULL)
elog(ERROR, "failed to find target tuple");
- if (change.kind == CHANGE_UPDATE_NEW)
+ if (kind == CHANGE_UPDATE_NEW)
apply_concurrent_update(rel, tup, tup_exist, dest->iistate,
index_slot);
else
@@ -2756,26 +2896,19 @@ apply_concurrent_changes(RepackDecodingState *dstate, ChangeDest *dest)
pfree(tup);
}
else
- elog(ERROR, "unrecognized kind of change: %d", change.kind);
+ elog(ERROR, "unrecognized kind of change: %d", kind);
/*
* If a change was applied now, increment CID for next writes and
* update the snapshot so it sees the changes we've applied so far.
*/
- if (change.kind != CHANGE_UPDATE_OLD)
+ if (kind != CHANGE_UPDATE_OLD)
{
CommandCounterIncrement();
UpdateActiveSnapshotCommandId();
}
-
- /* TTSOpsMinimalTuple has .get_heap_tuple==NULL. */
- Assert(shouldFree);
- pfree(tup_change);
}
- tuplestore_clear(dstate->tstore);
- dstate->nchanges = 0;
-
/* Cleanup. */
ExecDropSingleTupleTableSlot(index_slot);
ExecDropSingleTupleTableSlot(ident_slot);
@@ -2954,25 +3087,59 @@ find_target_tuple(Relation rel, ChangeDest *dest, HeapTuple tup_key,
}
/*
- * Decode and apply concurrent changes.
+ * Decode and apply concurrent changes, up to (and including) the record whose
+ * LSN is 'end_of_wal'.
*/
static void
-process_concurrent_changes(LogicalDecodingContext *decoding_ctx,
- XLogRecPtr end_of_wal, ChangeDest *dest)
+process_concurrent_changes(XLogRecPtr end_of_wal, ChangeDest *dest, bool done)
{
- RepackDecodingState *dstate;
+ DecodingWorkerShared *shared;
+ char fname[MAXPGPATH];
+ BufFile *file;
pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
PROGRESS_REPACK_PHASE_CATCH_UP);
- dstate = (RepackDecodingState *) decoding_ctx->output_writer_private;
+ /* Ask the worker for the file. */
+ shared = (DecodingWorkerShared *) dsm_segment_address(decoding_worker->seg);
+ SpinLockAcquire(&shared->mutex);
+ shared->lsn_upto = end_of_wal;
+ shared->done = done;
+ SpinLockRelease(&shared->mutex);
- repack_decode_concurrent_changes(decoding_ctx, end_of_wal);
+ /*
+ * The worker needs to finish processing of the current WAL record. Even
+ * if it's idle, it'll need to close the output file. Thus we're likely to
+ * wait, so prepare for sleep.
+ */
+ ConditionVariablePrepareToSleep(&shared->cv);
+ for (;;)
+ {
+ int last_exported;
- if (dstate->nchanges == 0)
- return;
+ SpinLockAcquire(&shared->mutex);
+ last_exported = shared->last_exported;
+ SpinLockRelease(&shared->mutex);
+
+ /*
+ * Has the worker exported the file we are waiting for?
+ */
+ if (last_exported == dest->file_seq)
+ break;
+
+ ConditionVariableSleep(&shared->cv, WAIT_EVENT_REPACK_WORKER_EXPORT);
+ }
+ ConditionVariableCancelSleep();
- apply_concurrent_changes(dstate, dest);
+ /* Open the file. */
+ DecodingWorkerFileName(fname, shared->relid, dest->file_seq);
+ file = BufFileOpenFileSet(&shared->sfs.fs, fname, O_RDONLY, false);
+ apply_concurrent_changes(file, dest);
+
+ BufFileClose(file);
+
+ /* Get ready for the next file. */
+ dest->file_seq++;
}
/*
@@ -3098,15 +3265,10 @@ cleanup_logical_decoding(LogicalDecodingContext *ctx)
dstate = (RepackDecodingState *) ctx->output_writer_private;
- ExecDropSingleTupleTableSlot(dstate->tsslot);
- FreeTupleDesc(dstate->tupdesc_change);
FreeTupleDesc(dstate->tupdesc);
- tuplestore_end(dstate->tstore);
-
FreeDecodingContext(ctx);
- ReplicationSlotRelease();
- ReplicationSlotDrop(NameStr(dstate->slotname), false);
+ ReplicationSlotDropAcquired();
pfree(dstate);
}
@@ -3121,7 +3283,6 @@ cleanup_logical_decoding(LogicalDecodingContext *ctx)
static void
rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
Relation cl_index,
- LogicalDecodingContext *decoding_ctx,
TransactionId frozenXid,
MultiXactId cutoffMulti)
{
@@ -3204,6 +3365,7 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
&chgdst.ident_index);
chgdst.ident_key = build_identity_key(ident_idx_new, OldHeap,
&chgdst.ident_key_nentries);
+ chgdst.file_seq = WORKER_FILE_SNAPSHOT + 1;
/*
* During testing, wait for another backend to perform concurrent data
@@ -3225,7 +3387,7 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
* hold AccessExclusiveLock. (Quite some amount of WAL could have been
* written during the data copying and index creation.)
*/
- process_concurrent_changes(decoding_ctx, end_of_wal, &chgdst);
+ process_concurrent_changes(end_of_wal, &chgdst, false);
/*
* Acquire AccessExclusiveLock on the table, its TOAST relation (if there
@@ -3319,8 +3481,11 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
XLogFlush(wal_insert_ptr);
end_of_wal = GetFlushRecPtr(NULL);
- /* Apply the concurrent changes again. */
- process_concurrent_changes(decoding_ctx, end_of_wal, &chgdst);
+ /*
+ * Apply the concurrent changes again. Indicate that the decoding worker
+ * won't be needed anymore.
+ */
+ process_concurrent_changes(end_of_wal, &chgdst, true);
/* Remember info about rel before closing OldHeap */
relpersistence = OldHeap->rd_rel->relpersistence;
@@ -3430,3 +3595,510 @@ build_new_indexes(Relation NewHeap, Relation OldHeap, List *OldIndexes)
return result;
}
+
+/*
+ * Try to start a background worker to perform logical decoding of data
+ * changes applied to relation while REPACK CONCURRENTLY is copying its
+ * contents to a new table.
+ */
+static void
+start_decoding_worker(Oid relid)
+{
+ Size size;
+ dsm_segment *seg;
+ DecodingWorkerShared *shared;
+ shm_mq *mq;
+ shm_mq_handle *mqh;
+ BackgroundWorker bgw;
+
+ /* Setup shared memory. */
+ size = BUFFERALIGN(offsetof(DecodingWorkerShared, error_queue)) +
+ BUFFERALIGN(REPACK_ERROR_QUEUE_SIZE);
+ seg = dsm_create(size, 0);
+ shared = (DecodingWorkerShared *) dsm_segment_address(seg);
+ shared->lsn_upto = InvalidXLogRecPtr;
+ shared->done = false;
+ SharedFileSetInit(&shared->sfs, seg);
+ shared->last_exported = -1;
+ SpinLockInit(&shared->mutex);
+ shared->dbid = MyDatabaseId;
+
+ /*
+ * This is the UserId set in cluster_rel(). Security context shouldn't be
+ * needed for decoding worker.
+ */
+ shared->roleid = GetUserId();
+ shared->relid = relid;
+ ConditionVariableInit(&shared->cv);
+ shared->backend_proc = MyProc;
+ shared->backend_pid = MyProcPid;
+ shared->backend_proc_number = MyProcNumber;
+
+ mq = shm_mq_create((char *) BUFFERALIGN(shared->error_queue),
+ REPACK_ERROR_QUEUE_SIZE);
+ shm_mq_set_receiver(mq, MyProc);
+ mqh = shm_mq_attach(mq, seg, NULL);
+
+ memset(&bgw, 0, sizeof(bgw));
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "REPACK decoding worker for relation \"%s\"",
+ get_rel_name(relid));
+ snprintf(bgw.bgw_type, BGW_MAXLEN, "REPACK decoding worker");
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_restart_time = BGW_NEVER_RESTART;
+ snprintf(bgw.bgw_library_name, MAXPGPATH, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "RepackWorkerMain");
+ bgw.bgw_main_arg = UInt32GetDatum(dsm_segment_handle(seg));
+ bgw.bgw_notify_pid = MyProcPid;
+
+ decoding_worker = palloc0_object(DecodingWorker);
+ if (!RegisterDynamicBackgroundWorker(&bgw, &decoding_worker->handle))
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg("out of background worker slots"),
+ errhint("You might need to increase \"%s\".", "max_worker_processes")));
+
+ decoding_worker->seg = seg;
+ decoding_worker->error_mqh = mqh;
+
+ /*
+ * The decoding setup must be done before the caller can have XID assigned
+ * for any reason, otherwise the worker might end up in a deadlock,
+ * waiting for the caller's transaction to end. Therefore wait here until
+ * the worker indicates that it has the logical decoding initialized.
+ */
+ ConditionVariablePrepareToSleep(&shared->cv);
+ for (;;)
+ {
+ int initialized;
+
+ SpinLockAcquire(&shared->mutex);
+ initialized = shared->initialized;
+ SpinLockRelease(&shared->mutex);
+
+ if (initialized)
+ break;
+
+ ConditionVariableSleep(&shared->cv, WAIT_EVENT_REPACK_WORKER_EXPORT);
+ }
+ ConditionVariableCancelSleep();
+}
+
+/*
+ * Stop the decoding worker and cleanup the related resources.
+ *
+ * The worker stops on its own when it knows there is no more work to do, but
+ * we need to stop it explicitly at least on ERROR in the launching backend.
+ */
+static void
+stop_decoding_worker(void)
+{
+ BgwHandleStatus status;
+
+ /* Haven't reached the worker startup? */
+ if (decoding_worker == NULL)
+ return;
+
+ /* Could not register the worker? */
+ if (decoding_worker->handle == NULL)
+ return;
+
+ TerminateBackgroundWorker(decoding_worker->handle);
+ /* The worker should really exit before the REPACK command does. */
+ HOLD_INTERRUPTS();
+ status = WaitForBackgroundWorkerShutdown(decoding_worker->handle);
+ RESUME_INTERRUPTS();
+
+ if (status == BGWH_POSTMASTER_DIED)
+ ereport(FATAL,
+ (errcode(ERRCODE_ADMIN_SHUTDOWN),
+ errmsg("postmaster exited during REPACK command")));
+
+ shm_mq_detach(decoding_worker->error_mqh);
+
+ /*
+ * If we could not cancel the current sleep due to ERROR, do that before
+ * we detach from the shared memory the condition variable is located in.
+ * If we did not, the bgworker ERROR handling code would try and fail
+ * badly.
+ */
+ ConditionVariableCancelSleep();
+
+ dsm_detach(decoding_worker->seg);
+ pfree(decoding_worker);
+ decoding_worker = NULL;
+}
+
+/* Is this process a REPACK worker? */
+static bool is_repack_worker = false;
+
+static pid_t backend_pid;
+static ProcNumber backend_proc_number;
+
+/*
+ * See ParallelWorkerShutdown for details.
+ */
+static void
+RepackWorkerShutdown(int code, Datum arg)
+{
+ SendProcSignal(backend_pid,
+ PROCSIG_REPACK_MESSAGE,
+ backend_proc_number);
+
+ dsm_detach((dsm_segment *) DatumGetPointer(arg));
+}
+
+/* REPACK decoding worker entry point */
+void
+RepackWorkerMain(Datum main_arg)
+{
+ dsm_segment *seg;
+ DecodingWorkerShared *shared;
+ shm_mq *mq;
+ shm_mq_handle *mqh;
+
+ is_repack_worker = true;
+
+ /*
+ * Override the default bgworker_die() with die() so we can use
+ * CHECK_FOR_INTERRUPTS().
+ */
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ seg = dsm_attach(DatumGetUInt32(main_arg));
+ if (seg == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("could not map dynamic shared memory segment")));
+
+ shared = (DecodingWorkerShared *) dsm_segment_address(seg);
+
+ /* Arrange to signal the leader if we exit. */
+ backend_pid = shared->backend_pid;
+ backend_proc_number = shared->backend_proc_number;
+ before_shmem_exit(RepackWorkerShutdown, PointerGetDatum(seg));
+
+ /*
+ * Join locking group - see the comments around the call of
+ * start_decoding_worker().
+ */
+ if (!BecomeLockGroupMember(shared->backend_proc, backend_pid))
+ /* The leader is not running anymore. */
+ return;
+
+ /*
+ * Setup a queue to send error messages to the backend that launched this
+ * worker.
+ */
+ mq = (shm_mq *) (char *) BUFFERALIGN(shared->error_queue);
+ shm_mq_set_sender(mq, MyProc);
+ mqh = shm_mq_attach(mq, seg, NULL);
+ pq_redirect_to_shm_mq(seg, mqh);
+ pq_set_parallel_leader(shared->backend_pid,
+ shared->backend_proc_number);
+
+ /* Connect to the database. */
+ BackgroundWorkerInitializeConnectionByOid(shared->dbid, shared->roleid, 0);
+
+ repack_worker_internal(seg);
+}
+
+static void
+repack_worker_internal(dsm_segment *seg)
+{
+ DecodingWorkerShared *shared;
+ LogicalDecodingContext *decoding_ctx;
+ SharedFileSet *sfs;
+ Snapshot snapshot;
+
+ /*
+ * Transaction is needed to open relation, and it also provides us with a
+ * resource owner.
+ */
+ StartTransactionCommand();
+
+ shared = (DecodingWorkerShared *) dsm_segment_address(seg);
+
+ /*
+ * Not sure the spinlock is needed here - the backend should not change
+ * anything in the shared memory until we have serialized the snapshot.
+ */
+ SpinLockAcquire(&shared->mutex);
+ Assert(XLogRecPtrIsInvalid(shared->lsn_upto));
+ sfs = &shared->sfs;
+ SpinLockRelease(&shared->mutex);
+
+ SharedFileSetAttach(sfs, seg);
+
+ /*
+ * Prepare to capture the concurrent data changes ourselves.
+ */
+ decoding_ctx = setup_logical_decoding(shared->relid);
+
+ /* Announce that we're ready. */
+ SpinLockAcquire(&shared->mutex);
+ shared->initialized = true;
+ SpinLockRelease(&shared->mutex);
+ ConditionVariableSignal(&shared->cv);
+
+ /* Build the initial snapshot and export it. */
+ snapshot = SnapBuildInitialSnapshotForRepack(decoding_ctx->snapshot_builder);
+ export_initial_snapshot(snapshot, shared);
+
+ /*
+ * Only historic snapshots should be used now. Do not let us restrict the
+ * progress of xmin horizon.
+ */
+ InvalidateCatalogSnapshot();
+
+ while (!decode_concurrent_changes(decoding_ctx, shared))
+ ;
+
+ /* Cleanup. */
+ cleanup_logical_decoding(decoding_ctx);
+ CommitTransactionCommand();
+}
+
+/*
+ * Make snapshot available to the backend that launched the decoding worker.
+ */
+static void
+export_initial_snapshot(Snapshot snapshot, DecodingWorkerShared *shared)
+{
+ char fname[MAXPGPATH];
+ BufFile *file;
+ Size snap_size;
+ char *snap_space;
+
+ snap_size = EstimateSnapshotSpace(snapshot);
+ snap_space = (char *) palloc(snap_size);
+ SerializeSnapshot(snapshot, snap_space);
+ FreeSnapshot(snapshot);
+
+ DecodingWorkerFileName(fname, shared->relid, shared->last_exported + 1);
+ file = BufFileCreateFileSet(&shared->sfs.fs, fname);
+ /* To make restoration easier, write the snapshot size first. */
+ BufFileWrite(file, &snap_size, sizeof(snap_size));
+ BufFileWrite(file, snap_space, snap_size);
+ pfree(snap_space);
+ BufFileClose(file);
+
+ /* Increase the counter to tell the backend that the file is available. */
+ SpinLockAcquire(&shared->mutex);
+ shared->last_exported++;
+ SpinLockRelease(&shared->mutex);
+ ConditionVariableSignal(&shared->cv);
+}
+
+/*
+ * Get the initial snapshot from the decoding worker.
+ */
+static Snapshot
+get_initial_snapshot(DecodingWorker *worker)
+{
+ DecodingWorkerShared *shared;
+ char fname[MAXPGPATH];
+ BufFile *file;
+ Size snap_size;
+ char *snap_space;
+ Snapshot snapshot;
+
+ shared = (DecodingWorkerShared *) dsm_segment_address(worker->seg);
+
+ /*
+ * The worker needs to initialize the logical decoding, which usually
+ * takes some time. Therefore it makes sense to prepare for the sleep
+ * first.
+ */
+ ConditionVariablePrepareToSleep(&shared->cv);
+ for (;;)
+ {
+ int last_exported;
+
+ SpinLockAcquire(&shared->mutex);
+ last_exported = shared->last_exported;
+ SpinLockRelease(&shared->mutex);
+
+ /*
+ * Has the worker exported the file we are waiting for?
+ */
+ if (last_exported == WORKER_FILE_SNAPSHOT)
+ break;
+
+ ConditionVariableSleep(&shared->cv, WAIT_EVENT_REPACK_WORKER_EXPORT);
+ }
+ ConditionVariableCancelSleep();
+
+ /* Read the snapshot from a file. */
+ DecodingWorkerFileName(fname, shared->relid, WORKER_FILE_SNAPSHOT);
+ file = BufFileOpenFileSet(&shared->sfs.fs, fname, O_RDONLY, false);
+ BufFileReadExact(file, &snap_size, sizeof(snap_size));
+ snap_space = (char *) palloc(snap_size);
+ BufFileReadExact(file, snap_space, snap_size);
+ BufFileClose(file);
+
+ /* Restore it. */
+ snapshot = RestoreSnapshot(snap_space);
+ pfree(snap_space);
+
+ return snapshot;
+}
+
+bool
+IsRepackWorker(void)
+{
+ return is_repack_worker;
+}
+
+/*
+ * Handle receipt of an interrupt indicating a repack worker message.
+ *
+ * Note: this is called within a signal handler! All we can do is set
+ * a flag that will cause the next CHECK_FOR_INTERRUPTS() to invoke
+ * ProcessRepackMessages().
+ */
+void
+HandleRepackMessageInterrupt(void)
+{
+ InterruptPending = true;
+ RepackMessagePending = true;
+ SetLatch(MyLatch);
+}
+
+/*
+ * Process any queued protocol messages received from parallel workers.
+ */
+void
+ProcessRepackMessages(void)
+{
+ MemoryContext oldcontext;
+
+ static MemoryContext hpm_context = NULL;
+
+ /*
+ * Nothing to do if we haven't launched the worker yet or have already
+ * terminated it.
+ */
+ if (decoding_worker == NULL)
+ return;
+
+ /*
+ * This is invoked from ProcessInterrupts(), and since some of the
+ * functions it calls contain CHECK_FOR_INTERRUPTS(), there is a potential
+ * for recursive calls if more signals are received while this runs. It's
+ * unclear that recursive entry would be safe, and it doesn't seem useful
+ * even if it is safe, so let's block interrupts until done.
+ */
+ HOLD_INTERRUPTS();
+
+ /*
+ * Moreover, CurrentMemoryContext might be pointing almost anywhere. We
+ * don't want to risk leaking data into long-lived contexts, so let's do
+ * our work here in a private context that we can reset on each use.
+ */
+ if (hpm_context == NULL) /* first time through? */
+ hpm_context = AllocSetContextCreate(TopMemoryContext,
+ "ProcessRepackMessages",
+ ALLOCSET_DEFAULT_SIZES);
+ else
+ MemoryContextReset(hpm_context);
+
+ oldcontext = MemoryContextSwitchTo(hpm_context);
+
+ /* OK to process messages. Reset the flag saying there are more to do. */
+ RepackMessagePending = false;
+
+ /*
+ * Read as many messages as we can from each worker, but stop when no more
+ * messages can be read from the worker without blocking.
+ */
+ while (true)
+ {
+ shm_mq_result res;
+ Size nbytes;
+ void *data;
+
+ res = shm_mq_receive(decoding_worker->error_mqh, &nbytes,
+ &data, true);
+ if (res == SHM_MQ_WOULD_BLOCK)
+ break;
+ else if (res == SHM_MQ_SUCCESS)
+ {
+ StringInfoData msg;
+
+ initStringInfo(&msg);
+ appendBinaryStringInfo(&msg, data, nbytes);
+ ProcessRepackMessage(&msg);
+ pfree(msg.data);
+ }
+ else
+ {
+ /*
+ * The decoding worker is special in that it exits as soon as it
+ * has its work done. Thus the DETACHED result code is fine.
+ */
+ Assert(res == SHM_MQ_DETACHED);
+
+ break;
+ }
+ }
+
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Might as well clear the context on our way out */
+ MemoryContextReset(hpm_context);
+
+ RESUME_INTERRUPTS();
+}
+
+/*
+ * Process a single protocol message received from a single parallel worker.
+ */
+static void
+ProcessRepackMessage(StringInfo msg)
+{
+ char msgtype;
+
+ msgtype = pq_getmsgbyte(msg);
+
+ switch (msgtype)
+ {
+ case PqMsg_ErrorResponse:
+ case PqMsg_NoticeResponse:
+ {
+ ErrorData edata;
+
+ /* Parse ErrorResponse or NoticeResponse. */
+ pq_parse_errornotice(msg, &edata);
+
+ /* Death of a worker isn't enough justification for suicide. */
+ edata.elevel = Min(edata.elevel, ERROR);
+
+ /*
+ * If desired, add a context line to show that this is a
+ * message propagated from a parallel worker. Otherwise, it
+ * can sometimes be confusing to understand what actually
+ * happened.
+ */
+ if (edata.context)
+ edata.context = psprintf("%s\n%s", edata.context,
+ _("decoding worker"));
+ else
+ edata.context = pstrdup(_("decoding worker"));
+
+ /* Rethrow error or print notice. */
+ ThrowErrorData(&edata);
+
+ break;
+ }
+
+ default:
+ {
+ elog(ERROR, "unrecognized message type received from decoding worker: %c (message length %d bytes)",
+ msgtype, msg->len);
+ }
+ }
+}
diff --git a/src/backend/libpq/pqmq.c b/src/backend/libpq/pqmq.c
index 6e4bbfb5aa1..42f6fa472c5 100644
--- a/src/backend/libpq/pqmq.c
+++ b/src/backend/libpq/pqmq.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/parallel.h"
+#include "commands/cluster.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "libpq/pqmq.h"
@@ -175,6 +176,10 @@ mq_putmessage(char msgtype, const char *s, size_t len)
SendProcSignal(pq_mq_parallel_leader_pid,
PROCSIG_PARALLEL_APPLY_MESSAGE,
pq_mq_parallel_leader_proc_number);
+ else if (IsRepackWorker())
+ SendProcSignal(pq_mq_parallel_leader_pid,
+ PROCSIG_REPACK_MESSAGE,
+ pq_mq_parallel_leader_proc_number);
else
{
Assert(IsParallelWorker());
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index 65deabe91a7..334bb708c5b 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -13,6 +13,7 @@
#include "postgres.h"
#include "access/parallel.h"
+#include "commands/cluster.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -136,6 +137,9 @@ static const struct
},
{
"SequenceSyncWorkerMain", SequenceSyncWorkerMain
+ },
+ {
+ "RepackWorkerMain", RepackWorkerMain
}
};
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index b0ef1a12520..35a46988285 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -194,7 +194,11 @@ StartupDecodingContext(List *output_plugin_options,
ctx->slot = slot;
- ctx->reader = XLogReaderAllocate(wal_segment_size, NULL, xl_routine, ctx);
+ /*
+ * TODO A separate patch for PG core, unless there's really a reason to
+ * pass ctx for private_data (May extensions expect ctx?).
+ */
+ ctx->reader = XLogReaderAllocate(wal_segment_size, NULL, xl_routine, NULL);
if (!ctx->reader)
ereport(ERROR,
(errcode(ERRCODE_OUT_OF_MEMORY),
diff --git a/src/backend/replication/pgoutput_repack/pgoutput_repack.c b/src/backend/replication/pgoutput_repack/pgoutput_repack.c
index c8930640a0d..fb9956d392d 100644
--- a/src/backend/replication/pgoutput_repack/pgoutput_repack.c
+++ b/src/backend/replication/pgoutput_repack/pgoutput_repack.c
@@ -168,17 +168,13 @@ store_change(LogicalDecodingContext *ctx, ConcurrentChangeKind kind,
HeapTuple tuple)
{
RepackDecodingState *dstate;
- char *change_raw;
- ConcurrentChange change;
+ char kind_byte = (char) kind;
bool flattened = false;
- Size size;
- Datum values[1];
- bool isnull[1];
- char *dst;
dstate = (RepackDecodingState *) ctx->output_writer_private;
- size = VARHDRSZ + SizeOfConcurrentChange;
+ /* Store the change kind. */
+ BufFileWrite(dstate->file, &kind_byte, 1);
/*
* ReorderBufferCommit() stores the TOAST chunks in its private memory
@@ -195,46 +191,12 @@ store_change(LogicalDecodingContext *ctx, ConcurrentChangeKind kind,
tuple = toast_flatten_tuple(tuple, dstate->tupdesc);
flattened = true;
}
+ /* Store the tuple size ... */
+ BufFileWrite(dstate->file, &tuple->t_len, sizeof(tuple->t_len));
+ /* ... and the tuple itself. */
+ BufFileWrite(dstate->file, tuple->t_data, tuple->t_len);
- size += tuple->t_len;
- if (size >= MaxAllocSize)
- elog(ERROR, "Change is too big.");
-
- /* Construct the change. */
- change_raw = (char *) palloc0(size);
- SET_VARSIZE(change_raw, size);
-
- /*
- * Since the varlena alignment might not be sufficient for the structure,
- * set the fields in a local instance and remember where it should
- * eventually be copied.
- */
- change.kind = kind;
- dst = (char *) VARDATA(change_raw);
-
- /*
- * Copy the tuple.
- *
- * Note: change->tup_data.t_data must be fixed on retrieval!
- */
- memcpy(&change.tup_data, tuple, sizeof(HeapTupleData));
- memcpy(dst, &change, SizeOfConcurrentChange);
- dst += SizeOfConcurrentChange;
- memcpy(dst, tuple->t_data, tuple->t_len);
-
- /* The data has been copied. */
+ /* Free the flat copy if created above. */
if (flattened)
pfree(tuple);
-
- /* Store as tuple of 1 bytea column. */
- values[0] = PointerGetDatum(change_raw);
- isnull[0] = false;
- tuplestore_putvalues(dstate->tstore, dstate->tupdesc_change,
- values, isnull);
-
- /* Accounting. */
- dstate->nchanges++;
-
- /* Cleanup. */
- pfree(change_raw);
}
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 8e56922dcea..6f9e7a7aab7 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -19,6 +19,7 @@
#include "access/parallel.h"
#include "commands/async.h"
+#include "commands/cluster.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bitutils.h"
@@ -697,6 +698,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
+ if (CheckProcSignal(PROCSIG_REPACK_MESSAGE))
+ HandleRepackMessageInterrupt();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
HandleRecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 015c67bbeba..566e5a50c30 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -36,6 +36,7 @@
#include "access/xact.h"
#include "catalog/pg_type.h"
#include "commands/async.h"
+#include "commands/cluster.h"
#include "commands/event_trigger.h"
#include "commands/explain_state.h"
#include "commands/prepare.h"
@@ -3541,6 +3542,9 @@ ProcessInterrupts(void)
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
+
+ if (RepackMessagePending)
+ ProcessRepackMessages();
}
/*
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3299de23bb3..73a3def69bc 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -62,6 +62,7 @@ LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
+REPACK_WORKER_MAIN "Waiting in main loop of REPACK decoding worker process."
REPLICATION_SLOTSYNC_MAIN "Waiting in main loop of slot synchronization."
REPLICATION_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down."
SYSLOGGER_MAIN "Waiting in main loop of syslogger process."
@@ -154,6 +155,7 @@ RECOVERY_CONFLICT_SNAPSHOT "Waiting for recovery conflict resolution for a vacuu
RECOVERY_CONFLICT_TABLESPACE "Waiting for recovery conflict resolution for dropping a tablespace."
RECOVERY_END_COMMAND "Waiting for <xref linkend="guc-recovery-end-command"/> to complete."
RECOVERY_PAUSE "Waiting for recovery to be resumed."
+REPACK_WORKER_EXPORT "Waiting for decoding worker to export a new output file."
REPLICATION_ORIGIN_DROP "Waiting for a replication origin to become inactive so it can be dropped."
REPLICATION_SLOT_DROP "Waiting for a replication slot to become inactive so it can be dropped."
RESTORE_COMMAND "Waiting for <xref linkend="guc-restore-command"/> to complete."
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 76aa993009a..15760363a1a 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -22,7 +22,6 @@
#include "access/xact.h"
#include "commands/vacuum.h"
#include "executor/tuptable.h"
-#include "replication/logical.h"
#include "storage/read_stream.h"
#include "utils/rel.h"
#include "utils/snapshot.h"
@@ -631,7 +630,6 @@ typedef struct TableAmRoutine
bool use_sort,
TransactionId OldestXmin,
Snapshot snapshot,
- LogicalDecodingContext *decoding_ctx,
TransactionId *xid_cutoff,
MultiXactId *multi_cutoff,
double *num_tuples,
@@ -1651,8 +1649,6 @@ table_relation_copy_data(Relation rel, const RelFileLocator *newrlocator)
* - *multi_cutoff - ditto
* - snapshot - if != NULL, ignore data changes done by transactions that this
* (MVCC) snapshot considers still in-progress or in the future.
- * - decoding_ctx - logical decoding context, to capture concurrent data
- * changes.
*
* Output parameters:
* - *xid_cutoff - rel's new relfrozenxid value, may be invalid
@@ -1666,7 +1662,6 @@ table_relation_copy_for_cluster(Relation OldTable, Relation NewTable,
bool use_sort,
TransactionId OldestXmin,
Snapshot snapshot,
- LogicalDecodingContext *decoding_ctx,
TransactionId *xid_cutoff,
MultiXactId *multi_cutoff,
double *num_tuples,
@@ -1675,7 +1670,7 @@ table_relation_copy_for_cluster(Relation OldTable, Relation NewTable,
{
OldTable->rd_tableam->relation_copy_for_cluster(OldTable, NewTable, OldIndex,
use_sort, OldestXmin,
- snapshot, decoding_ctx,
+ snapshot,
xid_cutoff, multi_cutoff,
num_tuples, tups_vacuumed,
tups_recently_dead);
diff --git a/src/include/commands/cluster.h b/src/include/commands/cluster.h
index 6a5c476294a..1b05d5d418b 100644
--- a/src/include/commands/cluster.h
+++ b/src/include/commands/cluster.h
@@ -17,11 +17,13 @@
#include "nodes/parsenodes.h"
#include "parser/parse_node.h"
#include "replication/decode.h"
+#include "postmaster/bgworker.h"
#include "replication/logical.h"
+#include "storage/buffile.h"
#include "storage/lock.h"
+#include "storage/shm_mq.h"
#include "utils/relcache.h"
#include "utils/resowner.h"
-#include "utils/tuplestore.h"
/* flag bits for ClusterParams->options */
@@ -44,6 +46,9 @@ typedef struct ClusterParams
* The following definitions are used by REPACK CONCURRENTLY.
*/
+/*
+ * Stored as a single byte in the output file.
+ */
typedef enum
{
CHANGE_INSERT,
@@ -52,68 +57,30 @@ typedef enum
CHANGE_DELETE
} ConcurrentChangeKind;
-typedef struct ConcurrentChange
-{
- /* See the enum above. */
- ConcurrentChangeKind kind;
-
- /*
- * The actual tuple.
- *
- * The tuple data follows the ConcurrentChange structure. Before use make
- * sure the tuple is correctly aligned (ConcurrentChange can be stored as
- * bytea) and that tuple->t_data is fixed.
- */
- HeapTupleData tup_data;
-} ConcurrentChange;
-
-#define SizeOfConcurrentChange (offsetof(ConcurrentChange, tup_data) + \
- sizeof(HeapTupleData))
-
/*
* Logical decoding state.
*
- * Here we store the data changes that we decode from WAL while the table
- * contents is being copied to a new storage. Also the necessary metadata
- * needed to apply these changes to the table is stored here.
+ * The output plugin uses it to store the data changes that it decodes from
+ * WAL while the table contents is being copied to a new storage.
*/
typedef struct RepackDecodingState
{
/* The relation whose changes we're decoding. */
Oid relid;
- /* Replication slot name. */
- NameData slotname;
-
- /*
- * Decoded changes are stored here. Although we try to avoid excessive
- * batches, it can happen that the changes need to be stored to disk. The
- * tuplestore does this transparently.
- */
- Tuplestorestate *tstore;
-
- /* The current number of changes in tstore. */
- double nchanges;
-
- /*
- * Descriptor to store the ConcurrentChange structure serialized (bytea).
- * We can't store the tuple directly because tuplestore only supports
- * minimum tuple and we may need to transfer OID system column from the
- * output plugin. Also we need to transfer the change kind, so it's better
- * to put everything in the structure than to use 2 tuplestores "in
- * parallel".
- */
- TupleDesc tupdesc_change;
-
- /* Tuple descriptor needed to update indexes. */
+ /* Tuple descriptor of the relation being processed. */
TupleDesc tupdesc;
- /* Slot to retrieve data from tstore. */
- TupleTableSlot *tsslot;
-
- ResourceOwner resowner;
+ /* The current output file. */
+ BufFile *file;
} RepackDecodingState;
+extern PGDLLIMPORT volatile sig_atomic_t RepackMessagePending;
+
+extern bool IsRepackWorker(void);
+extern void HandleRepackMessageInterrupt(void);
+extern void ProcessRepackMessages(void);
+
extern void ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel);
extern void cluster_rel(RepackCommand command, Relation OldHeap, Oid indexOid,
@@ -136,6 +103,6 @@ extern void finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap,
extern bool am_decoding_for_repack(void);
extern bool change_useless_for_repack(XLogRecordBuffer *buf);
-extern void repack_decode_concurrent_changes(LogicalDecodingContext *ctx,
- XLogRecPtr end_of_wal);
+
+extern void RepackWorkerMain(Datum main_arg);
#endif /* CLUSTER_H */
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index e52b8eb7697..3ef35ca6b80 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -36,6 +36,7 @@ typedef enum
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
+ PROCSIG_REPACK_MESSAGE, /* Message from repack worker */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_FIRST,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a0b7b38a5e2..d1a694f9008 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -496,7 +496,6 @@ CompressFileHandle
CompressionLocation
CompressorState
ComputeXidHorizonsResult
-ConcurrentChange
ConcurrentChangeKind
ConditionVariable
ConditionVariableMinimallyPadded
@@ -636,6 +635,9 @@ DeclareCursorStmt
DecodedBkpBlock
DecodedXLogRecord
DecodingOutputState
+DecodingWorker
+DecodingWorkerShared
+DecodingWorkerState
DefElem
DefElemAction
DefaultACLInfo
--
2.47.3
--=-=-=
Content-Type: text/plain
Content-Disposition: attachment;
filename=v29-0006-Use-multiple-snapshots-to-copy-the-data.patch
^ permalink raw reply [nested|flat] 25+ messages in thread
end of thread, other threads:[~2026-01-08 16:47 UTC | newest]
Thread overview: 25+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-07-15 22:35 [PATCH v13 3/6] snapshot scalability: Introduce dense array of in-progress xids. Andres Freund <[email protected]>
2022-10-01 00:20 Re: problems with making relfilenodes 56-bits Andres Freund <[email protected]>
2022-10-01 01:44 ` Re: problems with making relfilenodes 56-bits Peter Geoghegan <[email protected]>
2022-10-01 06:14 ` Re: problems with making relfilenodes 56-bits Dilip Kumar <[email protected]>
2022-10-03 12:12 ` Re: problems with making relfilenodes 56-bits Robert Haas <[email protected]>
2022-10-03 17:01 ` Re: problems with making relfilenodes 56-bits Andres Freund <[email protected]>
2022-10-03 17:40 ` Re: problems with making relfilenodes 56-bits Matthias van de Meent <[email protected]>
2022-10-03 21:25 ` Re: problems with making relfilenodes 56-bits Andres Freund <[email protected]>
2022-10-04 13:05 ` Re: problems with making relfilenodes 56-bits Matthias van de Meent <[email protected]>
2022-10-04 15:34 ` Re: problems with making relfilenodes 56-bits Andres Freund <[email protected]>
2022-10-04 17:36 ` Re: problems with making relfilenodes 56-bits Robert Haas <[email protected]>
2022-10-04 18:30 ` Re: problems with making relfilenodes 56-bits Andres Freund <[email protected]>
2022-10-04 18:53 ` Re: problems with making relfilenodes 56-bits Robert Haas <[email protected]>
2022-10-04 23:49 ` Re: problems with making relfilenodes 56-bits Andres Freund <[email protected]>
2022-10-10 09:16 ` Re: problems with making relfilenodes 56-bits Dilip Kumar <[email protected]>
2022-10-10 12:10 ` Re: problems with making relfilenodes 56-bits Robert Haas <[email protected]>
2022-10-10 21:22 ` Re: problems with making relfilenodes 56-bits Andres Freund <[email protected]>
2022-10-11 08:33 ` Re: problems with making relfilenodes 56-bits Dilip Kumar <[email protected]>
2022-10-12 20:05 ` Re: problems with making relfilenodes 56-bits Matthias van de Meent <[email protected]>
2022-10-12 21:13 ` Re: problems with making relfilenodes 56-bits Andres Freund <[email protected]>
2022-10-13 22:53 ` Re: problems with making relfilenodes 56-bits Matthias van de Meent <[email protected]>
2022-10-17 21:14 ` Re: problems with making relfilenodes 56-bits Robert Haas <[email protected]>
2022-10-19 19:21 ` Re: problems with making relfilenodes 56-bits Andres Freund <[email protected]>
2022-10-20 08:40 ` Re: problems with making relfilenodes 56-bits Dilip Kumar <[email protected]>
2026-01-08 16:47 [PATCH 5/6] Use background worker to do logical decoding. Antonin Houska <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox