agora inbox for [email protected]
help / color / mirror / Atom feedRe: Undo worker and transaction rollback
37+ messages / 14 participants
[nested] [flat]
* Re: Undo worker and transaction rollback
@ 2018-11-05 11:40 Dilip Kumar <[email protected]>
2018-11-05 12:32 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Dilip Kumar @ 2018-11-05 11:40 UTC (permalink / raw)
To: pgsql-hackers
On Thu, Oct 11, 2018 at 11:30 AM Dilip Kumar <[email protected]> wrote:
>
> Hello, hackers,
>
> In previous threads[1], we proposed patches for generating and storing
> undo records and now for undo-worker and the transaction rollback
> stuff. The idea is that undo remains relevant as long as the
> transaction is in progress and needs to be removed once it becomes
> irrelevant. Specifically, for a committed transaction, it remains
> relevant until the transaction becomes all-visible; zheap will use
> this for MVCC purposes. However, for an aborted transaction, it
> remains relevant until the “undo actions" described by the undo
> records have been performed. This patch introduces code to discard
> undo when it is no longer needed and reuse the associated storage.
> Additionally, this patch adds code to execute undo actions. Let me
> explain the finer details for each of the cases covered,
Latest rebased patch for undo worker and transaction rollback. This
patch includes some bug fixes from the main zheap branch and also
include code for executing undo action using RMGR (RMGR related code
is merged from Thomas' patch set for "POC: Cleaning up orphaned files
using undo logs"[1]. This patch can be applied on top of undolog and
undo-interface patches.
[1] https://www.postgresql.org/message-id/flat/CAEepm=0ULqYgM2aFeOnrx6YrtBg3xUdxALoyCG+XpssKqmezug@mail....
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/x-patch] undoworker-transaction-rollback_v2.patch (134.5K, ../../CAFiTN-t8fv-qYG9zynhS-1jRrvt_o5C-wCMRtzOsK8S=MXvKKw@mail.gmail.com/2-undoworker-transaction-rollback_v2.patch)
download | inline diff:
From d834ae8bcb7d077ea8064525985b15728c44e2c2 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <[email protected]>
Date: Mon, 5 Nov 2018 01:41:35 -0800
Subject: [PATCH 4/4] undoworker-transaction-rollback
Dilip Kumar, with contributions from Rafia Sabih, Amit Kapila and Mithun.CY.
---
src/backend/access/rmgrdesc/Makefile | 3 +-
src/backend/access/rmgrdesc/undoactiondesc.c | 64 +++
src/backend/access/rmgrdesc/xlogdesc.c | 4 +-
src/backend/access/transam/rmgr.c | 5 +-
src/backend/access/transam/twophase.c | 45 +-
src/backend/access/transam/varsup.c | 12 +
src/backend/access/transam/xact.c | 441 ++++++++++++++++-
src/backend/access/transam/xlog.c | 20 +
src/backend/access/undo/Makefile | 2 +-
src/backend/access/undo/undoaction.c | 684 ++++++++++++++++++++++++++
src/backend/access/undo/undoactionxlog.c | 49 ++
src/backend/access/undo/undodiscard.c | 458 +++++++++++++++++
src/backend/access/undo/undoinsert.c | 51 +-
src/backend/access/undo/undorecord.c | 3 +
src/backend/commands/vacuum.c | 11 +
src/backend/postmaster/Makefile | 4 +-
src/backend/postmaster/bgworker.c | 11 +
src/backend/postmaster/discardworker.c | 170 +++++++
src/backend/postmaster/pgstat.c | 13 +-
src/backend/postmaster/postmaster.c | 7 +
src/backend/postmaster/undoworker.c | 664 +++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 4 +
src/backend/storage/ipc/ipci.c | 6 +
src/backend/storage/lmgr/lwlocknames.txt | 2 +
src/backend/storage/lmgr/proc.c | 2 +
src/backend/utils/adt/lockfuncs.c | 1 +
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc.c | 11 +
src/backend/utils/misc/pg_controldata.c | 9 +-
src/backend/utils/misc/postgresql.conf.sample | 7 +
src/bin/pg_controldata/pg_controldata.c | 2 +
src/bin/pg_resetwal/pg_resetwal.c | 7 +
src/bin/pg_rewind/parsexlog.c | 2 +-
src/bin/pg_waldump/rmgrdesc.c | 3 +-
src/include/access/rmgr.h | 2 +-
src/include/access/rmgrlist.h | 47 +-
src/include/access/transam.h | 4 +
src/include/access/twophase.h | 2 +-
src/include/access/undoaction.h | 28 ++
src/include/access/undoaction_xlog.h | 74 +++
src/include/access/undodiscard.h | 31 ++
src/include/access/undoinsert.h | 2 +
src/include/access/undorecord.h | 13 +
src/include/access/xact.h | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlog_internal.h | 9 +-
src/include/catalog/pg_control.h | 7 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +
src/include/postmaster/discardworker.h | 25 +
src/include/postmaster/undoloop.h | 89 ++++
src/include/postmaster/undoworker.h | 39 ++
src/include/storage/lock.h | 10 +
src/include/storage/proc.h | 2 +
54 files changed, 3125 insertions(+), 45 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/undoactiondesc.c
create mode 100644 src/backend/access/undo/undoaction.c
create mode 100644 src/backend/access/undo/undoactionxlog.c
create mode 100644 src/backend/access/undo/undodiscard.c
create mode 100644 src/backend/postmaster/discardworker.c
create mode 100644 src/backend/postmaster/undoworker.c
create mode 100644 src/include/access/undoaction.h
create mode 100644 src/include/access/undoaction_xlog.h
create mode 100644 src/include/access/undodiscard.h
create mode 100644 src/include/postmaster/discardworker.h
create mode 100644 src/include/postmaster/undoloop.h
create mode 100644 src/include/postmaster/undoworker.h
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index 91ad1ef..640d37f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = brindesc.o clogdesc.o committsdesc.o dbasedesc.o genericdesc.o \
gindesc.o gistdesc.o hashdesc.o heapdesc.o logicalmsgdesc.o \
mxactdesc.o nbtdesc.o relmapdesc.o replorigindesc.o seqdesc.o \
- smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undologdesc.o xactdesc.o xlogdesc.o
+ smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undoactiondesc.o \
+ undologdesc.o xactdesc.o xlogdesc.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/rmgrdesc/undoactiondesc.c b/src/backend/access/rmgrdesc/undoactiondesc.c
new file mode 100644
index 0000000..89343b8
--- /dev/null
+++ b/src/backend/access/rmgrdesc/undoactiondesc.c
@@ -0,0 +1,64 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactiondesc.c
+ * rmgr descriptor routines for access/undo/undoactionxlog.c
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/access/rmgrdesc/undoactiondesc.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+
+void
+undoaction_desc(StringInfo buf, XLogReaderState *record)
+{
+ char *rec = XLogRecGetData(record);
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ if (info == XLOG_UNDO_PAGE)
+ {
+ uint8 *flags = (uint8 *) rec;
+
+ appendStringInfo(buf, "page_contains_tpd_slot: %c ",
+ (*flags & XLU_PAGE_CONTAINS_TPD_SLOT) ? 'T' : 'F');
+ appendStringInfo(buf, "is_page_initialized: %c ",
+ (*flags & XLU_INIT_PAGE) ? 'T' : 'F');
+ if (*flags & XLU_PAGE_CONTAINS_TPD_SLOT)
+ {
+ xl_undoaction_page *xlrec =
+ (xl_undoaction_page *) ((char *) flags + sizeof(uint8));
+
+ appendStringInfo(buf, "urec_ptr %lu xid %u trans_slot_id %u",
+ xlrec->urec_ptr, xlrec->xid, xlrec->trans_slot_id);
+ }
+ }
+ else if (info == XLOG_UNDO_APPLY_PROGRESS)
+ {
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) rec;
+
+ appendStringInfo(buf, "urec_ptr %lu progress %u",
+ xlrec->urec_ptr, xlrec->progress);
+ }
+}
+
+const char *
+undoaction_identify(uint8 info)
+{
+ const char *id = NULL;
+
+ switch (info & ~XLR_INFO_MASK)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ id = "UNDO APPLY PROGRESS";
+ break;
+ }
+
+ return id;
+}
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 00741c7..987e39c 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -47,7 +47,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
"tli %u; prev tli %u; fpw %s; xid %u:%u; oid %u; multi %u; offset %u; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
- "oldest running xid %u; %s",
+ "oldest running xid %u; "
+ "oldest xid with epoch having undo " UINT64_FORMAT "; %s",
(uint32) (checkpoint->redo >> 32), (uint32) checkpoint->redo,
checkpoint->ThisTimeLineID,
checkpoint->PrevTimeLineID,
@@ -63,6 +64,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
checkpoint->oldestCommitTsXid,
checkpoint->newestCommitTsXid,
checkpoint->oldestActiveXid,
+ checkpoint->oldestXidWithEpochHavingUndo,
(info == XLOG_CHECKPOINT_SHUTDOWN) ? "shutdown" : "online");
}
else if (info == XLOG_NEXTOID)
diff --git a/src/backend/access/transam/rmgr.c b/src/backend/access/transam/rmgr.c
index 8b05374..6238240 100644
--- a/src/backend/access/transam/rmgr.c
+++ b/src/backend/access/transam/rmgr.c
@@ -18,6 +18,7 @@
#include "access/multixact.h"
#include "access/nbtxlog.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -31,8 +32,8 @@
#include "utils/relmapper.h"
/* must be kept in sync with RmgrData definition in xlog_internal.h */
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
- { name, redo, desc, identify, startup, cleanup, mask },
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
+ { name, redo, desc, identify, startup, cleanup, mask, undo, undo_desc },
const RmgrData RmgrTable[RM_MAX_ID + 1] = {
#include "access/rmgrlist.h"
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 3942734..83241bc 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -93,6 +93,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/origin.h"
#include "replication/syncrep.h"
#include "replication/walsender.h"
@@ -915,6 +916,12 @@ typedef struct TwoPhaseFileHeader
uint16 gidlen; /* length of the GID - GID follows the header */
XLogRecPtr origin_lsn; /* lsn of this record at origin node */
TimestampTz origin_timestamp; /* time of prepare at origin node */
+ /*
+ * We need the locations of start and end undo record pointers when rollbacks
+ * are to be performed for prepared transactions using zheap relations.
+ */
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
} TwoPhaseFileHeader;
/*
@@ -989,7 +996,8 @@ save_state_data(const void *data, uint32 len)
* Initializes data structure and inserts the 2PC file header record.
*/
void
-StartPrepare(GlobalTransaction gxact)
+StartPrepare(GlobalTransaction gxact, UndoRecPtr *start_urec_ptr,
+ UndoRecPtr *end_urec_ptr)
{
PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
PGXACT *pgxact = &ProcGlobal->allPgXact[gxact->pgprocno];
@@ -1027,6 +1035,10 @@ StartPrepare(GlobalTransaction gxact)
&hdr.initfileinval);
hdr.gidlen = strlen(gxact->gid) + 1; /* Include '\0' */
+ /* save the start and end undo record pointers */
+ memcpy(hdr.start_urec_ptr, start_urec_ptr, sizeof(hdr.start_urec_ptr));
+ memcpy(hdr.end_urec_ptr, end_urec_ptr, sizeof(hdr.end_urec_ptr));
+
save_state_data(&hdr, sizeof(TwoPhaseFileHeader));
save_state_data(gxact->gid, hdr.gidlen);
@@ -1452,6 +1464,9 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
RelFileNode *delrels;
int ndelrels;
SharedInvalidationMessage *invalmsgs;
+ int i;
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
/*
* Validate the GID, and lock the GXACT to ensure that two backends do not
@@ -1489,6 +1504,34 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
invalmsgs = (SharedInvalidationMessage *) bufptr;
bufptr += MAXALIGN(hdr->ninvalmsgs * sizeof(SharedInvalidationMessage));
+ /* save the start and end undo record pointers */
+ memcpy(start_urec_ptr, hdr->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(end_urec_ptr, hdr->end_urec_ptr, sizeof(end_urec_ptr));
+
+ /*
+ * Perform undo actions, if there are undologs for this transaction.
+ * We need to perform undo actions while we are still in transaction.
+ * Never push rollbacks of temp tables to undo worker.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (end_urec_ptr[i] != InvalidUndoRecPtr && !isCommit)
+ {
+ bool result = false;
+ uint64 rollback_size = 0;
+
+ if (i != UNDO_TEMP)
+ rollback_size = end_urec_ptr[i] - start_urec_ptr[i];
+
+ if (rollback_size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(end_urec_ptr[i], start_urec_ptr[i], InvalidOid);
+
+ if (!result)
+ execute_undo_actions(end_urec_ptr[i], start_urec_ptr[i], true,
+ true, false);
+ }
+ }
+
/* compute latestXid among all children */
latestXid = TransactionIdLatest(xid, hdr->nsubxacts, children);
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index cede579..2274ab2 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -292,10 +292,22 @@ SetTransactionIdLimit(TransactionId oldest_datfrozenxid, Oid oldest_datoid)
TransactionId xidStopLimit;
TransactionId xidWrapLimit;
TransactionId curXid;
+ TransactionId oldestXidHavingUndo;
Assert(TransactionIdIsNormal(oldest_datfrozenxid));
/*
+ * To determine the last safe xid that can be allocated, we need to
+ * consider oldestXidHavingUndo. The oldestXidHavingUndo will be only
+ * valid for zheap storage engine, so it won't impact any other storage
+ * engine.
+ */
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ oldest_datfrozenxid = Min(oldest_datfrozenxid, oldestXidHavingUndo);
+
+ /*
* The place where we actually get into deep trouble is halfway around
* from the oldest potentially-existing XID. (This calculation is
* probably off by one or two counts, because the special XIDs reduce the
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 6b7f7fa..80fb1fa 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -41,6 +41,7 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/origin.h"
@@ -278,6 +279,20 @@ typedef struct SubXactCallbackItem
static SubXactCallbackItem *SubXact_callbacks = NULL;
+/* Location in undo log from where to start applying the undo actions. */
+static UndoRecPtr UndoActionStartPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+/* Location in undo log up to which undo actions need to be applied. */
+static UndoRecPtr UndoActionEndPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+/* Do we need to perform any undo actions? */
+static bool PerformUndoActions = false;
/* local function prototypes */
static void AssignTransactionId(TransactionState s);
@@ -1824,6 +1839,7 @@ StartTransaction(void)
{
TransactionState s;
VirtualTransactionId vxid;
+ int i;
/*
* Let's just make sure the state stack is empty
@@ -1884,6 +1900,13 @@ StartTransaction(void)
nUnreportedXids = 0;
s->didLogXid = false;
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
/*
* must initialize resource-management stuff first
*/
@@ -2207,7 +2230,7 @@ CommitTransaction(void)
* NB: if you change this routine, better look at CommitTransaction too!
*/
static void
-PrepareTransaction(void)
+PrepareTransaction(UndoRecPtr *start_urec_ptr, UndoRecPtr *end_urec_ptr)
{
TransactionState s = CurrentTransactionState;
TransactionId xid = GetCurrentTransactionId();
@@ -2355,7 +2378,7 @@ PrepareTransaction(void)
* PREPARED; in particular, pay attention to whether things should happen
* before or after releasing the transaction's locks.
*/
- StartPrepare(gxact);
+ StartPrepare(gxact, start_urec_ptr, end_urec_ptr);
AtPrepare_Notify();
AtPrepare_Locks();
@@ -2478,6 +2501,65 @@ PrepareTransaction(void)
RESUME_INTERRUPTS();
}
+static void
+AtAbort_Rollback(void)
+{
+ TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i;
+
+ /* XXX: TODO: check this logic, which was moved out of UserAbortTransactionBlock */
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (CurrentTransactionState->state == TRANS_INPROGRESS)
+ {
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (latest_urec_ptr[i])
+ {
+ if (i == UNDO_TEMP)
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ false, true, true);
+ else
+ {
+ uint64 size = latest_urec_ptr[i] - s->start_urec_ptr[i];
+ bool result = false;
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions
+ * later.
+ */
+ if (size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ true, true, true);
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
+}
/*
* AbortTransaction
@@ -2579,6 +2661,7 @@ AbortTransaction(void)
*/
AfterTriggerEndXact(false); /* 'false' means it's abort */
AtAbort_Portals();
+ AtAbort_Rollback();
AtEOXact_LargeObject(false);
AtAbort_Notify();
AtEOXact_RelationMap(false, is_parallel_worker);
@@ -2787,6 +2870,12 @@ void
CommitTransactionCommand(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i;
+
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(end_urec_ptr, s->latest_urec_ptr, sizeof(end_urec_ptr));
switch (s->blockState)
{
@@ -2876,7 +2965,7 @@ CommitTransactionCommand(void)
* return to the idle state.
*/
case TBLOCK_PREPARE:
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, end_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
break;
@@ -2922,6 +3011,23 @@ CommitTransactionCommand(void)
{
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
+
+ /*
+ * Update the end undo record pointer if it's not valid with
+ * the currently popped transaction's end undo record pointer.
+ * This is particularly required when the first command of
+ * the transaction is of type which does not require an undo,
+ * e.g. savepoint x.
+ * Accordingly, update the start undo record pointer.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(end_urec_ptr[i]))
+ end_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(s->start_urec_ptr[i]))
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
} while (s->blockState == TBLOCK_SUBCOMMIT);
/* If we had a COMMIT command, finish off the main xact too */
if (s->blockState == TBLOCK_END)
@@ -2933,7 +3039,7 @@ CommitTransactionCommand(void)
else if (s->blockState == TBLOCK_PREPARE)
{
Assert(s->parent == NULL);
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, end_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
}
else
@@ -3027,6 +3133,18 @@ void
AbortCurrentTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
+
+ /*
+ * The undo actions are allowed to be executed at the end of statement
+ * execution when we are not in transaction block, otherwise they are
+ * executed when user explicitly ends the transaction.
+ *
+ * So if we are in a transaction block don't set the PerformUndoActions
+ * because this flag will be set when user explicitly issue rollback or
+ * rollback to savepoint.
+ */
+ PerformUndoActions = false;
switch (s->blockState)
{
@@ -3061,6 +3179,16 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * We are outside the transaction block so remember the required
+ * information to perform undo actions and also set the
+ * PerformUndoActions so that we execute it before completing this
+ * command.
+ */
+ PerformUndoActions = true;
+ memcpy (UndoActionStartPtr, s->latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
break;
/*
@@ -3097,6 +3225,9 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3116,6 +3247,9 @@ AbortCurrentTransaction(void)
case TBLOCK_ABORT_END:
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3126,6 +3260,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Failed while executing the rollback command, need perform any
+ * pending undo actions.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3137,6 +3277,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Perform any pending actions if failed while preparing the
+ * transaction.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3159,6 +3305,17 @@ AbortCurrentTransaction(void)
case TBLOCK_SUBCOMMIT:
case TBLOCK_SUBABORT_PENDING:
case TBLOCK_SUBRESTART:
+ /*
+ * If we are here and still UndoActionStartPtr is valid that means
+ * the subtransaction failed while executing the undo action, so
+ * store its undo action start point in parent so that parent can
+ * start its undo action from this point.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ s->parent->latest_urec_ptr[i] = UndoActionStartPtr[i];
+ }
AbortSubTransaction();
CleanupSubTransaction();
AbortCurrentTransaction();
@@ -3176,6 +3333,109 @@ AbortCurrentTransaction(void)
}
/*
+ * XactPerformUndoActionsIfPending - Execute pending undo actions.
+ *
+ * If the parent transaction state is valid (when there is an error in the
+ * subtransaction and rollback to savepoint is executed), then allow to
+ * perform undo actions in it, otherwise perform them in a new transaction.
+ */
+void
+XactPerformUndoActionsIfPending()
+{
+ TransactionState s = CurrentTransactionState;
+ uint64 rollback_size = 0;
+ bool new_xact = true, result = false, no_pending_action = true;
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ if (!PerformUndoActions)
+ return;
+
+ /* If there is no undo log for any persistence level, then return. */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ {
+ no_pending_action = false;
+ break;
+ }
+ }
+
+ if (no_pending_action)
+ {
+ PerformUndoActions = false;
+ return;
+ }
+
+ /*
+ * Execute undo actions under parent transaction, if any. Otherwise start
+ * a new transaction.
+ */
+ if (GetTopTransactionIdIfAny() != InvalidTransactionId)
+ {
+ memcpy(parent_latest_urec_ptr, s->latest_urec_ptr,
+ sizeof (parent_latest_urec_ptr));
+ new_xact = false;
+ }
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions later.
+ * Never push the rollbacks for temp tables.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ continue;
+
+ if (i == UNDO_TEMP)
+ goto perform_rollback;
+ else
+ rollback_size = UndoActionStartPtr[i] - UndoActionEndPtr[i];
+
+ if (new_xact && rollback_size > rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+perform_rollback:
+ if (new_xact)
+ {
+ TransactionState xact;
+
+ /* Start a new transaction for performing the rollback */
+ StartTransactionCommand();
+ xact = CurrentTransactionState;
+
+ /*
+ * Store the previous transactions start and end undo record
+ * pointers into this transaction's state so that if there is
+ * some error while performing undo actions we can restart
+ * from begining.
+ */
+ memcpy(xact->start_urec_ptr, UndoActionEndPtr,
+ sizeof(UndoActionEndPtr));
+ memcpy(xact->latest_urec_ptr, UndoActionStartPtr,
+ sizeof(UndoActionStartPtr));
+ }
+
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ new_xact, true, true);
+
+ if (new_xact)
+ CommitTransactionCommand();
+ else
+ {
+ /* Restore parent's state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+ }
+
+ PerformUndoActions = false;
+}
+
+/*
* PreventInTransactionBlock
*
* This routine is to be called by statements that must not run inside
@@ -3576,6 +3836,10 @@ EndTransactionBlock(void)
{
TransactionState s = CurrentTransactionState;
bool result = false;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i ;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
switch (s->blockState)
{
@@ -3621,6 +3885,16 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+
+ /*
+ * We are calculating latest_urec_ptr, even though its a commit
+ * case. This is to handle any error during the commit path.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_END;
@@ -3646,6 +3920,11 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if(!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_ABORT_PENDING;
@@ -3698,6 +3977,18 @@ EndTransactionBlock(void)
break;
}
+ /*
+ * We need to perform undo actions if the transaction is failed. Remember
+ * the required information to perform undo actions at the end of
+ * statement execution.
+ */
+ if (!result)
+ PerformUndoActions = true;
+
+ memcpy(UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy(UndoActionEndPtr, TopTransactionStateData.start_urec_ptr,
+ sizeof(UndoActionEndPtr));
+
return result;
}
@@ -3955,6 +4246,12 @@ ReleaseSavepoint(const char *name)
TransactionState s = CurrentTransactionState;
TransactionState target,
xact;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4048,8 +4345,32 @@ ReleaseSavepoint(const char *name)
if (xact == target)
break;
xact = xact->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = xact->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(xact->start_urec_ptr[i]))
+ start_urec_ptr[i] = xact->start_urec_ptr[i];
+ }
+
Assert(PointerIsValid(xact));
}
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction. Also set the start_urec_ptr if
+ * parent start_urec_ptr is not valid.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ xact->parent->latest_urec_ptr[i] = latest_urec_ptr[i];
+ if (!UndoRecPtrIsValid(xact->parent->start_urec_ptr[i]))
+ xact->parent->start_urec_ptr[i] = start_urec_ptr[i];
+ }
}
/*
@@ -4260,6 +4581,7 @@ void
ReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4278,6 +4600,22 @@ ReleaseCurrentSubTransaction(void)
BlockStateAsString(s->blockState));
Assert(s->state == TRANS_INPROGRESS);
MemoryContextSwitchTo(CurTransactionContext);
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ s->parent->latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (!UndoRecPtrIsValid(s->parent->start_urec_ptr[i]))
+ s->parent->start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
Assert(s->state == TRANS_INPROGRESS);
@@ -4294,6 +4632,14 @@ void
RollbackAndReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i;
/*
* Unlike ReleaseCurrentSubTransaction(), this is nominally permitted
@@ -4340,6 +4686,19 @@ RollbackAndReleaseCurrentSubTransaction(void)
if (s->blockState == TBLOCK_SUBINPROGRESS)
AbortSubTransaction();
+ /*
+ * Remember the required information to perform undo actions before
+ * cleaning up the subtransaction state.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ {
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+ }
+
/* And clean it up, too */
CleanupSubTransaction();
@@ -4348,6 +4707,30 @@ RollbackAndReleaseCurrentSubTransaction(void)
s->blockState == TBLOCK_INPROGRESS ||
s->blockState == TBLOCK_IMPLICIT_INPROGRESS ||
s->blockState == TBLOCK_STARTED);
+
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ parent_latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ /*
+ * Store the undo action start point in the parent state so that
+ * we can apply undo actions these undos also during rollback of
+ * parent transaction in case of error while applying the undo
+ * actions.
+ */
+ s->latest_urec_ptr[i] = latest_urec_ptr[i];
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false,
+ true, true);
+
+ /* Restore parent state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+
+ /* Successfully performed undo actions so reset the flag. */
+ PerformUndoActions = false;
}
/*
@@ -4561,6 +4944,7 @@ static void
StartSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
if (s->state != TRANS_DEFAULT)
elog(WARNING, "StartSubTransaction while in %s state",
@@ -4578,6 +4962,13 @@ StartSubTransaction(void)
AtSubStart_Notify();
AfterTriggerBeginSubXact();
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
s->state = TRANS_INPROGRESS;
/*
@@ -4701,6 +5092,47 @@ CommitSubTransaction(void)
PopTransaction();
}
+static void
+AtSubAbort_Rollback(TransactionState s)
+{
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ /* XXX: TODO: Check this logic, which was moved out of RollbackToSavepoint() */
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (s->state == TRANS_INPROGRESS)
+ {
+ for ( i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false, true, false);
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
+}
+
/*
* AbortSubTransaction
*/
@@ -4795,6 +5227,7 @@ AbortSubTransaction(void)
s->parent->subTransactionId,
s->curTransactionOwner,
s->parent->curTransactionOwner);
+ AtSubAbort_Rollback(s);
AtEOSubXact_LargeObject(false, s->subTransactionId,
s->parent->subTransactionId);
AtSubAbort_Notify();
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 23f23e7..cdc085f 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -5172,6 +5172,7 @@ BootStrapXLOG(void)
checkPoint.newestCommitTsXid = InvalidTransactionId;
checkPoint.time = (pg_time_t) time(NULL);
checkPoint.oldestActiveXid = InvalidTransactionId;
+ checkPoint.oldestXidWithEpochHavingUndo = InvalidTransactionId;
ShmemVariableCache->nextXid = checkPoint.nextXid;
ShmemVariableCache->nextOid = checkPoint.nextOid;
@@ -6792,6 +6793,10 @@ StartupXLOG(void)
(errmsg_internal("commit timestamp Xid oldest/newest: %u/%u",
checkPoint.oldestCommitTsXid,
checkPoint.newestCommitTsXid)));
+ ereport(DEBUG1,
+ (errmsg_internal("oldest xid with epoch having undo: " UINT64_FORMAT,
+ checkPoint.oldestXidWithEpochHavingUndo)));
+
if (!TransactionIdIsNormal(checkPoint.nextXid))
ereport(PANIC,
(errmsg("invalid next transaction ID")));
@@ -6809,6 +6814,10 @@ StartupXLOG(void)
XLogCtl->ckptXidEpoch = checkPoint.nextXidEpoch;
XLogCtl->ckptXid = checkPoint.nextXid;
+ /* Read oldest xid having undo from checkpoint and set in proc global. */
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* Initialize replication slots, before there's a chance to remove
* required resources.
@@ -8979,6 +8988,9 @@ CreateCheckPoint(int flags)
checkPoint.nextOid += ShmemVariableCache->oidCount;
LWLockRelease(OidGenLock);
+ checkPoint.oldestXidWithEpochHavingUndo =
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo);
+
MultiXactGetCheckptMulti(shutdown,
&checkPoint.nextMulti,
&checkPoint.nextMultiOffset,
@@ -9889,6 +9901,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceOldest(checkPoint.oldestMulti,
checkPoint.oldestMultiDB);
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* No need to set oldestClogXid here as well; it'll be set when we
* redo an xl_clog_truncate if it changed since initialization.
@@ -9947,6 +9962,8 @@ xlog_redo(XLogReaderState *record)
/* ControlFile->checkPointCopy always tracks the latest ckpt XID */
ControlFile->checkPointCopy.nextXidEpoch = checkPoint.nextXidEpoch;
ControlFile->checkPointCopy.nextXid = checkPoint.nextXid;
+ ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo =
+ checkPoint.oldestXidWithEpochHavingUndo;
/* Update shared-memory copy of checkpoint XID/epoch */
SpinLockAcquire(&XLogCtl->info_lck);
@@ -9996,6 +10013,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceNextMXact(checkPoint.nextMulti,
checkPoint.nextMultiOffset);
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* NB: This may perform multixact truncation when replaying WAL
* generated by an older primary.
diff --git a/src/backend/access/undo/Makefile b/src/backend/access/undo/Makefile
index f41e8f7..fdf7f7d 100644
--- a/src/backend/access/undo/Makefile
+++ b/src/backend/access/undo/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/access/undo
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = undoinsert.o undolog.o undorecord.o
+OBJS = undoaction.o undoactionxlog.o undodiscard.o undoinsert.o undolog.o undorecord.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/undo/undoaction.c b/src/backend/access/undo/undoaction.c
new file mode 100644
index 0000000..696dac2
--- /dev/null
+++ b/src/backend/access/undo/undoaction.c
@@ -0,0 +1,684 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.c
+ * execute undo actions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undoaction.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/undoaction.h"
+#include "access/undoaction_xlog.h"
+#include "access/undolog.h"
+#include "access/undorecord.h"
+#include "access/visibilitymap.h"
+#include "access/xact.h"
+#include "access/xloginsert.h"
+#include "access/xlog_internal.h"
+#include "nodes/pg_list.h"
+#include "pgstat.h"
+#include "postmaster/undoloop.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "utils/rel.h"
+#include "utils/relfilenodemap.h"
+#include "miscadmin.h"
+#include "storage/shmem.h"
+#include "access/undodiscard.h"
+
+#define ROLLBACK_HT_SIZE 1024
+
+static bool execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr,
+ Oid reloid, TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool norellock, int options);
+static void RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr);
+
+/* This is the hash table to store all the rollabck requests. */
+static HTAB *RollbackHT;
+
+/*
+ * execute_undo_actions - Execute the undo actions
+ *
+ * from_urecptr - undo record pointer from where to start applying undo action.
+ * to_urecptr - undo record pointer upto which point apply undo action.
+ * nopartial - true if rollback is for complete transaction.
+ * rewind - whether to rewind the insert location of the undo log or not.
+ * Only the backend executed the transaction can rewind, but
+ * any other process e.g. undo worker should not rewind it.
+ * Because, if the backend have already inserted new undo records
+ * for the next transaction and if we rewind then we will loose
+ * the undo record inserted for the new transaction.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ */
+void
+execute_undo_actions(UndoRecPtr from_urecptr, UndoRecPtr to_urecptr,
+ bool nopartial, bool rewind, bool rellock)
+{
+ UnpackedUndoRecord *uur = NULL;
+ UndoRecPtr urec_ptr, prev_urec_ptr;
+ UndoRecPtr save_urec_ptr;
+ Oid prev_reloid = InvalidOid;
+ ForkNumber prev_fork = InvalidForkNumber;
+ BlockNumber prev_block = InvalidBlockNumber;
+ List *luinfo = NIL;
+ bool more_undo;
+ int options = 0;
+ TransactionId xid = InvalidTransactionId;
+ UndoRecInfo *urec_info;
+
+ Assert(from_urecptr != InvalidUndoRecPtr);
+ Assert(UndoRecPtrGetLogNo(from_urecptr) != UndoRecPtrGetLogNo(to_urecptr) ||
+ from_urecptr >= to_urecptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ * FIXME: this won't work if undolog crossed the limit of 1TB, because
+ * then from_urecptr and to_urecptr will be from different lognos.
+ */
+ if (to_urecptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(from_urecptr);
+ to_urecptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ save_urec_ptr = urec_ptr = from_urecptr;
+
+ if (nopartial)
+ {
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber, InvalidOffsetNumber,
+ InvalidTransactionId, NULL, NULL);
+ if (uur == NULL)
+ return;
+
+ xid = uur->uur_xid;
+ UndoRecordRelease(uur);
+ uur = NULL;
+
+ /*
+ * Grab the undo action apply lock before start applying the undo action
+ * this will prevent applying undo actions concurrently. If we do not
+ * get the lock that mean its already being applied concurrently or the
+ * discard worker might be pushing its request to the rollback hash
+ * table
+ */
+ if (!ConditionTransactionUndoActionLock(xid))
+ return;
+ }
+
+ prev_urec_ptr = InvalidUndoRecPtr;
+ while (prev_urec_ptr != to_urecptr)
+ {
+ Oid reloid = InvalidOid;
+ uint16 urec_prevlen;
+ bool non_page;
+
+ more_undo = true;
+
+ prev_urec_ptr = urec_ptr;
+
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId, NULL, NULL);
+
+ /* If there is no info block, this is not a page-based undo record. */
+ non_page = uur && !(uur->uur_info & UREC_INFO_BLOCK);
+
+ if (uur != NULL && !non_page)
+ reloid = RelidByRelfilenode(uur->uur_tsid, uur->uur_relfilenode);
+
+ xid = uur->uur_xid;
+
+ if (non_page)
+ {
+ prev_reloid = InvalidOid;
+ urec_prevlen = uur->uur_prevlen;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /*
+ * Execute individual undo actions not associated with a page
+ * immediately.
+ */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->uur = uur;
+ urec_info->urp = urec_ptr;
+ luinfo = lappend(luinfo, urec_info);
+ execute_undo_actions_page(luinfo, urec_ptr, reloid, xid,
+ InvalidBlockNumber, false, rellock,
+ 0);
+ pfree(urec_info);
+ urec_info = NULL;
+ list_free(luinfo);
+ luinfo = NIL;
+ UndoRecordRelease(uur);
+
+ /* Follow undo chain until to_urecptr. */
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ /*
+ * If the record is already discarded by undo worker or if the relation
+ * is dropped or truncated, then we cannot fetch record successfully.
+ * Hence, skip quietly.
+ *
+ * Note: reloid remains InvalidOid for a discarded record.
+ */
+ else if (!OidIsValid(reloid))
+ {
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ urec_prevlen = uur->uur_prevlen;
+
+ /* Release the just-fetched record */
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else if (!OidIsValid(prev_reloid) ||
+ (prev_reloid == reloid &&
+ prev_fork == uur->uur_fork &&
+ prev_block == uur->uur_block))
+ {
+ /* Collect the undo records that belong to the same page. */
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+
+ luinfo = lappend(luinfo, urec_info);
+ urec_prevlen = uur->uur_prevlen;
+ save_urec_ptr = uur->uur_blkprev;
+
+ if (uur->uur_info & UREC_INFO_PAYLOAD_CONTAINS_SLOT)
+ options |= UNDO_ACTION_UPDATE_TPD;
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else
+ {
+ more_undo = true;
+ }
+
+ /*
+ * If no more undo is left to be processed and we are rolling back the
+ * complete transaction, then we can consider that the undo chain for a
+ * block is complete.
+ * If the previous undo pointer in the page is invalid, then also the
+ * undo chain for the current block is completed.
+ */
+ if (luinfo &&
+ ((!more_undo && nopartial) || !UndoRecPtrIsValid(save_urec_ptr)))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, true, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+ else if (luinfo)
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, false, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ /*
+ * There are still more records to process, so keep moving backwards
+ * in the chain.
+ */
+ if (more_undo)
+ {
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+ luinfo = lappend(luinfo, urec_info);
+
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+ save_urec_ptr = uur->uur_blkprev;
+
+ if (uur->uur_info & UREC_INFO_PAYLOAD_CONTAINS_SLOT)
+ options |= UNDO_ACTION_UPDATE_TPD;
+
+ /*
+ * Continue to process the records if this is not the last undo
+ * record in chain.
+ */
+ urec_prevlen = uur->uur_prevlen;
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ else
+ break;
+ }
+ else
+ break;
+ }
+
+ /* Apply the undo actions for the remaining records. */
+ if (list_length(luinfo))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, nopartial ? true : false,
+ rellock, options);
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+ }
+
+ if (rewind)
+ {
+ /* Read the prevlen from the first record of this transaction. */
+ uur = UndoFetchRecord(to_urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If undo is already discarded before we rewind, then do nothing.
+ */
+ if (uur == NULL)
+ return;
+
+
+ /*
+ * Rewind the insert location to start of this transaction. This is
+ * to avoid reapplying some intermediate undo. We do not need to wal
+ * log this information here, because if the system crash before we
+ * rewind the insert pointer then after recovery we can identify
+ * whether the undo is already applied or not from the slot undo record
+ * pointer. Also set the correct prevlen value (what we have fetched
+ * from the undo).
+ */
+ UndoLogRewind(to_urecptr, uur->uur_prevlen);
+
+ UndoRecordRelease(uur);
+ }
+
+ if (nopartial)
+ {
+ /*
+ * Set undo action apply completed in the transaction header if this is
+ * a main transaction and we have not rewound its undo.
+ */
+ if (!rewind)
+ {
+ /*
+ * Undo action is applied so delete the hash table entry and release
+ * the undo action lock.
+ */
+ RollbackHTRemoveEntry(from_urecptr);
+
+ /*
+ * Prepare and update the progress of the undo action apply in the
+ * transaction header.
+ */
+ PrepareUpdateUndoActionProgress(to_urecptr, 1);
+
+ START_CRIT_SECTION();
+
+ /* Update the progress in the transaction header. */
+ UndoRecordUpdateTransInfo();
+
+ /* WAL log the undo apply progress. */
+ {
+ xl_undoapply_progress xlrec;
+
+ xlrec.urec_ptr = to_urecptr;
+ xlrec.progress = 1;
+
+ /*
+ * FIXME : We need to register undo buffers and set LSN for them
+ * that will be required for FPW of the undo buffers.
+ */
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, sizeof(xlrec));
+
+ (void) XLogInsert(RM_UNDOACTION_ID, XLOG_UNDO_APPLY_PROGRESS);
+ }
+
+ END_CRIT_SECTION();
+ UnlockReleaseUndoBuffers();
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+}
+
+/*
+ * execute_undo_actions_page - Execute the undo actions for a page
+ *
+ * After applying all the undo actions for a page, we clear the transaction
+ * slot on a page if the undo chain for block is complete, otherwise just
+ * rewind the undo pointer to the last record for that block that precedes
+ * the last undo record for which action is replayed.
+ *
+ * luinfo - list of undo records (along with their location) for which undo
+ * action needs to be replayed.
+ * urec_ptr - undo record pointer to which we need to rewind.
+ * reloid - OID of relation on which undo actions needs to be applied.
+ * blkno - block number on which undo actions needs to be applied.
+ * blk_chain_complete - indicates whether the undo chain for block is
+ * complete.
+ * nopartial - true if rollback is for complete transaction. If we are not
+ * rolling back the complete transaction then we need to apply the
+ * undo action for UNDO_INVALID_XACT_SLOT also because in such
+ * case we will rewind the insert undo location.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ * options - options for executing undo actions.
+ *
+ * returns true, if successfully applied the undo actions, otherwise, false.
+ */
+static bool
+execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock, int options)
+{
+ UndoRecInfo *first;
+
+ /*
+ * All records passed to us are for the same RMGR, so we just use the
+ * first record to dispatch.
+ */
+ Assert(luinfo != NIL);
+ first = (UndoRecInfo *) linitial(luinfo);
+
+ return RmgrTable[first->uur->uur_rmid].rm_undo(luinfo, urec_ptr, reloid,
+ xid, blkno,
+ blk_chain_complete, rellock,
+ options);
+}
+
+/*
+ * To return the size of the hash-table for rollbacks.
+ */
+int
+RollbackHTSize(void)
+{
+ return ROLLBACK_HT_SIZE * sizeof(RollbackHashEntry);
+}
+
+/*
+ * To initialize the hash-table for rollbacks in shared memory
+ * for the given size.
+ */
+void
+InitRollbackHashTable(void)
+{
+ int ht_size = RollbackHTSize();
+ HASHCTL info;
+ MemSet(&info, 0, sizeof(info));
+
+ info.keysize = sizeof(UndoRecPtr);
+ info.entrysize = sizeof(RollbackHashEntry);
+ info.hash = tag_hash;
+
+ RollbackHT = ShmemInitHash("Undo actions Lookup Table",
+ ht_size, ht_size, &info,
+ HASH_ELEM | HASH_FUNCTION);
+}
+
+/*
+ * To push the rollback requests from backend to the hash-table.
+ * Return true if the request is successfully added, else false
+ * and the caller may execute undo actions itself.
+ */
+bool
+PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr, Oid dbid)
+{
+ bool found = false;
+ RollbackHashEntry *rh;
+
+ Assert(UndoRecPtrGetLogNo(start_urec_ptr) != UndoRecPtrGetLogNo(end_urec_ptr) ||
+ start_urec_ptr >= end_urec_ptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ */
+ if (start_urec_ptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(end_urec_ptr);
+ start_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ Assert(UndoRecPtrIsValid(start_urec_ptr));
+
+ /* If there is no space to accomodate new request, then we can't proceed. */
+ if (RollbackHTIsFull())
+ return false;
+
+ if(!UndoRecPtrIsValid(end_urec_ptr))
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(start_urec_ptr);
+ end_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ rh = (RollbackHashEntry *) hash_search(RollbackHT, &start_urec_ptr,
+ HASH_ENTER_NULL, &found);
+ if (!rh)
+ {
+ LWLockRelease(RollbackHTLock);
+ return false;
+ }
+ /* We shouldn't try to push the same rollback request again. */
+ if (!found)
+ {
+ rh->start_urec_ptr = start_urec_ptr;
+ rh->end_urec_ptr = end_urec_ptr;
+ rh->dbid = (dbid == InvalidOid) ? MyDatabaseId : dbid;
+ }
+ LWLockRelease(RollbackHTLock);
+
+ return true;
+}
+
+/*
+ * To perform the undo actions for the transactions whose rollback
+ * requests are in hash table. Sequentially, scan the hash-table
+ * and perform the undo-actions for the respective transactions.
+ * Once, the undo-actions are applied, remove the entry from the
+ * hash table.
+ */
+void
+RollbackFromHT(Oid dbid)
+{
+ UndoRecPtr start[ROLLBACK_HT_SIZE];
+ UndoRecPtr end[ROLLBACK_HT_SIZE];
+ RollbackHashEntry *rh;
+ HASH_SEQ_STATUS status;
+ int i = 0;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ Assert(hash_get_num_entries(RollbackHT) <= ROLLBACK_HT_SIZE);
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ {
+ if (rh->dbid == dbid)
+ {
+ start[i] = rh->start_urec_ptr;
+ end[i] = rh->end_urec_ptr;
+ i++;
+ }
+ }
+
+ LWLockRelease(RollbackHTLock);
+
+ /* Execute the rollback requests */
+ while(--i >= 0)
+ {
+ Assert(UndoRecPtrIsValid(start[i]));
+ Assert(UndoRecPtrIsValid(end[i]));
+
+ StartTransactionCommand();
+ execute_undo_actions(start[i], end[i], true, false, true);
+ CommitTransactionCommand();
+ }
+}
+
+/*
+ * Remove the rollback request entry from the rollback hash table.
+ */
+static void
+RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr)
+{
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ hash_search(RollbackHT, &start_urec_ptr, HASH_REMOVE, NULL);
+
+ LWLockRelease(RollbackHTLock);
+}
+
+/*
+ * To check if the rollback requests in the hash table are all
+ * completed or not. This is required because we don't not want to
+ * expose RollbackHT in xact.c, where it is required to ensure
+ * that we push the resuests only when there is some space in
+ * the hash-table.
+ */
+bool
+RollbackHTIsFull(void)
+{
+ bool result = false;
+
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ if (hash_get_num_entries(RollbackHT) >= ROLLBACK_HT_SIZE)
+ result = true;
+
+ LWLockRelease(RollbackHTLock);
+
+ return result;
+}
+
+/*
+ * Get database list from the rollback hash table.
+ */
+List *
+RollbackHTGetDBList()
+{
+ HASH_SEQ_STATUS status;
+ RollbackHashEntry *rh;
+ List *dblist = NIL;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ dblist = list_append_unique_oid(dblist, rh->dbid);
+
+ LWLockRelease(RollbackHTLock);
+
+ return dblist;
+}
+
+/*
+ * ConditionTransactionUndoActionLock
+ *
+ * Insert a lock showing that the undo action for given transaction is in
+ * progress. This is only done for the main transaction not for the
+ * sub-transaction.
+ */
+bool
+ConditionTransactionUndoActionLock(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ if (LOCKACQUIRE_NOT_AVAIL == LockAcquire(&tag, ExclusiveLock, false, true))
+ return false;
+ else
+ return true;
+}
+
+/*
+ * TransactionUndoActionLockRelease
+ *
+ * Delete the lock showing that the undo action given transaction ID is in
+ * progress.
+ */
+void
+TransactionUndoActionLockRelease(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ LockRelease(&tag, ExclusiveLock, false);
+}
diff --git a/src/backend/access/undo/undoactionxlog.c b/src/backend/access/undo/undoactionxlog.c
new file mode 100644
index 0000000..546f43d
--- /dev/null
+++ b/src/backend/access/undo/undoactionxlog.c
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactionxlog.c
+ * WAL replay logic for undo actions.
+ *
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/undo/undoactionxlog.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+#include "access/undoinsert.h"
+#include "access/visibilitymap.h"
+#include "access/xlog.h"
+#include "access/xlogutils.h"
+
+/*
+ * Replay of undo apply progress.
+ */
+static void
+undo_xlog_apply_progress(XLogReaderState *record)
+{
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) XLogRecGetData(record);
+
+ /* Update the progress in the transaction header. */
+ PrepareUpdateUndoActionProgress(xlrec->urec_ptr, xlrec->progress);
+ UndoRecordUpdateTransInfo();
+}
+
+void
+undoaction_redo(XLogReaderState *record)
+{
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ switch (info)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ undo_xlog_apply_progress(record);
+ break;
+ default:
+ elog(PANIC, "undoaction_redo: unknown op code %u", info);
+ }
+}
diff --git a/src/backend/access/undo/undodiscard.c b/src/backend/access/undo/undodiscard.c
new file mode 100644
index 0000000..9d55588
--- /dev/null
+++ b/src/backend/access/undo/undodiscard.c
@@ -0,0 +1,458 @@
+/*-------------------------------------------------------------------------
+ *
+ * undodiscard.c
+ * discard undo records
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undodiscard.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/xlog.h"
+#include "access/xact.h"
+#include "access/undolog.h"
+#include "access/undodiscard.h"
+#include "catalog/pg_tablespace.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "storage/shmem.h"
+#include "storage/proc.h"
+#include "utils/resowner.h"
+#include "postmaster/undoloop.h"
+
+static UndoRecPtr FetchLatestUndoPtrForXid(UndoRecPtr urecptr,
+ UnpackedUndoRecord *uur_start,
+ UndoLogControl *log);
+
+/*
+ * Discard the undo for the log
+ *
+ * Search the undo log, get the start record for each transaction until we get
+ * the transaction with xid >= xmin or an invalid xid. Then call undolog
+ * routine to discard upto that point and update the memory structure for the
+ * log slot. We set the hibernate flag if we do not have any undo logs, this
+ * flag is passed to the undo worker wherein it determines if system is idle
+ * and it should sleep for sometime.
+ *
+ * Return the oldest xid remaining in this undo log (which should be >= xmin,
+ * since we'll discard everything older). Return InvalidTransactionId if the
+ * undo log is empty.
+ */
+static TransactionId
+UndoDiscardOneLog(UndoLogControl *log, TransactionId xmin, bool *hibernate)
+{
+ UndoRecPtr undo_recptr, next_insert, from_urecptr;
+ UndoRecPtr next_urecptr = InvalidUndoRecPtr;
+ UnpackedUndoRecord *uur = NULL;
+ bool need_discard = false;
+ bool log_complete = false;
+ TransactionId undoxid = InvalidTransactionId;
+ TransactionId xid = log->oldest_xid;
+ TransactionId latest_discardxid = InvalidTransactionId;
+ uint32 epoch = 0;
+
+ undo_recptr = log->oldest_data;
+
+ /* There might not be any undo log and hibernation might be needed. */
+ *hibernate = true;
+
+ /* Loop until we run out of discardable transactions. */
+ do
+ {
+ bool pending_abort = false;
+
+ next_insert = UndoLogGetNextInsertPtr(log->logno, xid);
+
+ /*
+ * If the next insert location in the undo log is same as the oldest
+ * data for the log then there is nothing more to discard in this log
+ * so discard upto this point.
+ */
+ if (next_insert == undo_recptr)
+ {
+ /*
+ * If the discard location and the insert location is same then
+ * there is nothing to discard.
+ */
+ if (undo_recptr == log->oldest_data)
+ break;
+ else
+ log_complete = true;
+ }
+ else
+ {
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+
+ Assert(uur != NULL);
+
+ if (!TransactionIdDidCommit(uur->uur_xid) &&
+ TransactionIdPrecedes(uur->uur_xid, xmin) &&
+ uur->uur_progress == 0)
+ {
+ /*
+ * At the time of recovery, we might not have a valid next undo
+ * record pointer and in that case we'll calculate the location
+ * of from pointer using the last record of next insert
+ * location.
+ */
+ if (ConditionTransactionUndoActionLock(uur->uur_xid))
+ {
+ TransactionId xid = uur->uur_xid;
+ UndoLogControl *log = NULL;
+ UndoLogNumber logno;
+
+ logno = UndoRecPtrGetLogNo(undo_recptr);
+ log = UndoLogGet(logno, false);
+
+ /*
+ * If the corresponding log got rewinded to a location
+ * prior to undo_recptr, the undo actions are already
+ * applied.
+ */
+ if (log->meta.insert > undo_recptr)
+ {
+ UndoRecordRelease(uur);
+
+ /* Fetch the undo record under undo action lock. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the undo actions for the aborted transaction is
+ * already applied then continue discarding the undo log
+ * otherwise discard till current point and stop processing
+ * this undo log.
+ * Also, check this is indeed the transaction id we're
+ * looking for. It is possible that after rewinding
+ * some other transaction has inserted an undo record.
+ */
+ if (uur->uur_xid == xid && uur->uur_progress == 0)
+ {
+ from_urecptr = FetchLatestUndoPtrForXid(undo_recptr, uur, log);
+ (void)PushRollbackReq(from_urecptr, undo_recptr, uur->uur_dbid);
+ pending_abort = true;
+ }
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+ else
+ pending_abort = true;
+ }
+
+ next_urecptr = uur->uur_next;
+ undoxid = uur->uur_xid;
+ xid = undoxid;
+ epoch = uur->uur_xidepoch;
+ }
+
+ /* we can discard upto this point. */
+ if (TransactionIdFollowsOrEquals(undoxid, xmin) ||
+ next_urecptr == SpecialUndoRecPtr ||
+ UndoRecPtrGetLogNo(next_urecptr) != log->logno ||
+ log_complete || pending_abort)
+ {
+ /* Hey, I got some undo log to discard, can not hibernate now. */
+ *hibernate = false;
+
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /*
+ * If Transaction id is smaller than the xmin that means this must
+ * be the last transaction in this undo log, so we need to get the
+ * last insert point in this undo log and discard till that point.
+ * Also, if the transaction has pending abort, we stop discarding
+ * undo from the same location.
+ */
+ if (TransactionIdPrecedes(undoxid, xmin) && !pending_abort)
+ {
+ UndoRecPtr next_insert = InvalidUndoRecPtr;
+
+ /*
+ * Get the last insert location for this transaction Id, if it
+ * returns invalid pointer that means there is new transaction
+ * has started for this undolog. So we need to refetch the undo
+ * and continue the process.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, undoxid);
+ if (!UndoRecPtrIsValid(next_insert))
+ continue;
+
+ undo_recptr = next_insert;
+ need_discard = true;
+ epoch = 0;
+ latest_discardxid = undoxid;
+ undoxid = InvalidTransactionId;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_EXCLUSIVE);
+
+ /*
+ * If no more pending undo logs then set the oldest transaction to
+ * InvalidTransactionId.
+ */
+ if (log_complete)
+ {
+ log->oldest_xid = InvalidTransactionId;
+ log->oldest_xidepoch = 0;
+ }
+ else
+ {
+ log->oldest_xid = undoxid;
+ log->oldest_xidepoch = epoch;
+ }
+
+ log->oldest_data = undo_recptr;
+ LWLockRelease(&log->discard_lock);
+
+ if (need_discard)
+ UndoLogDiscard(undo_recptr, latest_discardxid);
+
+ break;
+ }
+
+ /*
+ * This transaction is smaller than the xmin so lets jump to the next
+ * transaction.
+ */
+ undo_recptr = next_urecptr;
+ latest_discardxid = undoxid;
+
+ if(uur != NULL)
+ {
+ UndoRecordRelease(uur);
+ uur = NULL;
+ }
+
+ need_discard = true;
+ } while (true);
+
+ return undoxid;
+}
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+void
+UndoDiscard(TransactionId oldestXmin, bool *hibernate)
+{
+ TransactionId oldestXidHavingUndo = oldestXmin;
+ uint64 epoch = GetEpochForXid(oldestXmin);
+ UndoLogControl *log = NULL;
+
+ /*
+ * TODO: Ideally we'd arrange undo logs so that we can efficiently find
+ * those with oldest_xid < oldestXmin, but for now we'll just scan all of
+ * them.
+ */
+ while ((log = UndoLogNext(log)))
+ {
+ TransactionId oldest_xid = InvalidTransactionId;
+
+ /* We can't process temporary undo logs. */
+ if (log->meta.persistence == UNDO_TEMP)
+ continue;
+
+ /*
+ * If the first xid of the undo log is smaller than the xmin the try
+ * to discard the undo log.
+ */
+ if (TransactionIdPrecedes(log->oldest_xid, oldestXmin))
+ {
+ /*
+ * If the XID in the discard entry is invalid then start scanning
+ * from the first valid undorecord in the log.
+ */
+ if (!TransactionIdIsValid(log->oldest_xid))
+ {
+ bool full = false;
+ UndoRecPtr urp = UndoLogGetFirstValidRecord(log, &full);
+
+ if (!UndoRecPtrIsValid(urp))
+ {
+ /*
+ * There is nothing to be discarded. If there is also no
+ * more free space, then a call to UndoLogDiscard() will
+ * discard it the undo log completely and free up the
+ * UndoLogControl slot.
+ */
+ if (full)
+ UndoLogDiscard(MakeUndoRecPtr(log->meta.logno,
+ log->meta.discard),
+ InvalidTransactionId);
+ continue;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_SHARED);
+ log->oldest_data = urp;
+ LWLockRelease(&log->discard_lock);
+ }
+
+ /* Process the undo log. */
+ oldest_xid = UndoDiscardOneLog(log, oldestXmin, hibernate);
+ }
+
+ if (TransactionIdIsValid(oldest_xid) &&
+ TransactionIdPrecedes(oldest_xid, oldestXidHavingUndo))
+ {
+ oldestXidHavingUndo = oldest_xid;
+ epoch = GetEpochForXid(oldest_xid);
+ }
+ }
+
+ /*
+ * Update the oldestXidWithEpochHavingUndo in the shared memory.
+ *
+ * XXX In future if multiple worker can perform discard then we may need
+ * to use compare and swap for updating the shared memory value.
+ */
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ MakeEpochXid(epoch, oldestXidHavingUndo));
+}
+
+/*
+ * Fetch the latest urec pointer for the transaction.
+ */
+UndoRecPtr
+FetchLatestUndoPtrForXid(UndoRecPtr urecptr, UnpackedUndoRecord *uur_start,
+ UndoLogControl *log)
+{
+ UndoRecPtr next_urecptr, from_urecptr;
+ uint16 prevlen;
+ UndoLogOffset next_insert;
+ UnpackedUndoRecord *uur;
+ bool refetch = false;
+
+ uur = uur_start;
+
+ while (true)
+ {
+ /* fetch the undo record again if required. */
+ if (refetch)
+ {
+ uur = UndoFetchRecord(urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ refetch = false;
+ }
+
+ next_urecptr = uur->uur_next;
+ prevlen = UndoLogGetPrevLen(log->logno);
+
+ /*
+ * If this is the last transaction in the log then calculate the latest
+ * urec pointer using next insert location of the undo log. Otherwise,
+ * calculate using next transaction's start pointer.
+ */
+ if (uur->uur_next == SpecialUndoRecPtr)
+ {
+ /*
+ * While fetching the next insert location if the new transaction
+ * has already started in this log then lets re-fetch the undo
+ * record.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+ if (!UndoRecPtrIsValid(next_insert))
+ {
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ refetch = true;
+ continue;
+ }
+
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen);
+ break;
+ }
+ else if ((UndoRecPtrGetLogNo(next_urecptr) != log->logno) &&
+ UndoLogIsDiscarded(next_urecptr))
+ {
+ /*
+ * If next_urecptr is in different undolog and its already discarded
+ * that means the undo actions for this transaction which are in the
+ * next log has already been executed and we only need to execute
+ * which are remaining in this log.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+
+ Assert(UndoRecPtrIsValid(next_insert));
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen);
+ break;
+ }
+ else
+ {
+ UnpackedUndoRecord *next_uur;
+
+ next_uur = UndoFetchRecord(next_urecptr,
+ InvalidBlockNumber,
+ InvalidOffsetNumber,
+ InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the next_urecptr is in the same log then calculate the
+ * from pointer using prevlen.
+ */
+ if (UndoRecPtrGetLogNo(next_urecptr) == log->logno)
+ {
+ from_urecptr =
+ UndoGetPrevUndoRecptr(next_urecptr, next_uur->uur_prevlen);
+ UndoRecordRelease(next_uur);
+ break;
+ }
+ else
+ {
+ /*
+ * The transaction is overflowed to the next log, so restart
+ * the processing from then next log.
+ */
+ log = UndoLogGet(UndoRecPtrGetLogNo(next_urecptr), false);
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ uur = next_uur;
+ continue;
+ }
+
+ UndoRecordRelease(next_uur);
+ }
+ }
+
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+
+ return from_urecptr;
+}
+
+/*
+ * Discard the undo logs for temp tables.
+ */
+void
+TempUndoDiscard(UndoLogNumber logno)
+{
+ UndoLogControl *log = UndoLogGet(logno, false);
+
+ /*
+ * Discard the undo log for temp table only. Ensure that there is
+ * something to be discarded there.
+ */
+ Assert (log->meta.persistence == UNDO_TEMP);
+
+ /* Process the undo log. */
+ UndoLogDiscard(MakeUndoRecPtr(log->logno, log->meta.insert),
+ InvalidTransactionId);
+}
diff --git a/src/backend/access/undo/undoinsert.c b/src/backend/access/undo/undoinsert.c
index 2453cad..178b01b 100644
--- a/src/backend/access/undo/undoinsert.c
+++ b/src/backend/access/undo/undoinsert.c
@@ -133,7 +133,6 @@ static UnpackedUndoRecord* UndoGetOneRecord(UnpackedUndoRecord *urec,
UndoPersistence persistence);
static void PrepareUndoRecordUpdateTransInfo(UndoRecPtr urecptr,
bool log_switched);
-static void UndoRecordUpdateTransInfo(void);
static int InsertFindBufferSlot(RelFileNode rnode, BlockNumber blk,
ReadBufferMode rbm,
UndoPersistence persistence);
@@ -271,12 +270,60 @@ PrepareUndoRecordUpdateTransInfo(UndoRecPtr urecptr, bool log_switched)
}
/*
+ * Update the progress of the undo record in the transaction header.
+ */
+void
+PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress)
+{
+ Buffer buffer = InvalidBuffer;
+ BlockNumber cur_blk;
+ RelFileNode rnode;
+ UndoLogNumber logno = UndoRecPtrGetLogNo(urecptr);
+ UndoLogControl *log;
+ Page page;
+ int already_decoded = 0;
+ int starting_byte;
+ int bufidx;
+ int index = 0;
+
+ log = UndoLogGet(logno, false);
+
+ if (log->meta.persistence == UNDO_TEMP)
+ return;
+
+ UndoRecPtrAssignRelFileNode(rnode, urecptr);
+ cur_blk = UndoRecPtrGetBlockNum(urecptr);
+ starting_byte = UndoRecPtrGetPageOffset(urecptr);
+
+ while (true)
+ {
+ bufidx = InsertFindBufferSlot(rnode, cur_blk,
+ RBM_NORMAL,
+ log->meta.persistence);
+ prev_txn_info.prev_txn_undo_buffers[index] = bufidx;
+ buffer = undo_buffer[bufidx].buf;
+ page = BufferGetPage(buffer);
+ index++;
+
+ if (UnpackUndoRecord(&prev_txn_info.uur, page, starting_byte,
+ &already_decoded, true))
+ break;
+
+ starting_byte = UndoLogBlockHeaderSize;
+ cur_blk++;
+ }
+
+ prev_txn_info.prev_urecptr = urecptr;
+ prev_txn_info.uur.uur_progress = progress;
+}
+
+/*
* Overwrite the first undo record of the previous transaction to update its
* next pointer. This will just insert the already prepared record by
* PrepareUndoRecordUpdateTransInfo. This must be called under the critical
* section. This will just overwrite the undo header not the data.
*/
-static void
+void
UndoRecordUpdateTransInfo(void)
{
UndoLogNumber logno = UndoRecPtrGetLogNo(prev_txn_info.prev_urecptr);
diff --git a/src/backend/access/undo/undorecord.c b/src/backend/access/undo/undorecord.c
index 33bb153..5928784 100644
--- a/src/backend/access/undo/undorecord.c
+++ b/src/backend/access/undo/undorecord.c
@@ -95,6 +95,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
*/
if (*already_written == 0)
{
+ work_hdr.urec_rmid = uur->uur_rmid;
work_hdr.urec_type = uur->uur_type;
work_hdr.urec_info = uur->uur_info;
work_hdr.urec_prevlen = uur->uur_prevlen;
@@ -120,6 +121,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
* We should have been passed the same record descriptor as before,
* or caller has messed up.
*/
+ Assert(work_hdr.urec_rmid == uur->uur_rmid);
Assert(work_hdr.urec_type == uur->uur_type);
Assert(work_hdr.urec_info == uur->uur_info);
Assert(work_hdr.urec_prevlen == uur->uur_prevlen);
@@ -282,6 +284,7 @@ bool UnpackUndoRecord(UnpackedUndoRecord *uur, Page page, int starting_byte,
&my_bytes_decoded, already_decoded, false))
return false;
+ uur->uur_rmid = work_hdr.urec_rmid;
uur->uur_type = work_hdr.urec_type;
uur->uur_info = work_hdr.urec_info;
uur->uur_prevlen = work_hdr.urec_prevlen;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index a86963f..8548a65 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1381,6 +1381,7 @@ vac_truncate_clog(TransactionId frozenXID,
MultiXactId lastSaneMinMulti)
{
TransactionId nextXID = ReadNewTransactionId();
+ TransactionId oldestXidHavingUndo;
Relation relation;
HeapScanDesc scan;
HeapTuple tuple;
@@ -1475,6 +1476,16 @@ vac_truncate_clog(TransactionId frozenXID,
return;
/*
+ * We can't truncate the clog for transactions that still have undo. The
+ * oldestXidHavingUndo will be only valid for zheap storage engine, so it
+ * won't impact any other storage engine.
+ */
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ frozenXID = Min(frozenXID, oldestXidHavingUndo);
+
+ /*
* Advance the oldest value for commit timestamps before truncating, so
* that if a user requests a timestamp for a transaction we're truncating
* away right after this point, they get NULL instead of an ugly "file not
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 71c2321..9ce6ff0 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -12,7 +12,7 @@ subdir = src/backend/postmaster
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
- pgarch.o pgstat.o postmaster.o startup.o syslogger.o walwriter.o
+OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o discardworker.o fork_process.o \
+ pgarch.o pgstat.o postmaster.o startup.o syslogger.o undoworker.o walwriter.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index d2b695e..49df516 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -20,7 +20,9 @@
#include "pgstat.h"
#include "port/atomics.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "storage/dsm.h"
@@ -129,6 +131,15 @@ static const struct
},
{
"ApplyWorkerMain", ApplyWorkerMain
+ },
+ {
+ "UndoLauncherMain", UndoLauncherMain
+ },
+ {
+ "UndoWorkerMain", UndoWorkerMain
+ },
+ {
+ "DiscardWorkerMain", DiscardWorkerMain
}
};
diff --git a/src/backend/postmaster/discardworker.c b/src/backend/postmaster/discardworker.c
new file mode 100644
index 0000000..e4c6719
--- /dev/null
+++ b/src/backend/postmaster/discardworker.c
@@ -0,0 +1,170 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.c
+ * The undo discard worker for asynchronous undo management.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/discardworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include <unistd.h>
+
+/* These are always necessary for a bgworker. */
+#include "access/transam.h"
+#include "miscadmin.h"
+#include "postmaster/bgworker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/lwlock.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+#include "access/undodiscard.h"
+#include "pgstat.h"
+#include "postmaster/discardworker.h"
+#include "storage/procarray.h"
+#include "tcop/tcopprot.h"
+#include "utils/guc.h"
+#include "utils/resowner.h"
+
+static void undoworker_sigterm_handler(SIGNAL_ARGS);
+
+/* max sleep time between cycles (100 milliseconds) */
+#define MIN_NAPTIME_PER_CYCLE 100L
+#define DELAYED_NAPTIME 10 * MIN_NAPTIME_PER_CYCLE
+#define MAX_NAPTIME_PER_CYCLE 100 * MIN_NAPTIME_PER_CYCLE
+
+static bool got_SIGTERM = false;
+static bool hibernate = false;
+static long wait_time = MIN_NAPTIME_PER_CYCLE;
+
+/* SIGTERM: set flag to exit at next convenient time */
+static void
+undoworker_sigterm_handler(SIGNAL_ARGS)
+{
+ got_SIGTERM = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+}
+
+/*
+ * DiscardWorkerRegister -- Register a undo discard worker.
+ */
+void
+DiscardWorkerRegister(void)
+{
+ BackgroundWorker bgw;
+
+ /* TODO: This should be configurable. */
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "discard worker");
+ sprintf(bgw.bgw_library_name, "postgres");
+ sprintf(bgw.bgw_function_name, "DiscardWorkerMain");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * DiscardWorkerMain -- Main loop for the undo discard worker.
+ */
+void
+DiscardWorkerMain(Datum main_arg)
+{
+ ereport(LOG,
+ (errmsg("discard worker started")));
+
+ /* Establish signal handlers. */
+ pqsignal(SIGTERM, undoworker_sigterm_handler);
+ BackgroundWorkerUnblockSignals();
+
+ /* Make it easy to identify our processes. */
+ SetConfigOption("application_name", MyBgworkerEntry->bgw_name,
+ PGC_USERSET, PGC_S_SESSION);
+
+ /*
+ * Create resource owner for discard worker as it need to read the undo
+ * records outside the transaction blocks which intern access buffer read
+ * routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ /* Enter main loop */
+ while (!got_SIGTERM)
+ {
+ int rc;
+ TransactionId OldestXmin, oldestXidHavingUndo;
+
+ OldestXmin = GetOldestXmin(NULL, PROCARRAY_FLAGS_DEFAULT);
+
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+
+ /*
+ * Call the discard routine if there oldestXidHavingUndo is lagging
+ * behind OldestXmin.
+ */
+ if (OldestXmin != InvalidTransactionId &&
+ TransactionIdPrecedes(oldestXidHavingUndo, OldestXmin))
+ {
+ UndoDiscard(OldestXmin, &hibernate);
+
+ /*
+ * If we got some undo logs to discard or discarded something,
+ * then reset the wait_time as we have got work to do.
+ * Note that if there are some undologs that cannot be discarded,
+ * then above condition will remain unsatisified till oldestXmin
+ * remains unchanged and the wait_time will not reset in that case.
+ */
+ if (!hibernate)
+ wait_time = MIN_NAPTIME_PER_CYCLE;
+ }
+
+ /* Wait for more work. */
+ rc = WaitLatch(&MyProc->procLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wait_time,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN);
+
+ ResetLatch(&MyProc->procLatch);
+
+ /*
+ * Increase the wait_time based on the length of inactivity. If wait_time
+ * is within one second, then increment it by 100 ms at a time. Henceforth,
+ * increment it one second at a time, till it reaches ten seconds. Never
+ * increase the wait_time more than ten seconds, it will be too much of
+ * waiting otherwise.
+ */
+ if (rc & WL_TIMEOUT && hibernate)
+ {
+ wait_time += (wait_time < DELAYED_NAPTIME ?
+ MIN_NAPTIME_PER_CYCLE : DELAYED_NAPTIME);
+ if (wait_time > MAX_NAPTIME_PER_CYCLE)
+ wait_time = MAX_NAPTIME_PER_CYCLE;
+ }
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+
+ ReleaseAuxProcessResources(true);
+
+ /* we're done */
+ ereport(LOG,
+ (errmsg("discard worker shutting down")));
+
+ proc_exit(0);
+}
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index dc86307..ab3498f 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -3502,6 +3502,12 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_SYSLOGGER_MAIN:
event_name = "SysLoggerMain";
break;
+ case WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN:
+ event_name = "UndoDiscardWorkerMain";
+ break;
+ case WAIT_EVENT_UNDO_LAUNCHER_MAIN:
+ event_name = "UndoLauncherMain";
+ break;
case WAIT_EVENT_WAL_RECEIVER_MAIN:
event_name = "WalReceiverMain";
break;
@@ -3914,7 +3920,12 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_UNDO_FILE_SYNC:
event_name = "UndoFileSync";
break;
-
+ case WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN:
+ event_name = "UndoDiscardWorkerMain";
+ break;
+ case WAIT_EVENT_UNDO_LAUNCHER_MAIN:
+ event_name = "UndoLauncherMain";
+ break;
case WAIT_EVENT_WALSENDER_TIMELINE_HISTORY_READ:
event_name = "WALSenderTimelineHistoryRead";
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 688f462..603c733 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -111,10 +111,12 @@
#include "port/pg_bswap.h"
#include "postmaster/autovacuum.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/walsender.h"
#include "storage/fd.h"
@@ -978,6 +980,11 @@ PostmasterMain(int argc, char *argv[])
*/
ApplyLauncherRegister();
+ UndoLauncherRegister();
+
+ /* Register the Undo Discard worker. */
+ DiscardWorkerRegister();
+
/*
* process any libraries that should be preloaded at postmaster start
*/
diff --git a/src/backend/postmaster/undoworker.c b/src/backend/postmaster/undoworker.c
new file mode 100644
index 0000000..caa7db1
--- /dev/null
+++ b/src/backend/postmaster/undoworker.c
@@ -0,0 +1,664 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.c
+ * undo launcher and undo worker process.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/undoworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "funcapi.h"
+#include "miscadmin.h"
+#include "pgstat.h"
+
+#include "access/heapam.h"
+#include "access/htup.h"
+#include "access/htup_details.h"
+#include "access/sysattr.h"
+#include "access/xact.h"
+
+#include "catalog/indexing.h"
+#include "catalog/pg_database.h"
+
+#include "libpq/pqsignal.h"
+
+#include "postmaster/bgworker.h"
+#include "postmaster/fork_process.h"
+#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
+
+#include "replication/slot.h"
+#include "replication/worker_internal.h"
+
+#include "storage/ipc.h"
+#include "storage/lmgr.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+
+#include "tcop/tcopprot.h"
+
+#include "utils/fmgroids.h"
+#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/resowner.h"
+
+/* max sleep time between cycles (100 milliseconds) */
+#define DEFAULT_NAPTIME_PER_CYCLE 100L
+#define DEFAULT_RETRY_NAPTIME 50L
+
+int max_undo_workers = 5;
+
+typedef struct UndoApplyWorker
+{
+ /* Indicates if this slot is used or free. */
+ bool in_use;
+
+ /* Increased everytime the slot is taken by new worker. */
+ uint16 generation;
+
+ /* Pointer to proc array. NULL if not running. */
+ PGPROC *proc;
+
+ /* Database id to connect to. */
+ Oid dbid;
+} UndoApplyWorker;
+
+UndoApplyWorker *MyUndoWorker = NULL;
+
+typedef struct UndoApplyCtxStruct
+{
+ /* Supervisor process. */
+ pid_t launcher_pid;
+
+ /* Background workers. */
+ UndoApplyWorker workers[FLEXIBLE_ARRAY_MEMBER];
+} UndoApplyCtxStruct;
+
+UndoApplyCtxStruct *UndoApplyCtx;
+
+static void undo_worker_onexit(int code, Datum arg);
+static void undo_worker_cleanup(UndoApplyWorker *worker);
+
+static volatile sig_atomic_t got_SIGHUP = false;
+
+/*
+ * Wait for a background worker to start up and attach to the shmem context.
+ *
+ * This is only needed for cleaning up the shared memory in case the worker
+ * fails to attach.
+ */
+static void
+WaitForUndoWorkerAttach(UndoApplyWorker *worker,
+ uint16 generation,
+ BackgroundWorkerHandle *handle)
+{
+ BgwHandleStatus status;
+ int rc;
+
+ for (;;)
+ {
+ pid_t pid;
+
+ CHECK_FOR_INTERRUPTS();
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+
+ /* Worker either died or has started; no need to do anything. */
+ if (!worker->in_use || worker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ LWLockRelease(UndoWorkerLock);
+
+ /* Check if worker has died before attaching, and clean up after it. */
+ status = GetBackgroundWorkerPid(handle, &pid);
+
+ if (status == BGWH_STOPPED)
+ {
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ /* Ensure that this was indeed the worker we waited for. */
+ if (generation == worker->generation)
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ /*
+ * We need timeout because we generally don't get notified via latch
+ * about the worker attach. But we don't expect to have to wait long.
+ */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 10L, WAIT_EVENT_BGWORKER_STARTUP);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+ }
+
+ return;
+}
+
+/*
+ * Get dbid from the worker slot.
+ */
+static Oid
+slot_get_dbid(int slot)
+{
+ Oid dbid;
+
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty,",
+ slot)));
+ }
+
+ dbid = MyUndoWorker->dbid;
+
+ LWLockRelease(UndoWorkerLock);
+
+ return dbid;
+}
+
+/*
+ * Attach to a slot.
+ */
+static void
+undo_worker_attach(int slot)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty, cannot attach",
+ slot)));
+ }
+
+ if (MyUndoWorker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is already used by "
+ "another worker, cannot attach", slot)));
+ }
+
+ MyUndoWorker->proc = MyProc;
+ before_shmem_exit(undo_worker_onexit, (Datum) 0);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Walks the workers array and searches for one that matches given
+ * dbid.
+ */
+static UndoApplyWorker *
+undo_worker_find(Oid dbid)
+{
+ int i;
+ UndoApplyWorker *res = NULL;
+
+ Assert(LWLockHeldByMe(UndoWorkerLock));
+
+ /* Search for attached worker for a given db id. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (w->in_use && w->dbid == dbid)
+ {
+ res = w;
+ break;
+ }
+ }
+
+ return res;
+}
+
+/*
+ * Check whether the dbid exist or not.
+ *
+ * Refer comments from GetDatabaseTupleByOid.
+ * FIXME: Should we expose GetDatabaseTupleByOid and directly use it.
+ */
+static bool
+dbid_exist(Oid dboid)
+{
+ HeapTuple tuple;
+ Relation relation;
+ SysScanDesc scan;
+ ScanKeyData key[1];
+ bool result = false;
+ /*
+ * form a scan key
+ */
+ ScanKeyInit(&key[0],
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dboid));
+
+ relation = heap_open(DatabaseRelationId, AccessShareLock);
+ scan = systable_beginscan(relation, DatabaseOidIndexId,
+ criticalSharedRelcachesBuilt,
+ NULL,
+ 1, key);
+
+ tuple = systable_getnext(scan);
+
+ if (HeapTupleIsValid(tuple))
+ result = true;
+
+ /* all done */
+ systable_endscan(scan);
+ heap_close(relation, AccessShareLock);
+
+ return result;
+}
+
+/*
+ * Start new undo apply background worker, if possible otherwise return false.
+ */
+static bool
+undo_worker_launch(Oid dbid)
+{
+ BackgroundWorker bgw;
+ BackgroundWorkerHandle *bgw_handle;
+ uint16 generation;
+ int i;
+ int slot = 0;
+ UndoApplyWorker *worker = NULL;
+
+ /*
+ * We need to do the modification of the shared memory under lock so that
+ * we have consistent view.
+ */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ /* Find unused worker slot. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (!w->in_use)
+ {
+ worker = w;
+ slot = i;
+ break;
+ }
+ }
+
+ /* There are no more free worker slots */
+ if (worker == NULL)
+ return false;
+
+ /* Prepare the worker slot. */
+ worker->in_use = true;
+ worker->proc = NULL;
+ worker->dbid = dbid;
+ worker->generation++;
+
+ generation = worker->generation;
+ LWLockRelease(UndoWorkerLock);
+
+ /* Register the new dynamic worker. */
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoWorkerMain");
+ snprintf(bgw.bgw_type, BGW_MAXLEN, "undo apply worker");
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "undo apply worker");
+
+ bgw.bgw_restart_time = BGW_NEVER_RESTART;
+ bgw.bgw_notify_pid = MyProcPid;
+ bgw.bgw_main_arg = Int32GetDatum(slot);
+
+ StartTransactionCommand();
+ /* Check the database exists or not. */
+ if (!dbid_exist(dbid))
+ {
+ CommitTransactionCommand();
+
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return true;
+ }
+
+ /*
+ * Acquire database object lock before launching the worker so that it
+ * doesn't get dropped while worker is connecting to the database.
+ */
+ LockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+
+ /* Recheck whether database still exists or not. */
+ if (!dbid_exist(dbid))
+ {
+ CommitTransactionCommand();
+
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return true;
+ }
+
+ if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle))
+ {
+ /* Failed to start worker, so clean up the worker slot. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+
+ UnlockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+ CommitTransactionCommand();
+
+ return false;
+ }
+
+ /* Now wait until it attaches. */
+ WaitForUndoWorkerAttach(worker, generation, bgw_handle);
+
+ /*
+ * By this point the undo-worker has already connected to the database so we
+ * can release the database lock.
+ */
+ UnlockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+ CommitTransactionCommand();
+
+ return true;
+}
+
+/*
+ * Detach the worker (cleans up the worker info).
+ */
+static void
+undo_worker_detach(void)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ undo_worker_cleanup(MyUndoWorker);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Clean up worker info.
+ */
+static void
+undo_worker_cleanup(UndoApplyWorker *worker)
+{
+ Assert(LWLockHeldByMeInMode(UndoWorkerLock, LW_EXCLUSIVE));
+
+ worker->in_use = false;
+ worker->proc = NULL;
+ worker->dbid = InvalidOid;
+}
+
+/*
+ * Cleanup function for undo worker launcher.
+ *
+ * Called on undo worker launcher exit.
+ */
+static void
+undo_launcher_onexit(int code, Datum arg)
+{
+ UndoApplyCtx->launcher_pid = 0;
+}
+
+/* SIGHUP: set flag to reload configuration at next convenient time */
+static void
+undo_launcher_sighup(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGHUP = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+undo_worker_onexit(int code, Datum arg)
+{
+ undo_worker_detach();
+}
+
+/*
+ * UndoLauncherShmemSize
+ * Compute space needed for undo launcher shared memory
+ */
+Size
+UndoLauncherShmemSize(void)
+{
+ Size size;
+
+ /*
+ * Need the fixed struct and the array of LogicalRepWorker.
+ */
+ size = sizeof(UndoApplyCtxStruct);
+ size = MAXALIGN(size);
+ size = add_size(size, mul_size(max_undo_workers,
+ sizeof(UndoApplyWorker)));
+ return size;
+}
+
+/*
+ * UndoLauncherRegister
+ * Register a background worker running the undo worker launcher.
+ */
+void
+UndoLauncherRegister(void)
+{
+ BackgroundWorker bgw;
+
+ if (max_undo_workers == 0)
+ return;
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoLauncherMain");
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "undo worker launcher");
+ snprintf(bgw.bgw_type, BGW_MAXLEN,
+ "undo worker launcher");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * UndoLauncherShmemInit
+ * Allocate and initialize undo worker launcher shared memory
+ */
+void
+UndoLauncherShmemInit(void)
+{
+ bool found;
+
+ UndoApplyCtx = (UndoApplyCtxStruct *)
+ ShmemInitStruct("Undo Worker Launcher Data",
+ UndoLauncherShmemSize(),
+ &found);
+
+ if (!found)
+ memset(UndoApplyCtx, 0, UndoLauncherShmemSize());
+}
+
+/*
+ * Main loop for the undo worker launcher process.
+ */
+void
+UndoLauncherMain(Datum main_arg)
+{
+ MemoryContext tmpctx;
+ MemoryContext oldctx;
+
+ ereport(DEBUG1,
+ (errmsg("undo launcher started")));
+
+ before_shmem_exit(undo_launcher_onexit, (Datum) 0);
+
+ Assert(UndoApplyCtx->launcher_pid == 0);
+ UndoApplyCtx->launcher_pid = MyProcPid;
+
+ /* Establish signal handlers. */
+ pqsignal(SIGHUP, undo_launcher_sighup);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * Establish connection to nailed catalogs (we only ever access
+ * pg_subscription).
+ */
+ BackgroundWorkerInitializeConnection(NULL, NULL, 0);
+
+ /* Use temporary context for the database list and worker info. */
+ tmpctx = AllocSetContextCreate(TopMemoryContext,
+ "Undo worker Launcher context",
+ ALLOCSET_DEFAULT_SIZES);
+ /* Enter main loop */
+ for (;;)
+ {
+ int rc;
+ List *dblist;
+ ListCell *l;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* switch to the temp context. */
+ oldctx = MemoryContextSwitchTo(tmpctx);
+ dblist = RollbackHTGetDBList();
+
+ foreach(l, dblist)
+ {
+ UndoApplyWorker *w;
+ Oid dbid = lfirst_oid(l);
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+ w = undo_worker_find(dbid);
+ LWLockRelease(UndoWorkerLock);
+
+ if (w == NULL)
+ {
+retry:
+ if (!undo_worker_launch(dbid))
+ {
+ /* Could not launch the worker, retry after sometime, */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_RETRY_NAPTIME,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+ goto retry;
+ }
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+
+ /* Clean the temporary memory. */
+ MemoryContextReset(tmpctx);
+
+ /* Wait for more work. */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_NAPTIME_PER_CYCLE,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ if (got_SIGHUP)
+ {
+ got_SIGHUP = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+}
+
+/*
+ * UndoWorkerMain -- Main loop for the undo apply worker.
+ */
+void
+UndoWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ Oid dbid;
+
+ dbid = slot_get_dbid(worker_slot);
+
+ /* Setup signal handling */
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /* Connect to the database. */
+ BackgroundWorkerInitializeConnectionByOid(dbid, 0, 0);
+
+ /* Attach to slot */
+ undo_worker_attach(worker_slot);
+
+ /*
+ * Create resource owner for undo worker. Undo worker need this as it
+ * need to read the undo records outside the transaction blocks which
+ * intern access buffer read routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ RollbackFromHT(dbid);
+
+ ReleaseAuxProcessResources(true);
+
+ proc_exit(0);
+}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index f60ecc5..49e1028 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -159,6 +159,10 @@ LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *recor
ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(record),
buf.origptr);
break;
+ case RM_UNDOACTION_ID:
+ /* Logical decoding is not yet implemented for undoactions. */
+ Assert(0);
+ break;
case RM_NEXT_ID:
elog(ERROR, "unexpected RM_NEXT_ID rmgr_id: %u", (RmgrIds) XLogRecGetRmid(buf.record));
}
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 4725cbe..1470041 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -29,6 +29,8 @@
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
#include "replication/walreceiver.h"
@@ -152,6 +154,8 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
size = add_size(size, BackendRandomShmemSize());
+ size = add_size(size, RollbackHTSize());
+ size = add_size(size, UndoLauncherShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -226,6 +230,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
SUBTRANSShmemInit();
MultiXactShmemInit();
InitBufferPool();
+ InitRollbackHashTable();
/*
* Set up lock manager
@@ -264,6 +269,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
WalSndShmemInit();
WalRcvShmemInit();
ApplyLauncherShmemInit();
+ UndoLauncherShmemInit();
/*
* Set up other modules that need some shared memory space
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 554af46..52f6959 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -51,3 +51,5 @@ BackendRandomLock 43
LogicalRepWorkerLock 44
CLogTruncationLock 45
UndoLogLock 46
+RollbackHTLock 47
+UndoWorkerLock 48
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 6f9aaa5..3138b04 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -279,6 +279,8 @@ InitProcGlobal(void)
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemAlloc(sizeof(slock_t));
SpinLockInit(ProcStructLock);
+
+ pg_atomic_init_u64(&ProcGlobal->oldestXidWithEpochHavingUndo, 0);
}
/*
diff --git a/src/backend/utils/adt/lockfuncs.c b/src/backend/utils/adt/lockfuncs.c
index 66c09a1..67dda39 100644
--- a/src/backend/utils/adt/lockfuncs.c
+++ b/src/backend/utils/adt/lockfuncs.c
@@ -32,6 +32,7 @@ const char *const LockTagTypeNames[] = {
"virtualxid",
"speculative token",
"object",
+ "undoaction",
"userlock",
"advisory"
};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index c693977..12e7704 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -121,6 +121,7 @@ bool allowSystemTableMods = false;
int work_mem = 1024;
int maintenance_work_mem = 16384;
int max_parallel_maintenance_workers = 2;
+int rollback_overflow_size = 64;
/*
* Primary determinants of sizes of shared-memory structures.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 287ca00..85a7f11 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2806,6 +2806,17 @@ static struct config_int ConfigureNamesInt[] =
},
{
+ {"rollback_overflow_size", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Rollbacks greater than this size are done lazily"),
+ NULL,
+ GUC_UNIT_MB
+ },
+ &rollback_overflow_size,
+ 64, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
{"wal_segment_size", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the size of write ahead log segments."),
NULL,
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index 3fc8b6a..5a5d98c 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -78,8 +78,8 @@ pg_control_system(PG_FUNCTION_ARGS)
Datum
pg_control_checkpoint(PG_FUNCTION_ARGS)
{
- Datum values[19];
- bool nulls[19];
+ Datum values[20];
+ bool nulls[20];
TupleDesc tupdesc;
HeapTuple htup;
ControlFileData *ControlFile;
@@ -128,6 +128,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
XIDOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 18, "checkpoint_time",
TIMESTAMPTZOID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 19, "oldest_xid_with_epoch_having_undo",
+ TIMESTAMPTZOID, -1, 0);
tupdesc = BlessTupleDesc(tupdesc);
/* Read the control file. */
@@ -202,6 +204,9 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
time_t_to_timestamptz(ControlFile->checkPointCopy.time));
nulls[17] = false;
+ values[18] = Int64GetDatum(ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo);
+ nulls[18] = false;
+
htup = heap_form_tuple(tupdesc, values, nulls);
PG_RETURN_DATUM(HeapTupleGetDatum(htup));
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 4e61bc6..f3bf0fd 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -686,4 +686,11 @@
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
+# If often there are large transactions requiring rollbacks, then we can push
+# them to undo-workers for better performance. The size specifeid by the
+# parameter below, determines the minimum size of the rollback requests to be
+# sent to the undo-worker.
+#
+#rollback_overflow_size = 64
+
# Add settings for extensions here
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 895a51f..753929e 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -278,6 +278,8 @@ main(int argc, char *argv[])
ControlFile->checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile->checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo);
printf(_("Time of latest checkpoint: %s\n"),
ckpttime_str);
printf(_("Fake LSN counter for unlogged rels: %X/%X\n"),
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 6fb403a..72714dd 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -448,6 +448,7 @@ main(int argc, char *argv[])
if (ControlFile.checkPointCopy.oldestXid < FirstNormalTransactionId)
ControlFile.checkPointCopy.oldestXid += FirstNormalTransactionId;
ControlFile.checkPointCopy.oldestXidDB = InvalidOid;
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo = 0;
}
if (set_oldest_commit_ts_xid != 0)
@@ -716,6 +717,8 @@ GuessControlValues(void)
ControlFile.checkPointCopy.oldestMultiDB = InvalidOid;
ControlFile.checkPointCopy.time = (pg_time_t) time(NULL);
ControlFile.checkPointCopy.oldestActiveXid = InvalidTransactionId;
+ ControlFile.checkPointCopy.nextXid = 0;
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo = 0;
ControlFile.state = DB_SHUTDOWNED;
ControlFile.time = (pg_time_t) time(NULL);
@@ -808,6 +811,8 @@ PrintControlValues(bool guessed)
ControlFile.checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile.checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo);
printf(_("Maximum data alignment: %u\n"),
ControlFile.maxAlign);
/* we don't print floatFormat since can't say much useful about it */
@@ -884,6 +889,8 @@ PrintNewControlValues(void)
ControlFile.checkPointCopy.oldestXid);
printf(_("OldestXID's DB: %u\n"),
ControlFile.checkPointCopy.oldestXidDB);
+ printf(_("OldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo);
}
if (set_xid_epoch != -1)
diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c
index 4002847..e71a71e 100644
--- a/src/bin/pg_rewind/parsexlog.c
+++ b/src/bin/pg_rewind/parsexlog.c
@@ -29,7 +29,7 @@
* RmgrNames is an array of resource manager names, to make error messages
* a bit nicer.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
name,
static const char *RmgrNames[RM_MAX_ID + 1] = {
diff --git a/src/bin/pg_waldump/rmgrdesc.c b/src/bin/pg_waldump/rmgrdesc.c
index 938150d..3ef0a60 100644
--- a/src/bin/pg_waldump/rmgrdesc.c
+++ b/src/bin/pg_waldump/rmgrdesc.c
@@ -20,6 +20,7 @@
#include "access/nbtxlog.h"
#include "access/rmgr.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -33,7 +34,7 @@
#include "storage/standbydefs.h"
#include "utils/relmapper.h"
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
{ name, desc, identify},
const RmgrDescData RmgrDescTable[RM_MAX_ID + 1] = {
diff --git a/src/include/access/rmgr.h b/src/include/access/rmgr.h
index c9b5c56..e1fb42a 100644
--- a/src/include/access/rmgr.h
+++ b/src/include/access/rmgr.h
@@ -19,7 +19,7 @@ typedef uint8 RmgrId;
* Note: RM_MAX_ID must fit in RmgrId; widening that type will affect the XLOG
* file format.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
symname,
typedef enum RmgrIds
diff --git a/src/include/access/rmgrlist.h b/src/include/access/rmgrlist.h
index 9c6fca4..eee7835 100644
--- a/src/include/access/rmgrlist.h
+++ b/src/include/access/rmgrlist.h
@@ -25,26 +25,27 @@
*/
/* symbol name, textual name, redo, desc, identify, startup, cleanup */
-PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL)
-PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL)
-PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL)
-PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL)
-PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL)
-PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask)
-PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask)
-PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask)
-PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask)
-PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask)
-PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask)
-PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask)
-PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL)
-PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL)
-PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask)
-PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL)
-PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL)
+PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask, NULL, NULL)
+PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask, NULL, NULL)
+PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask, NULL, NULL)
+PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask, NULL, NULL)
+PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask, NULL, NULL)
+PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask, NULL, NULL)
+PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask, NULL, NULL)
+PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask, NULL, NULL)
+PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOACTION_ID, "UndoAction", undoaction_redo, undoaction_desc, undoaction_identify, NULL, NULL, NULL, NULL, NULL)
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 83ec3f1..7b983ef 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -68,6 +68,10 @@
(AssertMacro(TransactionIdIsNormal(id1) && TransactionIdIsNormal(id2)), \
(int32) ((id1) - (id2)) > 0)
+/* Extract xid from a value comprised of epoch and xid */
+#define GetXidFromEpochXid(epochxid) \
+ ((uint32) (epochxid) & 0XFFFFFFFF)
+
/* ----------
* Object ID (OID) zero is InvalidOid.
*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 0e932da..fb05c3b 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -41,7 +41,7 @@ extern GlobalTransaction MarkAsPreparing(TransactionId xid, const char *gid,
TimestampTz prepared_at,
Oid owner, Oid databaseid);
-extern void StartPrepare(GlobalTransaction gxact);
+extern void StartPrepare(GlobalTransaction gxact, UndoRecPtr *, UndoRecPtr *);
extern void EndPrepare(GlobalTransaction gxact);
extern bool StandbyTransactionIdIsPrepared(TransactionId xid);
diff --git a/src/include/access/undoaction.h b/src/include/access/undoaction.h
new file mode 100644
index 0000000..5455259
--- /dev/null
+++ b/src/include/access/undoaction.h
@@ -0,0 +1,28 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.h
+ * undo action prototypes
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_H
+#define UNDOACTION_H
+
+#include "postgres.h"
+
+#include "access/undolog.h"
+#include "access/undorecord.h"
+
+/* undo record information */
+typedef struct UndoRecInfo
+{
+ UndoRecPtr urp; /* undo recptr (undo record location). */
+ UnpackedUndoRecord *uur; /* actual undo record. */
+} UndoRecInfo;
+
+#endif
diff --git a/src/include/access/undoaction_xlog.h b/src/include/access/undoaction_xlog.h
new file mode 100644
index 0000000..bfc6418
--- /dev/null
+++ b/src/include/access/undoaction_xlog.h
@@ -0,0 +1,74 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction_xlog.h
+ * undo action XLOG definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction_xlog.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_XLOG_H
+#define UNDOACTION_XLOG_H
+
+#include "access/undolog.h"
+#include "access/xlogreader.h"
+#include "lib/stringinfo.h"
+#include "storage/off.h"
+
+/*
+ * WAL record definitions for undoactions.c's WAL operations
+ */
+#define XLOG_UNDO_PAGE 0x00
+#define XLOG_UNDO_RESET_SLOT 0x10
+#define XLOG_UNDO_APPLY_PROGRESS 0x20
+
+/*
+ * xl_undoaction_page flag values, 8 bits are available.
+ */
+#define XLU_PAGE_CONTAINS_TPD_SLOT (1<<0)
+#define XLU_PAGE_CLEAR_VISIBILITY_MAP (1<<1)
+#define XLU_CONTAINS_TPD_OFFSET_MAP (1<<2)
+#define XLU_INIT_PAGE (1<<3)
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_page
+{
+ UndoRecPtr urec_ptr;
+ TransactionId xid;
+ int trans_slot_id; /* transaction slot id */
+} xl_undoaction_page;
+
+#define SizeOfUndoActionPage (offsetof(xl_undoaction_page, trans_slot_id) + sizeof(int))
+
+/* This is what we need to know about undo apply progress */
+typedef struct xl_undoapply_progress
+{
+ UndoRecPtr urec_ptr;
+ uint32 progress;
+} xl_undoapply_progress;
+
+#define SizeOfUndoActionProgress (offsetof(xl_undoapply_progress, progress) + sizeof(uint32))
+
+/*
+ * xl_undoaction_reset_slot flag values, 8 bits are available.
+ */
+#define XLU_RESET_CONTAINS_TPD_SLOT (1<<0)
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_reset_slot
+{
+ UndoRecPtr urec_ptr;
+ int trans_slot_id; /* transaction slot id */
+ uint8 flags;
+} xl_undoaction_reset_slot;
+
+#define SizeOfUndoActionResetSlot (offsetof(xl_undoaction_reset_slot, flags) + sizeof(uint8))
+
+extern void undoaction_redo(XLogReaderState *record);
+extern void undoaction_desc(StringInfo buf, XLogReaderState *record);
+extern const char *undoaction_identify(uint8 info);
+
+#endif /* UNDOACTION_XLOG_H */
diff --git a/src/include/access/undodiscard.h b/src/include/access/undodiscard.h
new file mode 100644
index 0000000..70d6408
--- /dev/null
+++ b/src/include/access/undodiscard.h
@@ -0,0 +1,31 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoinsert.h
+ * undo discard definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undodiscard.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDODISCARD_H
+#define UNDODISCARD_H
+
+#include "access/undolog.h"
+#include "access/xlogdefs.h"
+#include "catalog/pg_class.h"
+#include "storage/lwlock.h"
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+extern void UndoDiscard(TransactionId xmin, bool *hibernate);
+
+#endif /* UNDODISCARD_H */
diff --git a/src/include/access/undoinsert.h b/src/include/access/undoinsert.h
index 2b73f9b..fda3082 100644
--- a/src/include/access/undoinsert.h
+++ b/src/include/access/undoinsert.h
@@ -96,6 +96,8 @@ extern void UndoSetPrepareSize(int max_prepare, UnpackedUndoRecord *undorecords,
extern UndoRecPtr UndoGetPrevUndoRecptr(UndoRecPtr urp, uint16 prevlen);
extern void UndoRecordOnUndoLogChange(UndoPersistence persistence);
+extern void PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress);
+extern void UndoRecordUpdateTransInfo(void);
#endif /* UNDOINSERT_H */
diff --git a/src/include/access/undorecord.h b/src/include/access/undorecord.h
index 85642ad..13512c9 100644
--- a/src/include/access/undorecord.h
+++ b/src/include/access/undorecord.h
@@ -20,6 +20,17 @@
#include "storage/buf.h"
#include "storage/off.h"
+typedef enum undorectype
+{
+ UNDO_INSERT,
+ UNDO_MULTI_INSERT,
+ UNDO_DELETE,
+ UNDO_INPLACE_UPDATE,
+ UNDO_UPDATE,
+ UNDO_XID_LOCK_ONLY,
+ UNDO_XID_MULTI_LOCK_ONLY,
+ UNDO_ITEMID_UNUSED
+} undorectype;
/*
* Every undo record begins with an UndoRecordHeader structure, which is
@@ -30,6 +41,7 @@
*/
typedef struct UndoRecordHeader
{
+ RmgrId urec_rmid; /* RMGR [XXX:TODO: this creates an alignment hole?] */
uint8 urec_type; /* record type code */
uint8 urec_info; /* flag bits */
uint16 urec_prevlen; /* length of previous record in bytes */
@@ -148,6 +160,7 @@ typedef struct UndoRecordPayload
*/
typedef struct UnpackedUndoRecord
{
+ RmgrId uur_rmid; /* rmgr ID */
uint8 uur_type; /* record type code */
uint8 uur_info; /* flag bits */
uint16 uur_prevlen; /* length of previous record */
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 73394c5..ad8cabe 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -419,6 +419,7 @@ extern XLogRecPtr XactLogAbortRecord(TimestampTz abort_time,
int xactflags, TransactionId twophase_xid,
const char *twophase_gid);
extern void xact_redo(XLogReaderState *record);
+extern void XactPerformUndoActionsIfPending(void);
/* xactdesc.c */
extern void xact_desc(StringInfo buf, XLogReaderState *record);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 8cfcd44..89dc2b5 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -192,6 +192,9 @@ extern bool XLOG_DEBUG;
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+/* Generate a 64-bit xid by using epoch and 32-bit xid. */
+#define MakeEpochXid(epoch, xid) \
+ ((epoch << 32) | (xid))
/* Checkpoint statistics */
typedef struct CheckpointStatsData
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 30610b3..9226cff 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -21,8 +21,10 @@
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
+#include "access/undorecord.h"
#include "datatype/timestamp.h"
#include "lib/stringinfo.h"
+#include "nodes/pg_list.h"
#include "pgtime.h"
#include "storage/block.h"
#include "storage/relfilenode.h"
@@ -294,9 +296,14 @@ typedef struct RmgrData
void (*rm_startup) (void);
void (*rm_cleanup) (void);
void (*rm_mask) (char *pagedata, BlockNumber blkno);
+ bool (*rm_undo) (List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock,
+ int options);
+ void (*rm_undo_desc) (StringInfo buf, UnpackedUndoRecord *record);
} RmgrData;
-extern const RmgrData RmgrTable[];
+extern PGDLLIMPORT const RmgrData RmgrTable[];
/*
* Exported to support xlog switching from checkpointer
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 773d9e6..6918efc 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -61,6 +61,13 @@ typedef struct CheckPoint
* set to InvalidTransactionId.
*/
TransactionId oldestActiveXid;
+
+ /*
+ * Oldest transaction id with epoc which is having undo. Include this value
+ * in the checkpoint record so that whenever server starts we get proper
+ * value.
+ */
+ uint64 oldestXidWithEpochHavingUndo;
} CheckPoint;
/* XLOG info values for XLOG rmgr */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index d6b32c0..549bb38 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -245,6 +245,7 @@ extern PGDLLIMPORT bool allowSystemTableMods;
extern PGDLLIMPORT int work_mem;
extern PGDLLIMPORT int maintenance_work_mem;
extern PGDLLIMPORT int max_parallel_maintenance_workers;
+extern PGDLLIMPORT int rollback_overflow_size;
extern int VacuumCostPageHit;
extern int VacuumCostPageMiss;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 763379e..7c5f05e 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -749,6 +749,7 @@ typedef enum BackendState
#define PG_WAIT_TIMEOUT 0x09000000U
#define PG_WAIT_IO 0x0A000000U
#define PG_WAIT_PAGE_TRANS_SLOT 0x0B000000U
+#define PG_WAIT_ROLLBACK_HT 0x0C000000U
/* ----------
* Wait Events - Activity
@@ -774,6 +775,8 @@ typedef enum
WAIT_EVENT_WAL_RECEIVER_MAIN,
WAIT_EVENT_WAL_SENDER_MAIN,
WAIT_EVENT_WAL_WRITER_MAIN,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN
} WaitEventActivity;
/* ----------
diff --git a/src/include/postmaster/discardworker.h b/src/include/postmaster/discardworker.h
new file mode 100644
index 0000000..f00c6c4
--- /dev/null
+++ b/src/include/postmaster/discardworker.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.h
+ * Exports from postmaster/discardworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/discardworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _DISCARDWORKER_H
+#define _DISCARDWORKER_H
+
+/*
+ * This function will perform multiple actions based on need. (a) retrieve
+ * transactions which have become all-visible and truncate the associated undo
+ * logs or will increment the tail pointer. (b) drop the buffers corresponding
+ * to truncated pages.
+ */
+extern void DiscardWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void DiscardWorkerRegister(void);
+
+#endif /* _DISCARDWORKER_H */
diff --git a/src/include/postmaster/undoloop.h b/src/include/postmaster/undoloop.h
new file mode 100644
index 0000000..21d91c3
--- /dev/null
+++ b/src/include/postmaster/undoloop.h
@@ -0,0 +1,89 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoloop.h
+ * Exports from postmaster/undoloop.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ *
+ * src/include/postmaster/undoloop.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOLOOP_H
+#define _UNDOLOOP_H
+
+#include "access/undoinsert.h"
+#include "utils/hsearch.h"
+#include "utils/relcache.h"
+
+
+/* Various options while executing the undo actions for the page. */
+#define UNDO_ACTION_UPDATE_TPD 0x0001
+
+/* Remembers the last seen RecentGlobalXmin */
+TransactionId latestRecentGlobalXmin;
+
+/*
+ * This function will read the undo records starting from the undo
+ * from_urecptr till to_urecptr and if to_urecptr is invalid then till the
+ * first undo location of transaction. This also discards the buffers by
+ * calling DropUndoBuffers for which undo log is removed. This function
+ * can be used by RollbackToSavePoint, by Rollback, by undoworker to complete
+ * the work of errored out transactions or when there is an error in single
+ * user mode.
+ */
+extern void execute_undo_actions(UndoRecPtr from_urecptr,
+ UndoRecPtr to_urecptr, bool nopartial, bool rewind, bool rellock);
+extern void process_and_execute_undo_actions_page(UndoRecPtr from_urecptr,
+ Relation rel, Buffer buffer, uint32 epoch,
+ TransactionId xid, int slot_no);
+
+/*
+ * This function will be responsible to truncate the undo logs
+ * for transactions that become all-visible after RecentGlobalXmin has
+ * advanced (value is different than latestRecentGlobalXmin). The easiest
+ * way could be to traverse the undo log array that contains least transaction
+ * id for that undo log and see if it precedes RecentGlobalXmin, then start
+ * discarding the undo log for that transaction (moving the tail pointer of
+ * undo log) till it finds the transaction which is not all-visible. This also
+ * discards the buffers by calling ForgetBuffer for which undo log is
+ * removed. This function can be invoked by undoworker or after commit in
+ * single user mode.
+ */
+extern void recover_undo_pages();
+
+/*
+ * To increase the efficiency of the zheap system, we create a hash table for
+ * the rollbacks. All the rollback requests exceeding certain threshold, are
+ * pushed to this table. Undo worker starts reading the entries from this hash
+ * table one at a time, performs undo actions related to the respective xid and
+ * removes them from the hash table. This way backend is free from performing the
+ * undo actions in case of heavy rollbacks. The data structures and the routines
+ * required for this infrastructure are as follows.
+ */
+
+/* This is the data structure for each hash table entry for rollbacks. */
+typedef struct RollbackHashEntry
+{
+ UndoRecPtr start_urec_ptr;
+ UndoRecPtr end_urec_ptr;
+ Oid dbid;
+} RollbackHashEntry;
+
+extern bool RollbackHTIsFull(void);
+
+/* To push the rollback requests from backend to the respective hash table */
+extern bool PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr,
+ Oid dbid);
+
+/* To perform the undo actions reading from the hash table */
+extern void RollbackFromHT(Oid dbid);
+/* To calculate the size of the hash table size for rollabcks. */
+extern int RollbackHTSize(void);
+
+/* To initialize the hash table in shared memory for rollbacks. */
+extern void InitRollbackHashTable(void);
+extern List *RollbackHTGetDBList(void);
+extern bool ConditionTransactionUndoActionLock(TransactionId xid);
+extern void TransactionUndoActionLockRelease(TransactionId xid);
+#endif /* _UNDOLOOP_H */
diff --git a/src/include/postmaster/undoworker.h b/src/include/postmaster/undoworker.h
new file mode 100644
index 0000000..c277e01
--- /dev/null
+++ b/src/include/postmaster/undoworker.h
@@ -0,0 +1,39 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.h
+ * Exports from postmaster/undoworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/undoworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOWORKER_H
+#define _UNDOWORKER_H
+
+/* GUC options */
+/* undo worker sleep time between rounds */
+extern int UndoWorkerDelay;
+
+/*
+ * This function will perform multiple actions based on need. (a) retreive
+ * transaction and its corresponding undopoiter from shared memory queue and
+ * call undoloop to perform undo actions. After applying all the undo records
+ * for a particular transaction, it will increment the tail pointer in undo log.
+ * (b) it needs to retrieve transactions which have become all-visible and truncate
+ * the associated undo logs or will increment the tail pointer. (c) udjust the
+ * number of undo workers based on the work required to perform undo actions
+ * (it could be size of shared memory queue containing transactions that needs
+ * aborts). (d) drop the buffers corresponding to truncated pages (e) Sleep for
+ * UndoWorkerDelay, if there is no more work.
+ */
+extern void UndoWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void UndoLauncherRegister(void);
+extern void UndoLauncherShmemInit(void);
+extern Size UndoLauncherShmemSize(void);
+extern void UndoLauncherMain(Datum main_arg);
+extern void UndoWorkerMain(Datum main_arg);
+
+#endif /* _UNDOWORKER_H */
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index a37fda7..675761c 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -150,6 +150,8 @@ typedef enum LockTagType
LOCKTAG_VIRTUALTRANSACTION, /* virtual transaction (ditto) */
/* ID info for a virtual transaction is its VirtualTransactionId */
LOCKTAG_SPECULATIVE_TOKEN, /* speculative insertion Xid and token */
+ /* ID info for an transaction undoaction is transaction id */
+ LOCKTAG_TRANSACTION_UNDOACTION, /* transaction (waiting for undoaction) */
/* ID info for a transaction is its TransactionId */
LOCKTAG_OBJECT, /* non-relation database object */
/* ID info for an object is DB OID + CLASS OID + OBJECT OID + SUBID */
@@ -246,6 +248,14 @@ typedef struct LOCKTAG
(locktag).locktag_type = LOCKTAG_SPECULATIVE_TOKEN, \
(locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+#define SET_LOCKTAG_TRANSACTION_UNDOACTION(locktag,xid) \
+ ((locktag).locktag_field1 = (xid), \
+ (locktag).locktag_field2 = 0, \
+ (locktag).locktag_field3 = 0, \
+ (locktag).locktag_field4 = 0, \
+ (locktag).locktag_type = LOCKTAG_TRANSACTION_UNDOACTION, \
+ (locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+
#define SET_LOCKTAG_OBJECT(locktag,dboid,classoid,objoid,objsubid) \
((locktag).locktag_field1 = (dboid), \
(locktag).locktag_field2 = (classoid), \
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index cb613c8..f6b9d98 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -270,6 +270,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Oldest transaction id which is having undo. */
+ pg_atomic_uint64 oldestXidWithEpochHavingUndo;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
1.8.3.1
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: Undo worker and transaction rollback
2018-11-05 11:40 Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
@ 2018-11-05 12:32 ` Dilip Kumar <[email protected]>
2018-11-30 15:13 ` Re: Undo worker and transaction rollback Dmitry Dolgov <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Dilip Kumar @ 2018-11-05 12:32 UTC (permalink / raw)
To: pgsql-hackers
On Mon, Nov 5, 2018 at 5:10 PM Dilip Kumar <[email protected]> wrote:
>
> On Thu, Oct 11, 2018 at 11:30 AM Dilip Kumar <[email protected]> wrote:
> >
> > Hello, hackers,
> >
> > In previous threads[1], we proposed patches for generating and storing
> > undo records and now for undo-worker and the transaction rollback
> > stuff. The idea is that undo remains relevant as long as the
> > transaction is in progress and needs to be removed once it becomes
> > irrelevant. Specifically, for a committed transaction, it remains
> > relevant until the transaction becomes all-visible; zheap will use
> > this for MVCC purposes. However, for an aborted transaction, it
> > remains relevant until the “undo actions" described by the undo
> > records have been performed. This patch introduces code to discard
> > undo when it is no longer needed and reuse the associated storage.
> > Additionally, this patch adds code to execute undo actions. Let me
> > explain the finer details for each of the cases covered,
>
> Latest rebased patch for undo worker and transaction rollback. This
> patch includes some bug fixes from the main zheap branch and also
> include code for executing undo action using RMGR (RMGR related code
> is merged from Thomas' patch set for "POC: Cleaning up orphaned files
> using undo logs"[1]. This patch can be applied on top of undolog and
> undo-interface patches.
>
> [1] https://www.postgresql.org/message-id/flat/CAEepm=0ULqYgM2aFeOnrx6YrtBg3xUdxALoyCG+XpssKqmezug@mail....
Updated patch, include defect fix from Kuntal posted on [1].
[1] https://www.postgresql.org/message-id/CAGz5QCKpCG6egFAdazC%2BJgyk7YSE1OBN9h-QpwCkg-NnSWN5AQ%40mail.g...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] undoworker-transaction-rollback-v3.patch (134.3K, ../../CAFiTN-uYfVVpFfyoVXansHfXWs3zmxsp1jUXF_=gPD6Bp2RoEA@mail.gmail.com/2-undoworker-transaction-rollback-v3.patch)
download | inline diff:
From bb3a1a0a51f427cc408b7891730309b4e0ef8629 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <[email protected]>
Date: Mon, 5 Nov 2018 04:20:16 -0800
Subject: [PATCH] undoworker-transaction-rollback-v3
Dilip Kumar, with help from Rafia Sabih, Amit Kapila, Mithun.CY, Thomas Munro and Kuntal Ghosh
---
src/backend/access/rmgrdesc/Makefile | 3 +-
src/backend/access/rmgrdesc/undoactiondesc.c | 64 +++
src/backend/access/rmgrdesc/xlogdesc.c | 4 +-
src/backend/access/transam/rmgr.c | 5 +-
src/backend/access/transam/twophase.c | 45 +-
src/backend/access/transam/varsup.c | 12 +
src/backend/access/transam/xact.c | 441 ++++++++++++++++-
src/backend/access/transam/xlog.c | 20 +
src/backend/access/undo/Makefile | 2 +-
src/backend/access/undo/undoaction.c | 683 ++++++++++++++++++++++++++
src/backend/access/undo/undoactionxlog.c | 49 ++
src/backend/access/undo/undodiscard.c | 458 +++++++++++++++++
src/backend/access/undo/undoinsert.c | 51 +-
src/backend/access/undo/undorecord.c | 3 +
src/backend/commands/vacuum.c | 11 +
src/backend/postmaster/Makefile | 4 +-
src/backend/postmaster/bgworker.c | 11 +
src/backend/postmaster/discardworker.c | 170 +++++++
src/backend/postmaster/pgstat.c | 7 +-
src/backend/postmaster/postmaster.c | 7 +
src/backend/postmaster/undoworker.c | 664 +++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 4 +
src/backend/storage/ipc/ipci.c | 6 +
src/backend/storage/lmgr/lwlocknames.txt | 2 +
src/backend/storage/lmgr/proc.c | 2 +
src/backend/utils/adt/lockfuncs.c | 1 +
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc.c | 11 +
src/backend/utils/misc/pg_controldata.c | 9 +-
src/backend/utils/misc/postgresql.conf.sample | 7 +
src/bin/pg_controldata/pg_controldata.c | 2 +
src/bin/pg_resetwal/pg_resetwal.c | 7 +
src/bin/pg_rewind/parsexlog.c | 2 +-
src/bin/pg_waldump/rmgrdesc.c | 3 +-
src/include/access/rmgr.h | 2 +-
src/include/access/rmgrlist.h | 47 +-
src/include/access/transam.h | 4 +
src/include/access/twophase.h | 2 +-
src/include/access/undoaction.h | 28 ++
src/include/access/undoaction_xlog.h | 74 +++
src/include/access/undodiscard.h | 31 ++
src/include/access/undoinsert.h | 2 +
src/include/access/undorecord.h | 13 +
src/include/access/xact.h | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlog_internal.h | 9 +-
src/include/catalog/pg_control.h | 7 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +
src/include/postmaster/discardworker.h | 25 +
src/include/postmaster/undoloop.h | 89 ++++
src/include/postmaster/undoworker.h | 39 ++
src/include/storage/lock.h | 10 +
src/include/storage/proc.h | 2 +
54 files changed, 3118 insertions(+), 45 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/undoactiondesc.c
create mode 100644 src/backend/access/undo/undoaction.c
create mode 100644 src/backend/access/undo/undoactionxlog.c
create mode 100644 src/backend/access/undo/undodiscard.c
create mode 100644 src/backend/postmaster/discardworker.c
create mode 100644 src/backend/postmaster/undoworker.c
create mode 100644 src/include/access/undoaction.h
create mode 100644 src/include/access/undoaction_xlog.h
create mode 100644 src/include/access/undodiscard.h
create mode 100644 src/include/postmaster/discardworker.h
create mode 100644 src/include/postmaster/undoloop.h
create mode 100644 src/include/postmaster/undoworker.h
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index 91ad1ef..640d37f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = brindesc.o clogdesc.o committsdesc.o dbasedesc.o genericdesc.o \
gindesc.o gistdesc.o hashdesc.o heapdesc.o logicalmsgdesc.o \
mxactdesc.o nbtdesc.o relmapdesc.o replorigindesc.o seqdesc.o \
- smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undologdesc.o xactdesc.o xlogdesc.o
+ smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undoactiondesc.o \
+ undologdesc.o xactdesc.o xlogdesc.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/rmgrdesc/undoactiondesc.c b/src/backend/access/rmgrdesc/undoactiondesc.c
new file mode 100644
index 0000000..89343b8
--- /dev/null
+++ b/src/backend/access/rmgrdesc/undoactiondesc.c
@@ -0,0 +1,64 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactiondesc.c
+ * rmgr descriptor routines for access/undo/undoactionxlog.c
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/access/rmgrdesc/undoactiondesc.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+
+void
+undoaction_desc(StringInfo buf, XLogReaderState *record)
+{
+ char *rec = XLogRecGetData(record);
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ if (info == XLOG_UNDO_PAGE)
+ {
+ uint8 *flags = (uint8 *) rec;
+
+ appendStringInfo(buf, "page_contains_tpd_slot: %c ",
+ (*flags & XLU_PAGE_CONTAINS_TPD_SLOT) ? 'T' : 'F');
+ appendStringInfo(buf, "is_page_initialized: %c ",
+ (*flags & XLU_INIT_PAGE) ? 'T' : 'F');
+ if (*flags & XLU_PAGE_CONTAINS_TPD_SLOT)
+ {
+ xl_undoaction_page *xlrec =
+ (xl_undoaction_page *) ((char *) flags + sizeof(uint8));
+
+ appendStringInfo(buf, "urec_ptr %lu xid %u trans_slot_id %u",
+ xlrec->urec_ptr, xlrec->xid, xlrec->trans_slot_id);
+ }
+ }
+ else if (info == XLOG_UNDO_APPLY_PROGRESS)
+ {
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) rec;
+
+ appendStringInfo(buf, "urec_ptr %lu progress %u",
+ xlrec->urec_ptr, xlrec->progress);
+ }
+}
+
+const char *
+undoaction_identify(uint8 info)
+{
+ const char *id = NULL;
+
+ switch (info & ~XLR_INFO_MASK)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ id = "UNDO APPLY PROGRESS";
+ break;
+ }
+
+ return id;
+}
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 00741c7..987e39c 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -47,7 +47,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
"tli %u; prev tli %u; fpw %s; xid %u:%u; oid %u; multi %u; offset %u; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
- "oldest running xid %u; %s",
+ "oldest running xid %u; "
+ "oldest xid with epoch having undo " UINT64_FORMAT "; %s",
(uint32) (checkpoint->redo >> 32), (uint32) checkpoint->redo,
checkpoint->ThisTimeLineID,
checkpoint->PrevTimeLineID,
@@ -63,6 +64,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
checkpoint->oldestCommitTsXid,
checkpoint->newestCommitTsXid,
checkpoint->oldestActiveXid,
+ checkpoint->oldestXidWithEpochHavingUndo,
(info == XLOG_CHECKPOINT_SHUTDOWN) ? "shutdown" : "online");
}
else if (info == XLOG_NEXTOID)
diff --git a/src/backend/access/transam/rmgr.c b/src/backend/access/transam/rmgr.c
index 8b05374..6238240 100644
--- a/src/backend/access/transam/rmgr.c
+++ b/src/backend/access/transam/rmgr.c
@@ -18,6 +18,7 @@
#include "access/multixact.h"
#include "access/nbtxlog.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -31,8 +32,8 @@
#include "utils/relmapper.h"
/* must be kept in sync with RmgrData definition in xlog_internal.h */
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
- { name, redo, desc, identify, startup, cleanup, mask },
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
+ { name, redo, desc, identify, startup, cleanup, mask, undo, undo_desc },
const RmgrData RmgrTable[RM_MAX_ID + 1] = {
#include "access/rmgrlist.h"
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 3942734..83241bc 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -93,6 +93,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/origin.h"
#include "replication/syncrep.h"
#include "replication/walsender.h"
@@ -915,6 +916,12 @@ typedef struct TwoPhaseFileHeader
uint16 gidlen; /* length of the GID - GID follows the header */
XLogRecPtr origin_lsn; /* lsn of this record at origin node */
TimestampTz origin_timestamp; /* time of prepare at origin node */
+ /*
+ * We need the locations of start and end undo record pointers when rollbacks
+ * are to be performed for prepared transactions using zheap relations.
+ */
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
} TwoPhaseFileHeader;
/*
@@ -989,7 +996,8 @@ save_state_data(const void *data, uint32 len)
* Initializes data structure and inserts the 2PC file header record.
*/
void
-StartPrepare(GlobalTransaction gxact)
+StartPrepare(GlobalTransaction gxact, UndoRecPtr *start_urec_ptr,
+ UndoRecPtr *end_urec_ptr)
{
PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
PGXACT *pgxact = &ProcGlobal->allPgXact[gxact->pgprocno];
@@ -1027,6 +1035,10 @@ StartPrepare(GlobalTransaction gxact)
&hdr.initfileinval);
hdr.gidlen = strlen(gxact->gid) + 1; /* Include '\0' */
+ /* save the start and end undo record pointers */
+ memcpy(hdr.start_urec_ptr, start_urec_ptr, sizeof(hdr.start_urec_ptr));
+ memcpy(hdr.end_urec_ptr, end_urec_ptr, sizeof(hdr.end_urec_ptr));
+
save_state_data(&hdr, sizeof(TwoPhaseFileHeader));
save_state_data(gxact->gid, hdr.gidlen);
@@ -1452,6 +1464,9 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
RelFileNode *delrels;
int ndelrels;
SharedInvalidationMessage *invalmsgs;
+ int i;
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
/*
* Validate the GID, and lock the GXACT to ensure that two backends do not
@@ -1489,6 +1504,34 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
invalmsgs = (SharedInvalidationMessage *) bufptr;
bufptr += MAXALIGN(hdr->ninvalmsgs * sizeof(SharedInvalidationMessage));
+ /* save the start and end undo record pointers */
+ memcpy(start_urec_ptr, hdr->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(end_urec_ptr, hdr->end_urec_ptr, sizeof(end_urec_ptr));
+
+ /*
+ * Perform undo actions, if there are undologs for this transaction.
+ * We need to perform undo actions while we are still in transaction.
+ * Never push rollbacks of temp tables to undo worker.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (end_urec_ptr[i] != InvalidUndoRecPtr && !isCommit)
+ {
+ bool result = false;
+ uint64 rollback_size = 0;
+
+ if (i != UNDO_TEMP)
+ rollback_size = end_urec_ptr[i] - start_urec_ptr[i];
+
+ if (rollback_size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(end_urec_ptr[i], start_urec_ptr[i], InvalidOid);
+
+ if (!result)
+ execute_undo_actions(end_urec_ptr[i], start_urec_ptr[i], true,
+ true, false);
+ }
+ }
+
/* compute latestXid among all children */
latestXid = TransactionIdLatest(xid, hdr->nsubxacts, children);
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index cede579..2274ab2 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -292,10 +292,22 @@ SetTransactionIdLimit(TransactionId oldest_datfrozenxid, Oid oldest_datoid)
TransactionId xidStopLimit;
TransactionId xidWrapLimit;
TransactionId curXid;
+ TransactionId oldestXidHavingUndo;
Assert(TransactionIdIsNormal(oldest_datfrozenxid));
/*
+ * To determine the last safe xid that can be allocated, we need to
+ * consider oldestXidHavingUndo. The oldestXidHavingUndo will be only
+ * valid for zheap storage engine, so it won't impact any other storage
+ * engine.
+ */
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ oldest_datfrozenxid = Min(oldest_datfrozenxid, oldestXidHavingUndo);
+
+ /*
* The place where we actually get into deep trouble is halfway around
* from the oldest potentially-existing XID. (This calculation is
* probably off by one or two counts, because the special XIDs reduce the
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 6b7f7fa..80fb1fa 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -41,6 +41,7 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/origin.h"
@@ -278,6 +279,20 @@ typedef struct SubXactCallbackItem
static SubXactCallbackItem *SubXact_callbacks = NULL;
+/* Location in undo log from where to start applying the undo actions. */
+static UndoRecPtr UndoActionStartPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+/* Location in undo log up to which undo actions need to be applied. */
+static UndoRecPtr UndoActionEndPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+/* Do we need to perform any undo actions? */
+static bool PerformUndoActions = false;
/* local function prototypes */
static void AssignTransactionId(TransactionState s);
@@ -1824,6 +1839,7 @@ StartTransaction(void)
{
TransactionState s;
VirtualTransactionId vxid;
+ int i;
/*
* Let's just make sure the state stack is empty
@@ -1884,6 +1900,13 @@ StartTransaction(void)
nUnreportedXids = 0;
s->didLogXid = false;
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
/*
* must initialize resource-management stuff first
*/
@@ -2207,7 +2230,7 @@ CommitTransaction(void)
* NB: if you change this routine, better look at CommitTransaction too!
*/
static void
-PrepareTransaction(void)
+PrepareTransaction(UndoRecPtr *start_urec_ptr, UndoRecPtr *end_urec_ptr)
{
TransactionState s = CurrentTransactionState;
TransactionId xid = GetCurrentTransactionId();
@@ -2355,7 +2378,7 @@ PrepareTransaction(void)
* PREPARED; in particular, pay attention to whether things should happen
* before or after releasing the transaction's locks.
*/
- StartPrepare(gxact);
+ StartPrepare(gxact, start_urec_ptr, end_urec_ptr);
AtPrepare_Notify();
AtPrepare_Locks();
@@ -2478,6 +2501,65 @@ PrepareTransaction(void)
RESUME_INTERRUPTS();
}
+static void
+AtAbort_Rollback(void)
+{
+ TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i;
+
+ /* XXX: TODO: check this logic, which was moved out of UserAbortTransactionBlock */
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (CurrentTransactionState->state == TRANS_INPROGRESS)
+ {
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (latest_urec_ptr[i])
+ {
+ if (i == UNDO_TEMP)
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ false, true, true);
+ else
+ {
+ uint64 size = latest_urec_ptr[i] - s->start_urec_ptr[i];
+ bool result = false;
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions
+ * later.
+ */
+ if (size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ true, true, true);
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
+}
/*
* AbortTransaction
@@ -2579,6 +2661,7 @@ AbortTransaction(void)
*/
AfterTriggerEndXact(false); /* 'false' means it's abort */
AtAbort_Portals();
+ AtAbort_Rollback();
AtEOXact_LargeObject(false);
AtAbort_Notify();
AtEOXact_RelationMap(false, is_parallel_worker);
@@ -2787,6 +2870,12 @@ void
CommitTransactionCommand(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i;
+
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(end_urec_ptr, s->latest_urec_ptr, sizeof(end_urec_ptr));
switch (s->blockState)
{
@@ -2876,7 +2965,7 @@ CommitTransactionCommand(void)
* return to the idle state.
*/
case TBLOCK_PREPARE:
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, end_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
break;
@@ -2922,6 +3011,23 @@ CommitTransactionCommand(void)
{
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
+
+ /*
+ * Update the end undo record pointer if it's not valid with
+ * the currently popped transaction's end undo record pointer.
+ * This is particularly required when the first command of
+ * the transaction is of type which does not require an undo,
+ * e.g. savepoint x.
+ * Accordingly, update the start undo record pointer.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(end_urec_ptr[i]))
+ end_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(s->start_urec_ptr[i]))
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
} while (s->blockState == TBLOCK_SUBCOMMIT);
/* If we had a COMMIT command, finish off the main xact too */
if (s->blockState == TBLOCK_END)
@@ -2933,7 +3039,7 @@ CommitTransactionCommand(void)
else if (s->blockState == TBLOCK_PREPARE)
{
Assert(s->parent == NULL);
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, end_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
}
else
@@ -3027,6 +3133,18 @@ void
AbortCurrentTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
+
+ /*
+ * The undo actions are allowed to be executed at the end of statement
+ * execution when we are not in transaction block, otherwise they are
+ * executed when user explicitly ends the transaction.
+ *
+ * So if we are in a transaction block don't set the PerformUndoActions
+ * because this flag will be set when user explicitly issue rollback or
+ * rollback to savepoint.
+ */
+ PerformUndoActions = false;
switch (s->blockState)
{
@@ -3061,6 +3179,16 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * We are outside the transaction block so remember the required
+ * information to perform undo actions and also set the
+ * PerformUndoActions so that we execute it before completing this
+ * command.
+ */
+ PerformUndoActions = true;
+ memcpy (UndoActionStartPtr, s->latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
break;
/*
@@ -3097,6 +3225,9 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3116,6 +3247,9 @@ AbortCurrentTransaction(void)
case TBLOCK_ABORT_END:
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3126,6 +3260,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Failed while executing the rollback command, need perform any
+ * pending undo actions.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3137,6 +3277,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Perform any pending actions if failed while preparing the
+ * transaction.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3159,6 +3305,17 @@ AbortCurrentTransaction(void)
case TBLOCK_SUBCOMMIT:
case TBLOCK_SUBABORT_PENDING:
case TBLOCK_SUBRESTART:
+ /*
+ * If we are here and still UndoActionStartPtr is valid that means
+ * the subtransaction failed while executing the undo action, so
+ * store its undo action start point in parent so that parent can
+ * start its undo action from this point.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ s->parent->latest_urec_ptr[i] = UndoActionStartPtr[i];
+ }
AbortSubTransaction();
CleanupSubTransaction();
AbortCurrentTransaction();
@@ -3176,6 +3333,109 @@ AbortCurrentTransaction(void)
}
/*
+ * XactPerformUndoActionsIfPending - Execute pending undo actions.
+ *
+ * If the parent transaction state is valid (when there is an error in the
+ * subtransaction and rollback to savepoint is executed), then allow to
+ * perform undo actions in it, otherwise perform them in a new transaction.
+ */
+void
+XactPerformUndoActionsIfPending()
+{
+ TransactionState s = CurrentTransactionState;
+ uint64 rollback_size = 0;
+ bool new_xact = true, result = false, no_pending_action = true;
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ if (!PerformUndoActions)
+ return;
+
+ /* If there is no undo log for any persistence level, then return. */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ {
+ no_pending_action = false;
+ break;
+ }
+ }
+
+ if (no_pending_action)
+ {
+ PerformUndoActions = false;
+ return;
+ }
+
+ /*
+ * Execute undo actions under parent transaction, if any. Otherwise start
+ * a new transaction.
+ */
+ if (GetTopTransactionIdIfAny() != InvalidTransactionId)
+ {
+ memcpy(parent_latest_urec_ptr, s->latest_urec_ptr,
+ sizeof (parent_latest_urec_ptr));
+ new_xact = false;
+ }
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions later.
+ * Never push the rollbacks for temp tables.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ continue;
+
+ if (i == UNDO_TEMP)
+ goto perform_rollback;
+ else
+ rollback_size = UndoActionStartPtr[i] - UndoActionEndPtr[i];
+
+ if (new_xact && rollback_size > rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+perform_rollback:
+ if (new_xact)
+ {
+ TransactionState xact;
+
+ /* Start a new transaction for performing the rollback */
+ StartTransactionCommand();
+ xact = CurrentTransactionState;
+
+ /*
+ * Store the previous transactions start and end undo record
+ * pointers into this transaction's state so that if there is
+ * some error while performing undo actions we can restart
+ * from begining.
+ */
+ memcpy(xact->start_urec_ptr, UndoActionEndPtr,
+ sizeof(UndoActionEndPtr));
+ memcpy(xact->latest_urec_ptr, UndoActionStartPtr,
+ sizeof(UndoActionStartPtr));
+ }
+
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ new_xact, true, true);
+
+ if (new_xact)
+ CommitTransactionCommand();
+ else
+ {
+ /* Restore parent's state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+ }
+
+ PerformUndoActions = false;
+}
+
+/*
* PreventInTransactionBlock
*
* This routine is to be called by statements that must not run inside
@@ -3576,6 +3836,10 @@ EndTransactionBlock(void)
{
TransactionState s = CurrentTransactionState;
bool result = false;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i ;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
switch (s->blockState)
{
@@ -3621,6 +3885,16 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+
+ /*
+ * We are calculating latest_urec_ptr, even though its a commit
+ * case. This is to handle any error during the commit path.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_END;
@@ -3646,6 +3920,11 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if(!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_ABORT_PENDING;
@@ -3698,6 +3977,18 @@ EndTransactionBlock(void)
break;
}
+ /*
+ * We need to perform undo actions if the transaction is failed. Remember
+ * the required information to perform undo actions at the end of
+ * statement execution.
+ */
+ if (!result)
+ PerformUndoActions = true;
+
+ memcpy(UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy(UndoActionEndPtr, TopTransactionStateData.start_urec_ptr,
+ sizeof(UndoActionEndPtr));
+
return result;
}
@@ -3955,6 +4246,12 @@ ReleaseSavepoint(const char *name)
TransactionState s = CurrentTransactionState;
TransactionState target,
xact;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4048,8 +4345,32 @@ ReleaseSavepoint(const char *name)
if (xact == target)
break;
xact = xact->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = xact->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(xact->start_urec_ptr[i]))
+ start_urec_ptr[i] = xact->start_urec_ptr[i];
+ }
+
Assert(PointerIsValid(xact));
}
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction. Also set the start_urec_ptr if
+ * parent start_urec_ptr is not valid.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ xact->parent->latest_urec_ptr[i] = latest_urec_ptr[i];
+ if (!UndoRecPtrIsValid(xact->parent->start_urec_ptr[i]))
+ xact->parent->start_urec_ptr[i] = start_urec_ptr[i];
+ }
}
/*
@@ -4260,6 +4581,7 @@ void
ReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4278,6 +4600,22 @@ ReleaseCurrentSubTransaction(void)
BlockStateAsString(s->blockState));
Assert(s->state == TRANS_INPROGRESS);
MemoryContextSwitchTo(CurTransactionContext);
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ s->parent->latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (!UndoRecPtrIsValid(s->parent->start_urec_ptr[i]))
+ s->parent->start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
Assert(s->state == TRANS_INPROGRESS);
@@ -4294,6 +4632,14 @@ void
RollbackAndReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i;
/*
* Unlike ReleaseCurrentSubTransaction(), this is nominally permitted
@@ -4340,6 +4686,19 @@ RollbackAndReleaseCurrentSubTransaction(void)
if (s->blockState == TBLOCK_SUBINPROGRESS)
AbortSubTransaction();
+ /*
+ * Remember the required information to perform undo actions before
+ * cleaning up the subtransaction state.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ {
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+ }
+
/* And clean it up, too */
CleanupSubTransaction();
@@ -4348,6 +4707,30 @@ RollbackAndReleaseCurrentSubTransaction(void)
s->blockState == TBLOCK_INPROGRESS ||
s->blockState == TBLOCK_IMPLICIT_INPROGRESS ||
s->blockState == TBLOCK_STARTED);
+
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ parent_latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ /*
+ * Store the undo action start point in the parent state so that
+ * we can apply undo actions these undos also during rollback of
+ * parent transaction in case of error while applying the undo
+ * actions.
+ */
+ s->latest_urec_ptr[i] = latest_urec_ptr[i];
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false,
+ true, true);
+
+ /* Restore parent state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+
+ /* Successfully performed undo actions so reset the flag. */
+ PerformUndoActions = false;
}
/*
@@ -4561,6 +4944,7 @@ static void
StartSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
if (s->state != TRANS_DEFAULT)
elog(WARNING, "StartSubTransaction while in %s state",
@@ -4578,6 +4962,13 @@ StartSubTransaction(void)
AtSubStart_Notify();
AfterTriggerBeginSubXact();
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
s->state = TRANS_INPROGRESS;
/*
@@ -4701,6 +5092,47 @@ CommitSubTransaction(void)
PopTransaction();
}
+static void
+AtSubAbort_Rollback(TransactionState s)
+{
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ /* XXX: TODO: Check this logic, which was moved out of RollbackToSavepoint() */
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (s->state == TRANS_INPROGRESS)
+ {
+ for ( i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false, true, false);
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
+}
+
/*
* AbortSubTransaction
*/
@@ -4795,6 +5227,7 @@ AbortSubTransaction(void)
s->parent->subTransactionId,
s->curTransactionOwner,
s->parent->curTransactionOwner);
+ AtSubAbort_Rollback(s);
AtEOSubXact_LargeObject(false, s->subTransactionId,
s->parent->subTransactionId);
AtSubAbort_Notify();
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 23f23e7..cdc085f 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -5172,6 +5172,7 @@ BootStrapXLOG(void)
checkPoint.newestCommitTsXid = InvalidTransactionId;
checkPoint.time = (pg_time_t) time(NULL);
checkPoint.oldestActiveXid = InvalidTransactionId;
+ checkPoint.oldestXidWithEpochHavingUndo = InvalidTransactionId;
ShmemVariableCache->nextXid = checkPoint.nextXid;
ShmemVariableCache->nextOid = checkPoint.nextOid;
@@ -6792,6 +6793,10 @@ StartupXLOG(void)
(errmsg_internal("commit timestamp Xid oldest/newest: %u/%u",
checkPoint.oldestCommitTsXid,
checkPoint.newestCommitTsXid)));
+ ereport(DEBUG1,
+ (errmsg_internal("oldest xid with epoch having undo: " UINT64_FORMAT,
+ checkPoint.oldestXidWithEpochHavingUndo)));
+
if (!TransactionIdIsNormal(checkPoint.nextXid))
ereport(PANIC,
(errmsg("invalid next transaction ID")));
@@ -6809,6 +6814,10 @@ StartupXLOG(void)
XLogCtl->ckptXidEpoch = checkPoint.nextXidEpoch;
XLogCtl->ckptXid = checkPoint.nextXid;
+ /* Read oldest xid having undo from checkpoint and set in proc global. */
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* Initialize replication slots, before there's a chance to remove
* required resources.
@@ -8979,6 +8988,9 @@ CreateCheckPoint(int flags)
checkPoint.nextOid += ShmemVariableCache->oidCount;
LWLockRelease(OidGenLock);
+ checkPoint.oldestXidWithEpochHavingUndo =
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo);
+
MultiXactGetCheckptMulti(shutdown,
&checkPoint.nextMulti,
&checkPoint.nextMultiOffset,
@@ -9889,6 +9901,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceOldest(checkPoint.oldestMulti,
checkPoint.oldestMultiDB);
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* No need to set oldestClogXid here as well; it'll be set when we
* redo an xl_clog_truncate if it changed since initialization.
@@ -9947,6 +9962,8 @@ xlog_redo(XLogReaderState *record)
/* ControlFile->checkPointCopy always tracks the latest ckpt XID */
ControlFile->checkPointCopy.nextXidEpoch = checkPoint.nextXidEpoch;
ControlFile->checkPointCopy.nextXid = checkPoint.nextXid;
+ ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo =
+ checkPoint.oldestXidWithEpochHavingUndo;
/* Update shared-memory copy of checkpoint XID/epoch */
SpinLockAcquire(&XLogCtl->info_lck);
@@ -9996,6 +10013,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceNextMXact(checkPoint.nextMulti,
checkPoint.nextMultiOffset);
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* NB: This may perform multixact truncation when replaying WAL
* generated by an older primary.
diff --git a/src/backend/access/undo/Makefile b/src/backend/access/undo/Makefile
index f41e8f7..fdf7f7d 100644
--- a/src/backend/access/undo/Makefile
+++ b/src/backend/access/undo/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/access/undo
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = undoinsert.o undolog.o undorecord.o
+OBJS = undoaction.o undoactionxlog.o undodiscard.o undoinsert.o undolog.o undorecord.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/undo/undoaction.c b/src/backend/access/undo/undoaction.c
new file mode 100644
index 0000000..a7bf506
--- /dev/null
+++ b/src/backend/access/undo/undoaction.c
@@ -0,0 +1,683 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.c
+ * execute undo actions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undoaction.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/undoaction.h"
+#include "access/undoaction_xlog.h"
+#include "access/undolog.h"
+#include "access/undorecord.h"
+#include "access/visibilitymap.h"
+#include "access/xact.h"
+#include "access/xloginsert.h"
+#include "access/xlog_internal.h"
+#include "nodes/pg_list.h"
+#include "pgstat.h"
+#include "postmaster/undoloop.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "utils/rel.h"
+#include "utils/relfilenodemap.h"
+#include "miscadmin.h"
+#include "storage/shmem.h"
+#include "access/undodiscard.h"
+
+#define ROLLBACK_HT_SIZE 1024
+
+static bool execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr,
+ Oid reloid, TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool norellock, int options);
+static void RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr);
+
+/* This is the hash table to store all the rollabck requests. */
+static HTAB *RollbackHT;
+
+/*
+ * execute_undo_actions - Execute the undo actions
+ *
+ * from_urecptr - undo record pointer from where to start applying undo action.
+ * to_urecptr - undo record pointer upto which point apply undo action.
+ * nopartial - true if rollback is for complete transaction.
+ * rewind - whether to rewind the insert location of the undo log or not.
+ * Only the backend executed the transaction can rewind, but
+ * any other process e.g. undo worker should not rewind it.
+ * Because, if the backend have already inserted new undo records
+ * for the next transaction and if we rewind then we will loose
+ * the undo record inserted for the new transaction.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ */
+void
+execute_undo_actions(UndoRecPtr from_urecptr, UndoRecPtr to_urecptr,
+ bool nopartial, bool rewind, bool rellock)
+{
+ UnpackedUndoRecord *uur = NULL;
+ UndoRecPtr urec_ptr, prev_urec_ptr;
+ UndoRecPtr save_urec_ptr;
+ Oid prev_reloid = InvalidOid;
+ ForkNumber prev_fork = InvalidForkNumber;
+ BlockNumber prev_block = InvalidBlockNumber;
+ List *luinfo = NIL;
+ bool more_undo;
+ int options = 0;
+ TransactionId xid = InvalidTransactionId;
+ UndoRecInfo *urec_info;
+
+ Assert(from_urecptr != InvalidUndoRecPtr);
+ Assert(UndoRecPtrGetLogNo(from_urecptr) != UndoRecPtrGetLogNo(to_urecptr) ||
+ from_urecptr >= to_urecptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ * FIXME: this won't work if undolog crossed the limit of 1TB, because
+ * then from_urecptr and to_urecptr will be from different lognos.
+ */
+ if (to_urecptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(from_urecptr);
+ to_urecptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ save_urec_ptr = urec_ptr = from_urecptr;
+
+ if (nopartial)
+ {
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber, InvalidOffsetNumber,
+ InvalidTransactionId, NULL, NULL);
+ if (uur == NULL)
+ return;
+
+ xid = uur->uur_xid;
+ UndoRecordRelease(uur);
+ uur = NULL;
+
+ /*
+ * Grab the undo action apply lock before start applying the undo action
+ * this will prevent applying undo actions concurrently. If we do not
+ * get the lock that mean its already being applied concurrently or the
+ * discard worker might be pushing its request to the rollback hash
+ * table
+ */
+ if (!ConditionTransactionUndoActionLock(xid))
+ return;
+ }
+
+ prev_urec_ptr = InvalidUndoRecPtr;
+ while (prev_urec_ptr != to_urecptr)
+ {
+ Oid reloid = InvalidOid;
+ uint16 urec_prevlen;
+ bool non_page;
+
+ more_undo = true;
+
+ prev_urec_ptr = urec_ptr;
+
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId, NULL, NULL);
+
+ /* If there is no info block, this is not a page-based undo record. */
+ non_page = uur && !(uur->uur_info & UREC_INFO_BLOCK);
+
+ if (uur != NULL && !non_page)
+ reloid = RelidByRelfilenode(uur->uur_tsid, uur->uur_relfilenode);
+
+ xid = uur->uur_xid;
+
+ if (non_page)
+ {
+ prev_reloid = InvalidOid;
+ urec_prevlen = uur->uur_prevlen;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /*
+ * Execute individual undo actions not associated with a page
+ * immediately.
+ */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->uur = uur;
+ urec_info->urp = urec_ptr;
+ luinfo = lappend(luinfo, urec_info);
+ execute_undo_actions_page(luinfo, urec_ptr, reloid, xid,
+ InvalidBlockNumber, false, rellock,
+ 0);
+ pfree(urec_info);
+ urec_info = NULL;
+ list_free(luinfo);
+ luinfo = NIL;
+ UndoRecordRelease(uur);
+
+ /* Follow undo chain until to_urecptr. */
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ /*
+ * If the record is already discarded by undo worker or if the relation
+ * is dropped or truncated, then we cannot fetch record successfully.
+ * Hence, skip quietly.
+ *
+ * Note: reloid remains InvalidOid for a discarded record.
+ */
+ else if (!OidIsValid(reloid))
+ {
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ urec_prevlen = uur->uur_prevlen;
+
+ /* Release the just-fetched record */
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else if (!OidIsValid(prev_reloid) ||
+ (prev_reloid == reloid &&
+ prev_fork == uur->uur_fork &&
+ prev_block == uur->uur_block))
+ {
+ /* Collect the undo records that belong to the same page. */
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+
+ luinfo = lappend(luinfo, urec_info);
+ urec_prevlen = uur->uur_prevlen;
+ save_urec_ptr = uur->uur_blkprev;
+
+ if (uur->uur_info & UREC_INFO_PAYLOAD_CONTAINS_SLOT)
+ options |= UNDO_ACTION_UPDATE_TPD;
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else
+ {
+ more_undo = true;
+ }
+
+ /*
+ * If no more undo is left to be processed and we are rolling back the
+ * complete transaction, then we can consider that the undo chain for a
+ * block is complete.
+ * If the previous undo pointer in the page is invalid, then also the
+ * undo chain for the current block is completed.
+ */
+ if (luinfo &&
+ ((!more_undo && nopartial) || !UndoRecPtrIsValid(save_urec_ptr)))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, true, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+ else if (luinfo)
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, false, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ /*
+ * There are still more records to process, so keep moving backwards
+ * in the chain.
+ */
+ if (more_undo)
+ {
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+ luinfo = lappend(luinfo, urec_info);
+
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+ save_urec_ptr = uur->uur_blkprev;
+
+ if (uur->uur_info & UREC_INFO_PAYLOAD_CONTAINS_SLOT)
+ options |= UNDO_ACTION_UPDATE_TPD;
+
+ /*
+ * Continue to process the records if this is not the last undo
+ * record in chain.
+ */
+ urec_prevlen = uur->uur_prevlen;
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ else
+ break;
+ }
+ else
+ break;
+ }
+
+ /* Apply the undo actions for the remaining records. */
+ if (list_length(luinfo))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, nopartial ? true : false,
+ rellock, options);
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+ }
+
+ if (rewind)
+ {
+ /* Read the prevlen from the first record of this transaction. */
+ uur = UndoFetchRecord(to_urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If undo is already discarded before we rewind, then do nothing.
+ */
+ if (uur == NULL)
+ return;
+
+
+ /*
+ * Rewind the insert location to start of this transaction. This is
+ * to avoid reapplying some intermediate undo. We do not need to wal
+ * log this information here, because if the system crash before we
+ * rewind the insert pointer then after recovery we can identify
+ * whether the undo is already applied or not from the slot undo record
+ * pointer. Also set the correct prevlen value (what we have fetched
+ * from the undo).
+ */
+ UndoLogRewind(to_urecptr, uur->uur_prevlen);
+
+ UndoRecordRelease(uur);
+ }
+
+ if (nopartial)
+ {
+ /*
+ * Set undo action apply completed in the transaction header if this is
+ * a main transaction and we have not rewound its undo.
+ */
+ if (!rewind)
+ {
+ /*
+ * Undo action is applied so delete the hash table entry and release
+ * the undo action lock.
+ */
+ RollbackHTRemoveEntry(from_urecptr);
+
+ /*
+ * Prepare and update the progress of the undo action apply in the
+ * transaction header.
+ */
+ PrepareUpdateUndoActionProgress(to_urecptr, 1);
+
+ START_CRIT_SECTION();
+
+ /* Update the progress in the transaction header. */
+ UndoRecordUpdateTransInfo();
+
+ /* WAL log the undo apply progress. */
+ {
+ xl_undoapply_progress xlrec;
+
+ xlrec.urec_ptr = to_urecptr;
+ xlrec.progress = 1;
+
+ /*
+ * FIXME : We need to register undo buffers and set LSN for them
+ * that will be required for FPW of the undo buffers.
+ */
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, sizeof(xlrec));
+
+ (void) XLogInsert(RM_UNDOACTION_ID, XLOG_UNDO_APPLY_PROGRESS);
+ }
+
+ END_CRIT_SECTION();
+ UnlockReleaseUndoBuffers();
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+}
+
+/*
+ * execute_undo_actions_page - Execute the undo actions for a page
+ *
+ * After applying all the undo actions for a page, we clear the transaction
+ * slot on a page if the undo chain for block is complete, otherwise just
+ * rewind the undo pointer to the last record for that block that precedes
+ * the last undo record for which action is replayed.
+ *
+ * luinfo - list of undo records (along with their location) for which undo
+ * action needs to be replayed.
+ * urec_ptr - undo record pointer to which we need to rewind.
+ * reloid - OID of relation on which undo actions needs to be applied.
+ * blkno - block number on which undo actions needs to be applied.
+ * blk_chain_complete - indicates whether the undo chain for block is
+ * complete.
+ * nopartial - true if rollback is for complete transaction. If we are not
+ * rolling back the complete transaction then we need to apply the
+ * undo action for UNDO_INVALID_XACT_SLOT also because in such
+ * case we will rewind the insert undo location.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ * options - options for executing undo actions.
+ *
+ * returns true, if successfully applied the undo actions, otherwise, false.
+ */
+static bool
+execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock, int options)
+{
+ UndoRecInfo *first;
+
+ /*
+ * All records passed to us are for the same RMGR, so we just use the
+ * first record to dispatch.
+ */
+ Assert(luinfo != NIL);
+ first = (UndoRecInfo *) linitial(luinfo);
+
+ return RmgrTable[first->uur->uur_rmid].rm_undo(luinfo, urec_ptr, reloid,
+ xid, blkno,
+ blk_chain_complete, rellock,
+ options);
+}
+
+/*
+ * To return the size of the hash-table for rollbacks.
+ */
+int
+RollbackHTSize(void)
+{
+ return hash_estimate_size(ROLLBACK_HT_SIZE, sizeof(RollbackHashEntry));
+}
+
+/*
+ * To initialize the hash-table for rollbacks in shared memory
+ * for the given size.
+ */
+void
+InitRollbackHashTable(void)
+{
+ HASHCTL info;
+ MemSet(&info, 0, sizeof(info));
+
+ info.keysize = sizeof(UndoRecPtr);
+ info.entrysize = sizeof(RollbackHashEntry);
+ info.hash = tag_hash;
+
+ RollbackHT = ShmemInitHash("Undo actions Lookup Table",
+ ROLLBACK_HT_SIZE, ROLLBACK_HT_SIZE, &info,
+ HASH_ELEM | HASH_FUNCTION | HASH_FIXED_SIZE);
+}
+
+/*
+ * To push the rollback requests from backend to the hash-table.
+ * Return true if the request is successfully added, else false
+ * and the caller may execute undo actions itself.
+ */
+bool
+PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr, Oid dbid)
+{
+ bool found = false;
+ RollbackHashEntry *rh;
+
+ Assert(UndoRecPtrGetLogNo(start_urec_ptr) != UndoRecPtrGetLogNo(end_urec_ptr) ||
+ start_urec_ptr >= end_urec_ptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ */
+ if (start_urec_ptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(end_urec_ptr);
+ start_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ Assert(UndoRecPtrIsValid(start_urec_ptr));
+
+ /* If there is no space to accomodate new request, then we can't proceed. */
+ if (RollbackHTIsFull())
+ return false;
+
+ if(!UndoRecPtrIsValid(end_urec_ptr))
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(start_urec_ptr);
+ end_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ rh = (RollbackHashEntry *) hash_search(RollbackHT, &start_urec_ptr,
+ HASH_ENTER_NULL, &found);
+ if (!rh)
+ {
+ LWLockRelease(RollbackHTLock);
+ return false;
+ }
+ /* We shouldn't try to push the same rollback request again. */
+ if (!found)
+ {
+ rh->start_urec_ptr = start_urec_ptr;
+ rh->end_urec_ptr = end_urec_ptr;
+ rh->dbid = (dbid == InvalidOid) ? MyDatabaseId : dbid;
+ }
+ LWLockRelease(RollbackHTLock);
+
+ return true;
+}
+
+/*
+ * To perform the undo actions for the transactions whose rollback
+ * requests are in hash table. Sequentially, scan the hash-table
+ * and perform the undo-actions for the respective transactions.
+ * Once, the undo-actions are applied, remove the entry from the
+ * hash table.
+ */
+void
+RollbackFromHT(Oid dbid)
+{
+ UndoRecPtr start[ROLLBACK_HT_SIZE];
+ UndoRecPtr end[ROLLBACK_HT_SIZE];
+ RollbackHashEntry *rh;
+ HASH_SEQ_STATUS status;
+ int i = 0;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ Assert(hash_get_num_entries(RollbackHT) <= ROLLBACK_HT_SIZE);
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ {
+ if (rh->dbid == dbid)
+ {
+ start[i] = rh->start_urec_ptr;
+ end[i] = rh->end_urec_ptr;
+ i++;
+ }
+ }
+
+ LWLockRelease(RollbackHTLock);
+
+ /* Execute the rollback requests */
+ while(--i >= 0)
+ {
+ Assert(UndoRecPtrIsValid(start[i]));
+ Assert(UndoRecPtrIsValid(end[i]));
+
+ StartTransactionCommand();
+ execute_undo_actions(start[i], end[i], true, false, true);
+ CommitTransactionCommand();
+ }
+}
+
+/*
+ * Remove the rollback request entry from the rollback hash table.
+ */
+static void
+RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr)
+{
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ hash_search(RollbackHT, &start_urec_ptr, HASH_REMOVE, NULL);
+
+ LWLockRelease(RollbackHTLock);
+}
+
+/*
+ * To check if the rollback requests in the hash table are all
+ * completed or not. This is required because we don't not want to
+ * expose RollbackHT in xact.c, where it is required to ensure
+ * that we push the resuests only when there is some space in
+ * the hash-table.
+ */
+bool
+RollbackHTIsFull(void)
+{
+ bool result = false;
+
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ if (hash_get_num_entries(RollbackHT) >= ROLLBACK_HT_SIZE)
+ result = true;
+
+ LWLockRelease(RollbackHTLock);
+
+ return result;
+}
+
+/*
+ * Get database list from the rollback hash table.
+ */
+List *
+RollbackHTGetDBList()
+{
+ HASH_SEQ_STATUS status;
+ RollbackHashEntry *rh;
+ List *dblist = NIL;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ dblist = list_append_unique_oid(dblist, rh->dbid);
+
+ LWLockRelease(RollbackHTLock);
+
+ return dblist;
+}
+
+/*
+ * ConditionTransactionUndoActionLock
+ *
+ * Insert a lock showing that the undo action for given transaction is in
+ * progress. This is only done for the main transaction not for the
+ * sub-transaction.
+ */
+bool
+ConditionTransactionUndoActionLock(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ if (LOCKACQUIRE_NOT_AVAIL == LockAcquire(&tag, ExclusiveLock, false, true))
+ return false;
+ else
+ return true;
+}
+
+/*
+ * TransactionUndoActionLockRelease
+ *
+ * Delete the lock showing that the undo action given transaction ID is in
+ * progress.
+ */
+void
+TransactionUndoActionLockRelease(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ LockRelease(&tag, ExclusiveLock, false);
+}
diff --git a/src/backend/access/undo/undoactionxlog.c b/src/backend/access/undo/undoactionxlog.c
new file mode 100644
index 0000000..546f43d
--- /dev/null
+++ b/src/backend/access/undo/undoactionxlog.c
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactionxlog.c
+ * WAL replay logic for undo actions.
+ *
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/undo/undoactionxlog.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+#include "access/undoinsert.h"
+#include "access/visibilitymap.h"
+#include "access/xlog.h"
+#include "access/xlogutils.h"
+
+/*
+ * Replay of undo apply progress.
+ */
+static void
+undo_xlog_apply_progress(XLogReaderState *record)
+{
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) XLogRecGetData(record);
+
+ /* Update the progress in the transaction header. */
+ PrepareUpdateUndoActionProgress(xlrec->urec_ptr, xlrec->progress);
+ UndoRecordUpdateTransInfo();
+}
+
+void
+undoaction_redo(XLogReaderState *record)
+{
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ switch (info)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ undo_xlog_apply_progress(record);
+ break;
+ default:
+ elog(PANIC, "undoaction_redo: unknown op code %u", info);
+ }
+}
diff --git a/src/backend/access/undo/undodiscard.c b/src/backend/access/undo/undodiscard.c
new file mode 100644
index 0000000..9d55588
--- /dev/null
+++ b/src/backend/access/undo/undodiscard.c
@@ -0,0 +1,458 @@
+/*-------------------------------------------------------------------------
+ *
+ * undodiscard.c
+ * discard undo records
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undodiscard.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/xlog.h"
+#include "access/xact.h"
+#include "access/undolog.h"
+#include "access/undodiscard.h"
+#include "catalog/pg_tablespace.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "storage/shmem.h"
+#include "storage/proc.h"
+#include "utils/resowner.h"
+#include "postmaster/undoloop.h"
+
+static UndoRecPtr FetchLatestUndoPtrForXid(UndoRecPtr urecptr,
+ UnpackedUndoRecord *uur_start,
+ UndoLogControl *log);
+
+/*
+ * Discard the undo for the log
+ *
+ * Search the undo log, get the start record for each transaction until we get
+ * the transaction with xid >= xmin or an invalid xid. Then call undolog
+ * routine to discard upto that point and update the memory structure for the
+ * log slot. We set the hibernate flag if we do not have any undo logs, this
+ * flag is passed to the undo worker wherein it determines if system is idle
+ * and it should sleep for sometime.
+ *
+ * Return the oldest xid remaining in this undo log (which should be >= xmin,
+ * since we'll discard everything older). Return InvalidTransactionId if the
+ * undo log is empty.
+ */
+static TransactionId
+UndoDiscardOneLog(UndoLogControl *log, TransactionId xmin, bool *hibernate)
+{
+ UndoRecPtr undo_recptr, next_insert, from_urecptr;
+ UndoRecPtr next_urecptr = InvalidUndoRecPtr;
+ UnpackedUndoRecord *uur = NULL;
+ bool need_discard = false;
+ bool log_complete = false;
+ TransactionId undoxid = InvalidTransactionId;
+ TransactionId xid = log->oldest_xid;
+ TransactionId latest_discardxid = InvalidTransactionId;
+ uint32 epoch = 0;
+
+ undo_recptr = log->oldest_data;
+
+ /* There might not be any undo log and hibernation might be needed. */
+ *hibernate = true;
+
+ /* Loop until we run out of discardable transactions. */
+ do
+ {
+ bool pending_abort = false;
+
+ next_insert = UndoLogGetNextInsertPtr(log->logno, xid);
+
+ /*
+ * If the next insert location in the undo log is same as the oldest
+ * data for the log then there is nothing more to discard in this log
+ * so discard upto this point.
+ */
+ if (next_insert == undo_recptr)
+ {
+ /*
+ * If the discard location and the insert location is same then
+ * there is nothing to discard.
+ */
+ if (undo_recptr == log->oldest_data)
+ break;
+ else
+ log_complete = true;
+ }
+ else
+ {
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+
+ Assert(uur != NULL);
+
+ if (!TransactionIdDidCommit(uur->uur_xid) &&
+ TransactionIdPrecedes(uur->uur_xid, xmin) &&
+ uur->uur_progress == 0)
+ {
+ /*
+ * At the time of recovery, we might not have a valid next undo
+ * record pointer and in that case we'll calculate the location
+ * of from pointer using the last record of next insert
+ * location.
+ */
+ if (ConditionTransactionUndoActionLock(uur->uur_xid))
+ {
+ TransactionId xid = uur->uur_xid;
+ UndoLogControl *log = NULL;
+ UndoLogNumber logno;
+
+ logno = UndoRecPtrGetLogNo(undo_recptr);
+ log = UndoLogGet(logno, false);
+
+ /*
+ * If the corresponding log got rewinded to a location
+ * prior to undo_recptr, the undo actions are already
+ * applied.
+ */
+ if (log->meta.insert > undo_recptr)
+ {
+ UndoRecordRelease(uur);
+
+ /* Fetch the undo record under undo action lock. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the undo actions for the aborted transaction is
+ * already applied then continue discarding the undo log
+ * otherwise discard till current point and stop processing
+ * this undo log.
+ * Also, check this is indeed the transaction id we're
+ * looking for. It is possible that after rewinding
+ * some other transaction has inserted an undo record.
+ */
+ if (uur->uur_xid == xid && uur->uur_progress == 0)
+ {
+ from_urecptr = FetchLatestUndoPtrForXid(undo_recptr, uur, log);
+ (void)PushRollbackReq(from_urecptr, undo_recptr, uur->uur_dbid);
+ pending_abort = true;
+ }
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+ else
+ pending_abort = true;
+ }
+
+ next_urecptr = uur->uur_next;
+ undoxid = uur->uur_xid;
+ xid = undoxid;
+ epoch = uur->uur_xidepoch;
+ }
+
+ /* we can discard upto this point. */
+ if (TransactionIdFollowsOrEquals(undoxid, xmin) ||
+ next_urecptr == SpecialUndoRecPtr ||
+ UndoRecPtrGetLogNo(next_urecptr) != log->logno ||
+ log_complete || pending_abort)
+ {
+ /* Hey, I got some undo log to discard, can not hibernate now. */
+ *hibernate = false;
+
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /*
+ * If Transaction id is smaller than the xmin that means this must
+ * be the last transaction in this undo log, so we need to get the
+ * last insert point in this undo log and discard till that point.
+ * Also, if the transaction has pending abort, we stop discarding
+ * undo from the same location.
+ */
+ if (TransactionIdPrecedes(undoxid, xmin) && !pending_abort)
+ {
+ UndoRecPtr next_insert = InvalidUndoRecPtr;
+
+ /*
+ * Get the last insert location for this transaction Id, if it
+ * returns invalid pointer that means there is new transaction
+ * has started for this undolog. So we need to refetch the undo
+ * and continue the process.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, undoxid);
+ if (!UndoRecPtrIsValid(next_insert))
+ continue;
+
+ undo_recptr = next_insert;
+ need_discard = true;
+ epoch = 0;
+ latest_discardxid = undoxid;
+ undoxid = InvalidTransactionId;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_EXCLUSIVE);
+
+ /*
+ * If no more pending undo logs then set the oldest transaction to
+ * InvalidTransactionId.
+ */
+ if (log_complete)
+ {
+ log->oldest_xid = InvalidTransactionId;
+ log->oldest_xidepoch = 0;
+ }
+ else
+ {
+ log->oldest_xid = undoxid;
+ log->oldest_xidepoch = epoch;
+ }
+
+ log->oldest_data = undo_recptr;
+ LWLockRelease(&log->discard_lock);
+
+ if (need_discard)
+ UndoLogDiscard(undo_recptr, latest_discardxid);
+
+ break;
+ }
+
+ /*
+ * This transaction is smaller than the xmin so lets jump to the next
+ * transaction.
+ */
+ undo_recptr = next_urecptr;
+ latest_discardxid = undoxid;
+
+ if(uur != NULL)
+ {
+ UndoRecordRelease(uur);
+ uur = NULL;
+ }
+
+ need_discard = true;
+ } while (true);
+
+ return undoxid;
+}
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+void
+UndoDiscard(TransactionId oldestXmin, bool *hibernate)
+{
+ TransactionId oldestXidHavingUndo = oldestXmin;
+ uint64 epoch = GetEpochForXid(oldestXmin);
+ UndoLogControl *log = NULL;
+
+ /*
+ * TODO: Ideally we'd arrange undo logs so that we can efficiently find
+ * those with oldest_xid < oldestXmin, but for now we'll just scan all of
+ * them.
+ */
+ while ((log = UndoLogNext(log)))
+ {
+ TransactionId oldest_xid = InvalidTransactionId;
+
+ /* We can't process temporary undo logs. */
+ if (log->meta.persistence == UNDO_TEMP)
+ continue;
+
+ /*
+ * If the first xid of the undo log is smaller than the xmin the try
+ * to discard the undo log.
+ */
+ if (TransactionIdPrecedes(log->oldest_xid, oldestXmin))
+ {
+ /*
+ * If the XID in the discard entry is invalid then start scanning
+ * from the first valid undorecord in the log.
+ */
+ if (!TransactionIdIsValid(log->oldest_xid))
+ {
+ bool full = false;
+ UndoRecPtr urp = UndoLogGetFirstValidRecord(log, &full);
+
+ if (!UndoRecPtrIsValid(urp))
+ {
+ /*
+ * There is nothing to be discarded. If there is also no
+ * more free space, then a call to UndoLogDiscard() will
+ * discard it the undo log completely and free up the
+ * UndoLogControl slot.
+ */
+ if (full)
+ UndoLogDiscard(MakeUndoRecPtr(log->meta.logno,
+ log->meta.discard),
+ InvalidTransactionId);
+ continue;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_SHARED);
+ log->oldest_data = urp;
+ LWLockRelease(&log->discard_lock);
+ }
+
+ /* Process the undo log. */
+ oldest_xid = UndoDiscardOneLog(log, oldestXmin, hibernate);
+ }
+
+ if (TransactionIdIsValid(oldest_xid) &&
+ TransactionIdPrecedes(oldest_xid, oldestXidHavingUndo))
+ {
+ oldestXidHavingUndo = oldest_xid;
+ epoch = GetEpochForXid(oldest_xid);
+ }
+ }
+
+ /*
+ * Update the oldestXidWithEpochHavingUndo in the shared memory.
+ *
+ * XXX In future if multiple worker can perform discard then we may need
+ * to use compare and swap for updating the shared memory value.
+ */
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ MakeEpochXid(epoch, oldestXidHavingUndo));
+}
+
+/*
+ * Fetch the latest urec pointer for the transaction.
+ */
+UndoRecPtr
+FetchLatestUndoPtrForXid(UndoRecPtr urecptr, UnpackedUndoRecord *uur_start,
+ UndoLogControl *log)
+{
+ UndoRecPtr next_urecptr, from_urecptr;
+ uint16 prevlen;
+ UndoLogOffset next_insert;
+ UnpackedUndoRecord *uur;
+ bool refetch = false;
+
+ uur = uur_start;
+
+ while (true)
+ {
+ /* fetch the undo record again if required. */
+ if (refetch)
+ {
+ uur = UndoFetchRecord(urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ refetch = false;
+ }
+
+ next_urecptr = uur->uur_next;
+ prevlen = UndoLogGetPrevLen(log->logno);
+
+ /*
+ * If this is the last transaction in the log then calculate the latest
+ * urec pointer using next insert location of the undo log. Otherwise,
+ * calculate using next transaction's start pointer.
+ */
+ if (uur->uur_next == SpecialUndoRecPtr)
+ {
+ /*
+ * While fetching the next insert location if the new transaction
+ * has already started in this log then lets re-fetch the undo
+ * record.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+ if (!UndoRecPtrIsValid(next_insert))
+ {
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ refetch = true;
+ continue;
+ }
+
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen);
+ break;
+ }
+ else if ((UndoRecPtrGetLogNo(next_urecptr) != log->logno) &&
+ UndoLogIsDiscarded(next_urecptr))
+ {
+ /*
+ * If next_urecptr is in different undolog and its already discarded
+ * that means the undo actions for this transaction which are in the
+ * next log has already been executed and we only need to execute
+ * which are remaining in this log.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+
+ Assert(UndoRecPtrIsValid(next_insert));
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen);
+ break;
+ }
+ else
+ {
+ UnpackedUndoRecord *next_uur;
+
+ next_uur = UndoFetchRecord(next_urecptr,
+ InvalidBlockNumber,
+ InvalidOffsetNumber,
+ InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the next_urecptr is in the same log then calculate the
+ * from pointer using prevlen.
+ */
+ if (UndoRecPtrGetLogNo(next_urecptr) == log->logno)
+ {
+ from_urecptr =
+ UndoGetPrevUndoRecptr(next_urecptr, next_uur->uur_prevlen);
+ UndoRecordRelease(next_uur);
+ break;
+ }
+ else
+ {
+ /*
+ * The transaction is overflowed to the next log, so restart
+ * the processing from then next log.
+ */
+ log = UndoLogGet(UndoRecPtrGetLogNo(next_urecptr), false);
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ uur = next_uur;
+ continue;
+ }
+
+ UndoRecordRelease(next_uur);
+ }
+ }
+
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+
+ return from_urecptr;
+}
+
+/*
+ * Discard the undo logs for temp tables.
+ */
+void
+TempUndoDiscard(UndoLogNumber logno)
+{
+ UndoLogControl *log = UndoLogGet(logno, false);
+
+ /*
+ * Discard the undo log for temp table only. Ensure that there is
+ * something to be discarded there.
+ */
+ Assert (log->meta.persistence == UNDO_TEMP);
+
+ /* Process the undo log. */
+ UndoLogDiscard(MakeUndoRecPtr(log->logno, log->meta.insert),
+ InvalidTransactionId);
+}
diff --git a/src/backend/access/undo/undoinsert.c b/src/backend/access/undo/undoinsert.c
index 2453cad..178b01b 100644
--- a/src/backend/access/undo/undoinsert.c
+++ b/src/backend/access/undo/undoinsert.c
@@ -133,7 +133,6 @@ static UnpackedUndoRecord* UndoGetOneRecord(UnpackedUndoRecord *urec,
UndoPersistence persistence);
static void PrepareUndoRecordUpdateTransInfo(UndoRecPtr urecptr,
bool log_switched);
-static void UndoRecordUpdateTransInfo(void);
static int InsertFindBufferSlot(RelFileNode rnode, BlockNumber blk,
ReadBufferMode rbm,
UndoPersistence persistence);
@@ -271,12 +270,60 @@ PrepareUndoRecordUpdateTransInfo(UndoRecPtr urecptr, bool log_switched)
}
/*
+ * Update the progress of the undo record in the transaction header.
+ */
+void
+PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress)
+{
+ Buffer buffer = InvalidBuffer;
+ BlockNumber cur_blk;
+ RelFileNode rnode;
+ UndoLogNumber logno = UndoRecPtrGetLogNo(urecptr);
+ UndoLogControl *log;
+ Page page;
+ int already_decoded = 0;
+ int starting_byte;
+ int bufidx;
+ int index = 0;
+
+ log = UndoLogGet(logno, false);
+
+ if (log->meta.persistence == UNDO_TEMP)
+ return;
+
+ UndoRecPtrAssignRelFileNode(rnode, urecptr);
+ cur_blk = UndoRecPtrGetBlockNum(urecptr);
+ starting_byte = UndoRecPtrGetPageOffset(urecptr);
+
+ while (true)
+ {
+ bufidx = InsertFindBufferSlot(rnode, cur_blk,
+ RBM_NORMAL,
+ log->meta.persistence);
+ prev_txn_info.prev_txn_undo_buffers[index] = bufidx;
+ buffer = undo_buffer[bufidx].buf;
+ page = BufferGetPage(buffer);
+ index++;
+
+ if (UnpackUndoRecord(&prev_txn_info.uur, page, starting_byte,
+ &already_decoded, true))
+ break;
+
+ starting_byte = UndoLogBlockHeaderSize;
+ cur_blk++;
+ }
+
+ prev_txn_info.prev_urecptr = urecptr;
+ prev_txn_info.uur.uur_progress = progress;
+}
+
+/*
* Overwrite the first undo record of the previous transaction to update its
* next pointer. This will just insert the already prepared record by
* PrepareUndoRecordUpdateTransInfo. This must be called under the critical
* section. This will just overwrite the undo header not the data.
*/
-static void
+void
UndoRecordUpdateTransInfo(void)
{
UndoLogNumber logno = UndoRecPtrGetLogNo(prev_txn_info.prev_urecptr);
diff --git a/src/backend/access/undo/undorecord.c b/src/backend/access/undo/undorecord.c
index 33bb153..5928784 100644
--- a/src/backend/access/undo/undorecord.c
+++ b/src/backend/access/undo/undorecord.c
@@ -95,6 +95,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
*/
if (*already_written == 0)
{
+ work_hdr.urec_rmid = uur->uur_rmid;
work_hdr.urec_type = uur->uur_type;
work_hdr.urec_info = uur->uur_info;
work_hdr.urec_prevlen = uur->uur_prevlen;
@@ -120,6 +121,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
* We should have been passed the same record descriptor as before,
* or caller has messed up.
*/
+ Assert(work_hdr.urec_rmid == uur->uur_rmid);
Assert(work_hdr.urec_type == uur->uur_type);
Assert(work_hdr.urec_info == uur->uur_info);
Assert(work_hdr.urec_prevlen == uur->uur_prevlen);
@@ -282,6 +284,7 @@ bool UnpackUndoRecord(UnpackedUndoRecord *uur, Page page, int starting_byte,
&my_bytes_decoded, already_decoded, false))
return false;
+ uur->uur_rmid = work_hdr.urec_rmid;
uur->uur_type = work_hdr.urec_type;
uur->uur_info = work_hdr.urec_info;
uur->uur_prevlen = work_hdr.urec_prevlen;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index a86963f..8548a65 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1381,6 +1381,7 @@ vac_truncate_clog(TransactionId frozenXID,
MultiXactId lastSaneMinMulti)
{
TransactionId nextXID = ReadNewTransactionId();
+ TransactionId oldestXidHavingUndo;
Relation relation;
HeapScanDesc scan;
HeapTuple tuple;
@@ -1475,6 +1476,16 @@ vac_truncate_clog(TransactionId frozenXID,
return;
/*
+ * We can't truncate the clog for transactions that still have undo. The
+ * oldestXidHavingUndo will be only valid for zheap storage engine, so it
+ * won't impact any other storage engine.
+ */
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ frozenXID = Min(frozenXID, oldestXidHavingUndo);
+
+ /*
* Advance the oldest value for commit timestamps before truncating, so
* that if a user requests a timestamp for a transaction we're truncating
* away right after this point, they get NULL instead of an ugly "file not
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 71c2321..9ce6ff0 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -12,7 +12,7 @@ subdir = src/backend/postmaster
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
- pgarch.o pgstat.o postmaster.o startup.o syslogger.o walwriter.o
+OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o discardworker.o fork_process.o \
+ pgarch.o pgstat.o postmaster.o startup.o syslogger.o undoworker.o walwriter.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index d2b695e..49df516 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -20,7 +20,9 @@
#include "pgstat.h"
#include "port/atomics.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "storage/dsm.h"
@@ -129,6 +131,15 @@ static const struct
},
{
"ApplyWorkerMain", ApplyWorkerMain
+ },
+ {
+ "UndoLauncherMain", UndoLauncherMain
+ },
+ {
+ "UndoWorkerMain", UndoWorkerMain
+ },
+ {
+ "DiscardWorkerMain", DiscardWorkerMain
}
};
diff --git a/src/backend/postmaster/discardworker.c b/src/backend/postmaster/discardworker.c
new file mode 100644
index 0000000..e4c6719
--- /dev/null
+++ b/src/backend/postmaster/discardworker.c
@@ -0,0 +1,170 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.c
+ * The undo discard worker for asynchronous undo management.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/discardworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include <unistd.h>
+
+/* These are always necessary for a bgworker. */
+#include "access/transam.h"
+#include "miscadmin.h"
+#include "postmaster/bgworker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/lwlock.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+#include "access/undodiscard.h"
+#include "pgstat.h"
+#include "postmaster/discardworker.h"
+#include "storage/procarray.h"
+#include "tcop/tcopprot.h"
+#include "utils/guc.h"
+#include "utils/resowner.h"
+
+static void undoworker_sigterm_handler(SIGNAL_ARGS);
+
+/* max sleep time between cycles (100 milliseconds) */
+#define MIN_NAPTIME_PER_CYCLE 100L
+#define DELAYED_NAPTIME 10 * MIN_NAPTIME_PER_CYCLE
+#define MAX_NAPTIME_PER_CYCLE 100 * MIN_NAPTIME_PER_CYCLE
+
+static bool got_SIGTERM = false;
+static bool hibernate = false;
+static long wait_time = MIN_NAPTIME_PER_CYCLE;
+
+/* SIGTERM: set flag to exit at next convenient time */
+static void
+undoworker_sigterm_handler(SIGNAL_ARGS)
+{
+ got_SIGTERM = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+}
+
+/*
+ * DiscardWorkerRegister -- Register a undo discard worker.
+ */
+void
+DiscardWorkerRegister(void)
+{
+ BackgroundWorker bgw;
+
+ /* TODO: This should be configurable. */
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "discard worker");
+ sprintf(bgw.bgw_library_name, "postgres");
+ sprintf(bgw.bgw_function_name, "DiscardWorkerMain");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * DiscardWorkerMain -- Main loop for the undo discard worker.
+ */
+void
+DiscardWorkerMain(Datum main_arg)
+{
+ ereport(LOG,
+ (errmsg("discard worker started")));
+
+ /* Establish signal handlers. */
+ pqsignal(SIGTERM, undoworker_sigterm_handler);
+ BackgroundWorkerUnblockSignals();
+
+ /* Make it easy to identify our processes. */
+ SetConfigOption("application_name", MyBgworkerEntry->bgw_name,
+ PGC_USERSET, PGC_S_SESSION);
+
+ /*
+ * Create resource owner for discard worker as it need to read the undo
+ * records outside the transaction blocks which intern access buffer read
+ * routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ /* Enter main loop */
+ while (!got_SIGTERM)
+ {
+ int rc;
+ TransactionId OldestXmin, oldestXidHavingUndo;
+
+ OldestXmin = GetOldestXmin(NULL, PROCARRAY_FLAGS_DEFAULT);
+
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+
+ /*
+ * Call the discard routine if there oldestXidHavingUndo is lagging
+ * behind OldestXmin.
+ */
+ if (OldestXmin != InvalidTransactionId &&
+ TransactionIdPrecedes(oldestXidHavingUndo, OldestXmin))
+ {
+ UndoDiscard(OldestXmin, &hibernate);
+
+ /*
+ * If we got some undo logs to discard or discarded something,
+ * then reset the wait_time as we have got work to do.
+ * Note that if there are some undologs that cannot be discarded,
+ * then above condition will remain unsatisified till oldestXmin
+ * remains unchanged and the wait_time will not reset in that case.
+ */
+ if (!hibernate)
+ wait_time = MIN_NAPTIME_PER_CYCLE;
+ }
+
+ /* Wait for more work. */
+ rc = WaitLatch(&MyProc->procLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wait_time,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN);
+
+ ResetLatch(&MyProc->procLatch);
+
+ /*
+ * Increase the wait_time based on the length of inactivity. If wait_time
+ * is within one second, then increment it by 100 ms at a time. Henceforth,
+ * increment it one second at a time, till it reaches ten seconds. Never
+ * increase the wait_time more than ten seconds, it will be too much of
+ * waiting otherwise.
+ */
+ if (rc & WL_TIMEOUT && hibernate)
+ {
+ wait_time += (wait_time < DELAYED_NAPTIME ?
+ MIN_NAPTIME_PER_CYCLE : DELAYED_NAPTIME);
+ if (wait_time > MAX_NAPTIME_PER_CYCLE)
+ wait_time = MAX_NAPTIME_PER_CYCLE;
+ }
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+
+ ReleaseAuxProcessResources(true);
+
+ /* we're done */
+ ereport(LOG,
+ (errmsg("discard worker shutting down")));
+
+ proc_exit(0);
+}
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index dc86307..f4b0492 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -3502,6 +3502,12 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_SYSLOGGER_MAIN:
event_name = "SysLoggerMain";
break;
+ case WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN:
+ event_name = "UndoDiscardWorkerMain";
+ break;
+ case WAIT_EVENT_UNDO_LAUNCHER_MAIN:
+ event_name = "UndoLauncherMain";
+ break;
case WAIT_EVENT_WAL_RECEIVER_MAIN:
event_name = "WalReceiverMain";
break;
@@ -3914,7 +3920,6 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_UNDO_FILE_SYNC:
event_name = "UndoFileSync";
break;
-
case WAIT_EVENT_WALSENDER_TIMELINE_HISTORY_READ:
event_name = "WALSenderTimelineHistoryRead";
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 688f462..603c733 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -111,10 +111,12 @@
#include "port/pg_bswap.h"
#include "postmaster/autovacuum.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/walsender.h"
#include "storage/fd.h"
@@ -978,6 +980,11 @@ PostmasterMain(int argc, char *argv[])
*/
ApplyLauncherRegister();
+ UndoLauncherRegister();
+
+ /* Register the Undo Discard worker. */
+ DiscardWorkerRegister();
+
/*
* process any libraries that should be preloaded at postmaster start
*/
diff --git a/src/backend/postmaster/undoworker.c b/src/backend/postmaster/undoworker.c
new file mode 100644
index 0000000..caa7db1
--- /dev/null
+++ b/src/backend/postmaster/undoworker.c
@@ -0,0 +1,664 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.c
+ * undo launcher and undo worker process.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/undoworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "funcapi.h"
+#include "miscadmin.h"
+#include "pgstat.h"
+
+#include "access/heapam.h"
+#include "access/htup.h"
+#include "access/htup_details.h"
+#include "access/sysattr.h"
+#include "access/xact.h"
+
+#include "catalog/indexing.h"
+#include "catalog/pg_database.h"
+
+#include "libpq/pqsignal.h"
+
+#include "postmaster/bgworker.h"
+#include "postmaster/fork_process.h"
+#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
+
+#include "replication/slot.h"
+#include "replication/worker_internal.h"
+
+#include "storage/ipc.h"
+#include "storage/lmgr.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+
+#include "tcop/tcopprot.h"
+
+#include "utils/fmgroids.h"
+#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/resowner.h"
+
+/* max sleep time between cycles (100 milliseconds) */
+#define DEFAULT_NAPTIME_PER_CYCLE 100L
+#define DEFAULT_RETRY_NAPTIME 50L
+
+int max_undo_workers = 5;
+
+typedef struct UndoApplyWorker
+{
+ /* Indicates if this slot is used or free. */
+ bool in_use;
+
+ /* Increased everytime the slot is taken by new worker. */
+ uint16 generation;
+
+ /* Pointer to proc array. NULL if not running. */
+ PGPROC *proc;
+
+ /* Database id to connect to. */
+ Oid dbid;
+} UndoApplyWorker;
+
+UndoApplyWorker *MyUndoWorker = NULL;
+
+typedef struct UndoApplyCtxStruct
+{
+ /* Supervisor process. */
+ pid_t launcher_pid;
+
+ /* Background workers. */
+ UndoApplyWorker workers[FLEXIBLE_ARRAY_MEMBER];
+} UndoApplyCtxStruct;
+
+UndoApplyCtxStruct *UndoApplyCtx;
+
+static void undo_worker_onexit(int code, Datum arg);
+static void undo_worker_cleanup(UndoApplyWorker *worker);
+
+static volatile sig_atomic_t got_SIGHUP = false;
+
+/*
+ * Wait for a background worker to start up and attach to the shmem context.
+ *
+ * This is only needed for cleaning up the shared memory in case the worker
+ * fails to attach.
+ */
+static void
+WaitForUndoWorkerAttach(UndoApplyWorker *worker,
+ uint16 generation,
+ BackgroundWorkerHandle *handle)
+{
+ BgwHandleStatus status;
+ int rc;
+
+ for (;;)
+ {
+ pid_t pid;
+
+ CHECK_FOR_INTERRUPTS();
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+
+ /* Worker either died or has started; no need to do anything. */
+ if (!worker->in_use || worker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ LWLockRelease(UndoWorkerLock);
+
+ /* Check if worker has died before attaching, and clean up after it. */
+ status = GetBackgroundWorkerPid(handle, &pid);
+
+ if (status == BGWH_STOPPED)
+ {
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ /* Ensure that this was indeed the worker we waited for. */
+ if (generation == worker->generation)
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ /*
+ * We need timeout because we generally don't get notified via latch
+ * about the worker attach. But we don't expect to have to wait long.
+ */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 10L, WAIT_EVENT_BGWORKER_STARTUP);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+ }
+
+ return;
+}
+
+/*
+ * Get dbid from the worker slot.
+ */
+static Oid
+slot_get_dbid(int slot)
+{
+ Oid dbid;
+
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty,",
+ slot)));
+ }
+
+ dbid = MyUndoWorker->dbid;
+
+ LWLockRelease(UndoWorkerLock);
+
+ return dbid;
+}
+
+/*
+ * Attach to a slot.
+ */
+static void
+undo_worker_attach(int slot)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty, cannot attach",
+ slot)));
+ }
+
+ if (MyUndoWorker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is already used by "
+ "another worker, cannot attach", slot)));
+ }
+
+ MyUndoWorker->proc = MyProc;
+ before_shmem_exit(undo_worker_onexit, (Datum) 0);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Walks the workers array and searches for one that matches given
+ * dbid.
+ */
+static UndoApplyWorker *
+undo_worker_find(Oid dbid)
+{
+ int i;
+ UndoApplyWorker *res = NULL;
+
+ Assert(LWLockHeldByMe(UndoWorkerLock));
+
+ /* Search for attached worker for a given db id. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (w->in_use && w->dbid == dbid)
+ {
+ res = w;
+ break;
+ }
+ }
+
+ return res;
+}
+
+/*
+ * Check whether the dbid exist or not.
+ *
+ * Refer comments from GetDatabaseTupleByOid.
+ * FIXME: Should we expose GetDatabaseTupleByOid and directly use it.
+ */
+static bool
+dbid_exist(Oid dboid)
+{
+ HeapTuple tuple;
+ Relation relation;
+ SysScanDesc scan;
+ ScanKeyData key[1];
+ bool result = false;
+ /*
+ * form a scan key
+ */
+ ScanKeyInit(&key[0],
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dboid));
+
+ relation = heap_open(DatabaseRelationId, AccessShareLock);
+ scan = systable_beginscan(relation, DatabaseOidIndexId,
+ criticalSharedRelcachesBuilt,
+ NULL,
+ 1, key);
+
+ tuple = systable_getnext(scan);
+
+ if (HeapTupleIsValid(tuple))
+ result = true;
+
+ /* all done */
+ systable_endscan(scan);
+ heap_close(relation, AccessShareLock);
+
+ return result;
+}
+
+/*
+ * Start new undo apply background worker, if possible otherwise return false.
+ */
+static bool
+undo_worker_launch(Oid dbid)
+{
+ BackgroundWorker bgw;
+ BackgroundWorkerHandle *bgw_handle;
+ uint16 generation;
+ int i;
+ int slot = 0;
+ UndoApplyWorker *worker = NULL;
+
+ /*
+ * We need to do the modification of the shared memory under lock so that
+ * we have consistent view.
+ */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ /* Find unused worker slot. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (!w->in_use)
+ {
+ worker = w;
+ slot = i;
+ break;
+ }
+ }
+
+ /* There are no more free worker slots */
+ if (worker == NULL)
+ return false;
+
+ /* Prepare the worker slot. */
+ worker->in_use = true;
+ worker->proc = NULL;
+ worker->dbid = dbid;
+ worker->generation++;
+
+ generation = worker->generation;
+ LWLockRelease(UndoWorkerLock);
+
+ /* Register the new dynamic worker. */
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoWorkerMain");
+ snprintf(bgw.bgw_type, BGW_MAXLEN, "undo apply worker");
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "undo apply worker");
+
+ bgw.bgw_restart_time = BGW_NEVER_RESTART;
+ bgw.bgw_notify_pid = MyProcPid;
+ bgw.bgw_main_arg = Int32GetDatum(slot);
+
+ StartTransactionCommand();
+ /* Check the database exists or not. */
+ if (!dbid_exist(dbid))
+ {
+ CommitTransactionCommand();
+
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return true;
+ }
+
+ /*
+ * Acquire database object lock before launching the worker so that it
+ * doesn't get dropped while worker is connecting to the database.
+ */
+ LockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+
+ /* Recheck whether database still exists or not. */
+ if (!dbid_exist(dbid))
+ {
+ CommitTransactionCommand();
+
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return true;
+ }
+
+ if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle))
+ {
+ /* Failed to start worker, so clean up the worker slot. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+
+ UnlockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+ CommitTransactionCommand();
+
+ return false;
+ }
+
+ /* Now wait until it attaches. */
+ WaitForUndoWorkerAttach(worker, generation, bgw_handle);
+
+ /*
+ * By this point the undo-worker has already connected to the database so we
+ * can release the database lock.
+ */
+ UnlockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+ CommitTransactionCommand();
+
+ return true;
+}
+
+/*
+ * Detach the worker (cleans up the worker info).
+ */
+static void
+undo_worker_detach(void)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ undo_worker_cleanup(MyUndoWorker);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Clean up worker info.
+ */
+static void
+undo_worker_cleanup(UndoApplyWorker *worker)
+{
+ Assert(LWLockHeldByMeInMode(UndoWorkerLock, LW_EXCLUSIVE));
+
+ worker->in_use = false;
+ worker->proc = NULL;
+ worker->dbid = InvalidOid;
+}
+
+/*
+ * Cleanup function for undo worker launcher.
+ *
+ * Called on undo worker launcher exit.
+ */
+static void
+undo_launcher_onexit(int code, Datum arg)
+{
+ UndoApplyCtx->launcher_pid = 0;
+}
+
+/* SIGHUP: set flag to reload configuration at next convenient time */
+static void
+undo_launcher_sighup(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGHUP = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+undo_worker_onexit(int code, Datum arg)
+{
+ undo_worker_detach();
+}
+
+/*
+ * UndoLauncherShmemSize
+ * Compute space needed for undo launcher shared memory
+ */
+Size
+UndoLauncherShmemSize(void)
+{
+ Size size;
+
+ /*
+ * Need the fixed struct and the array of LogicalRepWorker.
+ */
+ size = sizeof(UndoApplyCtxStruct);
+ size = MAXALIGN(size);
+ size = add_size(size, mul_size(max_undo_workers,
+ sizeof(UndoApplyWorker)));
+ return size;
+}
+
+/*
+ * UndoLauncherRegister
+ * Register a background worker running the undo worker launcher.
+ */
+void
+UndoLauncherRegister(void)
+{
+ BackgroundWorker bgw;
+
+ if (max_undo_workers == 0)
+ return;
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoLauncherMain");
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "undo worker launcher");
+ snprintf(bgw.bgw_type, BGW_MAXLEN,
+ "undo worker launcher");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * UndoLauncherShmemInit
+ * Allocate and initialize undo worker launcher shared memory
+ */
+void
+UndoLauncherShmemInit(void)
+{
+ bool found;
+
+ UndoApplyCtx = (UndoApplyCtxStruct *)
+ ShmemInitStruct("Undo Worker Launcher Data",
+ UndoLauncherShmemSize(),
+ &found);
+
+ if (!found)
+ memset(UndoApplyCtx, 0, UndoLauncherShmemSize());
+}
+
+/*
+ * Main loop for the undo worker launcher process.
+ */
+void
+UndoLauncherMain(Datum main_arg)
+{
+ MemoryContext tmpctx;
+ MemoryContext oldctx;
+
+ ereport(DEBUG1,
+ (errmsg("undo launcher started")));
+
+ before_shmem_exit(undo_launcher_onexit, (Datum) 0);
+
+ Assert(UndoApplyCtx->launcher_pid == 0);
+ UndoApplyCtx->launcher_pid = MyProcPid;
+
+ /* Establish signal handlers. */
+ pqsignal(SIGHUP, undo_launcher_sighup);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * Establish connection to nailed catalogs (we only ever access
+ * pg_subscription).
+ */
+ BackgroundWorkerInitializeConnection(NULL, NULL, 0);
+
+ /* Use temporary context for the database list and worker info. */
+ tmpctx = AllocSetContextCreate(TopMemoryContext,
+ "Undo worker Launcher context",
+ ALLOCSET_DEFAULT_SIZES);
+ /* Enter main loop */
+ for (;;)
+ {
+ int rc;
+ List *dblist;
+ ListCell *l;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* switch to the temp context. */
+ oldctx = MemoryContextSwitchTo(tmpctx);
+ dblist = RollbackHTGetDBList();
+
+ foreach(l, dblist)
+ {
+ UndoApplyWorker *w;
+ Oid dbid = lfirst_oid(l);
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+ w = undo_worker_find(dbid);
+ LWLockRelease(UndoWorkerLock);
+
+ if (w == NULL)
+ {
+retry:
+ if (!undo_worker_launch(dbid))
+ {
+ /* Could not launch the worker, retry after sometime, */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_RETRY_NAPTIME,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+ goto retry;
+ }
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+
+ /* Clean the temporary memory. */
+ MemoryContextReset(tmpctx);
+
+ /* Wait for more work. */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_NAPTIME_PER_CYCLE,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ if (got_SIGHUP)
+ {
+ got_SIGHUP = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+}
+
+/*
+ * UndoWorkerMain -- Main loop for the undo apply worker.
+ */
+void
+UndoWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ Oid dbid;
+
+ dbid = slot_get_dbid(worker_slot);
+
+ /* Setup signal handling */
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /* Connect to the database. */
+ BackgroundWorkerInitializeConnectionByOid(dbid, 0, 0);
+
+ /* Attach to slot */
+ undo_worker_attach(worker_slot);
+
+ /*
+ * Create resource owner for undo worker. Undo worker need this as it
+ * need to read the undo records outside the transaction blocks which
+ * intern access buffer read routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ RollbackFromHT(dbid);
+
+ ReleaseAuxProcessResources(true);
+
+ proc_exit(0);
+}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index f60ecc5..49e1028 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -159,6 +159,10 @@ LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *recor
ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(record),
buf.origptr);
break;
+ case RM_UNDOACTION_ID:
+ /* Logical decoding is not yet implemented for undoactions. */
+ Assert(0);
+ break;
case RM_NEXT_ID:
elog(ERROR, "unexpected RM_NEXT_ID rmgr_id: %u", (RmgrIds) XLogRecGetRmid(buf.record));
}
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 4725cbe..1470041 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -29,6 +29,8 @@
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
#include "replication/walreceiver.h"
@@ -152,6 +154,8 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
size = add_size(size, BackendRandomShmemSize());
+ size = add_size(size, RollbackHTSize());
+ size = add_size(size, UndoLauncherShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -226,6 +230,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
SUBTRANSShmemInit();
MultiXactShmemInit();
InitBufferPool();
+ InitRollbackHashTable();
/*
* Set up lock manager
@@ -264,6 +269,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
WalSndShmemInit();
WalRcvShmemInit();
ApplyLauncherShmemInit();
+ UndoLauncherShmemInit();
/*
* Set up other modules that need some shared memory space
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 554af46..52f6959 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -51,3 +51,5 @@ BackendRandomLock 43
LogicalRepWorkerLock 44
CLogTruncationLock 45
UndoLogLock 46
+RollbackHTLock 47
+UndoWorkerLock 48
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 6f9aaa5..3138b04 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -279,6 +279,8 @@ InitProcGlobal(void)
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemAlloc(sizeof(slock_t));
SpinLockInit(ProcStructLock);
+
+ pg_atomic_init_u64(&ProcGlobal->oldestXidWithEpochHavingUndo, 0);
}
/*
diff --git a/src/backend/utils/adt/lockfuncs.c b/src/backend/utils/adt/lockfuncs.c
index 66c09a1..67dda39 100644
--- a/src/backend/utils/adt/lockfuncs.c
+++ b/src/backend/utils/adt/lockfuncs.c
@@ -32,6 +32,7 @@ const char *const LockTagTypeNames[] = {
"virtualxid",
"speculative token",
"object",
+ "undoaction",
"userlock",
"advisory"
};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index c693977..12e7704 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -121,6 +121,7 @@ bool allowSystemTableMods = false;
int work_mem = 1024;
int maintenance_work_mem = 16384;
int max_parallel_maintenance_workers = 2;
+int rollback_overflow_size = 64;
/*
* Primary determinants of sizes of shared-memory structures.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 287ca00..85a7f11 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2806,6 +2806,17 @@ static struct config_int ConfigureNamesInt[] =
},
{
+ {"rollback_overflow_size", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Rollbacks greater than this size are done lazily"),
+ NULL,
+ GUC_UNIT_MB
+ },
+ &rollback_overflow_size,
+ 64, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
{"wal_segment_size", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the size of write ahead log segments."),
NULL,
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index 3fc8b6a..5a5d98c 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -78,8 +78,8 @@ pg_control_system(PG_FUNCTION_ARGS)
Datum
pg_control_checkpoint(PG_FUNCTION_ARGS)
{
- Datum values[19];
- bool nulls[19];
+ Datum values[20];
+ bool nulls[20];
TupleDesc tupdesc;
HeapTuple htup;
ControlFileData *ControlFile;
@@ -128,6 +128,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
XIDOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 18, "checkpoint_time",
TIMESTAMPTZOID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 19, "oldest_xid_with_epoch_having_undo",
+ TIMESTAMPTZOID, -1, 0);
tupdesc = BlessTupleDesc(tupdesc);
/* Read the control file. */
@@ -202,6 +204,9 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
time_t_to_timestamptz(ControlFile->checkPointCopy.time));
nulls[17] = false;
+ values[18] = Int64GetDatum(ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo);
+ nulls[18] = false;
+
htup = heap_form_tuple(tupdesc, values, nulls);
PG_RETURN_DATUM(HeapTupleGetDatum(htup));
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 4e61bc6..f3bf0fd 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -686,4 +686,11 @@
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
+# If often there are large transactions requiring rollbacks, then we can push
+# them to undo-workers for better performance. The size specifeid by the
+# parameter below, determines the minimum size of the rollback requests to be
+# sent to the undo-worker.
+#
+#rollback_overflow_size = 64
+
# Add settings for extensions here
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 895a51f..753929e 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -278,6 +278,8 @@ main(int argc, char *argv[])
ControlFile->checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile->checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo);
printf(_("Time of latest checkpoint: %s\n"),
ckpttime_str);
printf(_("Fake LSN counter for unlogged rels: %X/%X\n"),
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 6fb403a..72714dd 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -448,6 +448,7 @@ main(int argc, char *argv[])
if (ControlFile.checkPointCopy.oldestXid < FirstNormalTransactionId)
ControlFile.checkPointCopy.oldestXid += FirstNormalTransactionId;
ControlFile.checkPointCopy.oldestXidDB = InvalidOid;
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo = 0;
}
if (set_oldest_commit_ts_xid != 0)
@@ -716,6 +717,8 @@ GuessControlValues(void)
ControlFile.checkPointCopy.oldestMultiDB = InvalidOid;
ControlFile.checkPointCopy.time = (pg_time_t) time(NULL);
ControlFile.checkPointCopy.oldestActiveXid = InvalidTransactionId;
+ ControlFile.checkPointCopy.nextXid = 0;
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo = 0;
ControlFile.state = DB_SHUTDOWNED;
ControlFile.time = (pg_time_t) time(NULL);
@@ -808,6 +811,8 @@ PrintControlValues(bool guessed)
ControlFile.checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile.checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo);
printf(_("Maximum data alignment: %u\n"),
ControlFile.maxAlign);
/* we don't print floatFormat since can't say much useful about it */
@@ -884,6 +889,8 @@ PrintNewControlValues(void)
ControlFile.checkPointCopy.oldestXid);
printf(_("OldestXID's DB: %u\n"),
ControlFile.checkPointCopy.oldestXidDB);
+ printf(_("OldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo);
}
if (set_xid_epoch != -1)
diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c
index 4002847..e71a71e 100644
--- a/src/bin/pg_rewind/parsexlog.c
+++ b/src/bin/pg_rewind/parsexlog.c
@@ -29,7 +29,7 @@
* RmgrNames is an array of resource manager names, to make error messages
* a bit nicer.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
name,
static const char *RmgrNames[RM_MAX_ID + 1] = {
diff --git a/src/bin/pg_waldump/rmgrdesc.c b/src/bin/pg_waldump/rmgrdesc.c
index 938150d..3ef0a60 100644
--- a/src/bin/pg_waldump/rmgrdesc.c
+++ b/src/bin/pg_waldump/rmgrdesc.c
@@ -20,6 +20,7 @@
#include "access/nbtxlog.h"
#include "access/rmgr.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -33,7 +34,7 @@
#include "storage/standbydefs.h"
#include "utils/relmapper.h"
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
{ name, desc, identify},
const RmgrDescData RmgrDescTable[RM_MAX_ID + 1] = {
diff --git a/src/include/access/rmgr.h b/src/include/access/rmgr.h
index c9b5c56..e1fb42a 100644
--- a/src/include/access/rmgr.h
+++ b/src/include/access/rmgr.h
@@ -19,7 +19,7 @@ typedef uint8 RmgrId;
* Note: RM_MAX_ID must fit in RmgrId; widening that type will affect the XLOG
* file format.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
symname,
typedef enum RmgrIds
diff --git a/src/include/access/rmgrlist.h b/src/include/access/rmgrlist.h
index 9c6fca4..eee7835 100644
--- a/src/include/access/rmgrlist.h
+++ b/src/include/access/rmgrlist.h
@@ -25,26 +25,27 @@
*/
/* symbol name, textual name, redo, desc, identify, startup, cleanup */
-PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL)
-PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL)
-PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL)
-PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL)
-PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL)
-PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask)
-PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask)
-PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask)
-PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask)
-PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask)
-PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask)
-PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask)
-PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL)
-PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL)
-PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask)
-PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL)
-PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL)
+PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask, NULL, NULL)
+PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask, NULL, NULL)
+PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask, NULL, NULL)
+PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask, NULL, NULL)
+PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask, NULL, NULL)
+PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask, NULL, NULL)
+PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask, NULL, NULL)
+PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask, NULL, NULL)
+PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOACTION_ID, "UndoAction", undoaction_redo, undoaction_desc, undoaction_identify, NULL, NULL, NULL, NULL, NULL)
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 83ec3f1..7b983ef 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -68,6 +68,10 @@
(AssertMacro(TransactionIdIsNormal(id1) && TransactionIdIsNormal(id2)), \
(int32) ((id1) - (id2)) > 0)
+/* Extract xid from a value comprised of epoch and xid */
+#define GetXidFromEpochXid(epochxid) \
+ ((uint32) (epochxid) & 0XFFFFFFFF)
+
/* ----------
* Object ID (OID) zero is InvalidOid.
*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 0e932da..fb05c3b 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -41,7 +41,7 @@ extern GlobalTransaction MarkAsPreparing(TransactionId xid, const char *gid,
TimestampTz prepared_at,
Oid owner, Oid databaseid);
-extern void StartPrepare(GlobalTransaction gxact);
+extern void StartPrepare(GlobalTransaction gxact, UndoRecPtr *, UndoRecPtr *);
extern void EndPrepare(GlobalTransaction gxact);
extern bool StandbyTransactionIdIsPrepared(TransactionId xid);
diff --git a/src/include/access/undoaction.h b/src/include/access/undoaction.h
new file mode 100644
index 0000000..5455259
--- /dev/null
+++ b/src/include/access/undoaction.h
@@ -0,0 +1,28 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.h
+ * undo action prototypes
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_H
+#define UNDOACTION_H
+
+#include "postgres.h"
+
+#include "access/undolog.h"
+#include "access/undorecord.h"
+
+/* undo record information */
+typedef struct UndoRecInfo
+{
+ UndoRecPtr urp; /* undo recptr (undo record location). */
+ UnpackedUndoRecord *uur; /* actual undo record. */
+} UndoRecInfo;
+
+#endif
diff --git a/src/include/access/undoaction_xlog.h b/src/include/access/undoaction_xlog.h
new file mode 100644
index 0000000..bfc6418
--- /dev/null
+++ b/src/include/access/undoaction_xlog.h
@@ -0,0 +1,74 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction_xlog.h
+ * undo action XLOG definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction_xlog.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_XLOG_H
+#define UNDOACTION_XLOG_H
+
+#include "access/undolog.h"
+#include "access/xlogreader.h"
+#include "lib/stringinfo.h"
+#include "storage/off.h"
+
+/*
+ * WAL record definitions for undoactions.c's WAL operations
+ */
+#define XLOG_UNDO_PAGE 0x00
+#define XLOG_UNDO_RESET_SLOT 0x10
+#define XLOG_UNDO_APPLY_PROGRESS 0x20
+
+/*
+ * xl_undoaction_page flag values, 8 bits are available.
+ */
+#define XLU_PAGE_CONTAINS_TPD_SLOT (1<<0)
+#define XLU_PAGE_CLEAR_VISIBILITY_MAP (1<<1)
+#define XLU_CONTAINS_TPD_OFFSET_MAP (1<<2)
+#define XLU_INIT_PAGE (1<<3)
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_page
+{
+ UndoRecPtr urec_ptr;
+ TransactionId xid;
+ int trans_slot_id; /* transaction slot id */
+} xl_undoaction_page;
+
+#define SizeOfUndoActionPage (offsetof(xl_undoaction_page, trans_slot_id) + sizeof(int))
+
+/* This is what we need to know about undo apply progress */
+typedef struct xl_undoapply_progress
+{
+ UndoRecPtr urec_ptr;
+ uint32 progress;
+} xl_undoapply_progress;
+
+#define SizeOfUndoActionProgress (offsetof(xl_undoapply_progress, progress) + sizeof(uint32))
+
+/*
+ * xl_undoaction_reset_slot flag values, 8 bits are available.
+ */
+#define XLU_RESET_CONTAINS_TPD_SLOT (1<<0)
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_reset_slot
+{
+ UndoRecPtr urec_ptr;
+ int trans_slot_id; /* transaction slot id */
+ uint8 flags;
+} xl_undoaction_reset_slot;
+
+#define SizeOfUndoActionResetSlot (offsetof(xl_undoaction_reset_slot, flags) + sizeof(uint8))
+
+extern void undoaction_redo(XLogReaderState *record);
+extern void undoaction_desc(StringInfo buf, XLogReaderState *record);
+extern const char *undoaction_identify(uint8 info);
+
+#endif /* UNDOACTION_XLOG_H */
diff --git a/src/include/access/undodiscard.h b/src/include/access/undodiscard.h
new file mode 100644
index 0000000..70d6408
--- /dev/null
+++ b/src/include/access/undodiscard.h
@@ -0,0 +1,31 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoinsert.h
+ * undo discard definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undodiscard.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDODISCARD_H
+#define UNDODISCARD_H
+
+#include "access/undolog.h"
+#include "access/xlogdefs.h"
+#include "catalog/pg_class.h"
+#include "storage/lwlock.h"
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+extern void UndoDiscard(TransactionId xmin, bool *hibernate);
+
+#endif /* UNDODISCARD_H */
diff --git a/src/include/access/undoinsert.h b/src/include/access/undoinsert.h
index 2b73f9b..fda3082 100644
--- a/src/include/access/undoinsert.h
+++ b/src/include/access/undoinsert.h
@@ -96,6 +96,8 @@ extern void UndoSetPrepareSize(int max_prepare, UnpackedUndoRecord *undorecords,
extern UndoRecPtr UndoGetPrevUndoRecptr(UndoRecPtr urp, uint16 prevlen);
extern void UndoRecordOnUndoLogChange(UndoPersistence persistence);
+extern void PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress);
+extern void UndoRecordUpdateTransInfo(void);
#endif /* UNDOINSERT_H */
diff --git a/src/include/access/undorecord.h b/src/include/access/undorecord.h
index 85642ad..13512c9 100644
--- a/src/include/access/undorecord.h
+++ b/src/include/access/undorecord.h
@@ -20,6 +20,17 @@
#include "storage/buf.h"
#include "storage/off.h"
+typedef enum undorectype
+{
+ UNDO_INSERT,
+ UNDO_MULTI_INSERT,
+ UNDO_DELETE,
+ UNDO_INPLACE_UPDATE,
+ UNDO_UPDATE,
+ UNDO_XID_LOCK_ONLY,
+ UNDO_XID_MULTI_LOCK_ONLY,
+ UNDO_ITEMID_UNUSED
+} undorectype;
/*
* Every undo record begins with an UndoRecordHeader structure, which is
@@ -30,6 +41,7 @@
*/
typedef struct UndoRecordHeader
{
+ RmgrId urec_rmid; /* RMGR [XXX:TODO: this creates an alignment hole?] */
uint8 urec_type; /* record type code */
uint8 urec_info; /* flag bits */
uint16 urec_prevlen; /* length of previous record in bytes */
@@ -148,6 +160,7 @@ typedef struct UndoRecordPayload
*/
typedef struct UnpackedUndoRecord
{
+ RmgrId uur_rmid; /* rmgr ID */
uint8 uur_type; /* record type code */
uint8 uur_info; /* flag bits */
uint16 uur_prevlen; /* length of previous record */
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 73394c5..ad8cabe 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -419,6 +419,7 @@ extern XLogRecPtr XactLogAbortRecord(TimestampTz abort_time,
int xactflags, TransactionId twophase_xid,
const char *twophase_gid);
extern void xact_redo(XLogReaderState *record);
+extern void XactPerformUndoActionsIfPending(void);
/* xactdesc.c */
extern void xact_desc(StringInfo buf, XLogReaderState *record);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 8cfcd44..89dc2b5 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -192,6 +192,9 @@ extern bool XLOG_DEBUG;
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+/* Generate a 64-bit xid by using epoch and 32-bit xid. */
+#define MakeEpochXid(epoch, xid) \
+ ((epoch << 32) | (xid))
/* Checkpoint statistics */
typedef struct CheckpointStatsData
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 30610b3..9226cff 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -21,8 +21,10 @@
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
+#include "access/undorecord.h"
#include "datatype/timestamp.h"
#include "lib/stringinfo.h"
+#include "nodes/pg_list.h"
#include "pgtime.h"
#include "storage/block.h"
#include "storage/relfilenode.h"
@@ -294,9 +296,14 @@ typedef struct RmgrData
void (*rm_startup) (void);
void (*rm_cleanup) (void);
void (*rm_mask) (char *pagedata, BlockNumber blkno);
+ bool (*rm_undo) (List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock,
+ int options);
+ void (*rm_undo_desc) (StringInfo buf, UnpackedUndoRecord *record);
} RmgrData;
-extern const RmgrData RmgrTable[];
+extern PGDLLIMPORT const RmgrData RmgrTable[];
/*
* Exported to support xlog switching from checkpointer
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 773d9e6..6918efc 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -61,6 +61,13 @@ typedef struct CheckPoint
* set to InvalidTransactionId.
*/
TransactionId oldestActiveXid;
+
+ /*
+ * Oldest transaction id with epoc which is having undo. Include this value
+ * in the checkpoint record so that whenever server starts we get proper
+ * value.
+ */
+ uint64 oldestXidWithEpochHavingUndo;
} CheckPoint;
/* XLOG info values for XLOG rmgr */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index d6b32c0..549bb38 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -245,6 +245,7 @@ extern PGDLLIMPORT bool allowSystemTableMods;
extern PGDLLIMPORT int work_mem;
extern PGDLLIMPORT int maintenance_work_mem;
extern PGDLLIMPORT int max_parallel_maintenance_workers;
+extern PGDLLIMPORT int rollback_overflow_size;
extern int VacuumCostPageHit;
extern int VacuumCostPageMiss;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 763379e..7c5f05e 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -749,6 +749,7 @@ typedef enum BackendState
#define PG_WAIT_TIMEOUT 0x09000000U
#define PG_WAIT_IO 0x0A000000U
#define PG_WAIT_PAGE_TRANS_SLOT 0x0B000000U
+#define PG_WAIT_ROLLBACK_HT 0x0C000000U
/* ----------
* Wait Events - Activity
@@ -774,6 +775,8 @@ typedef enum
WAIT_EVENT_WAL_RECEIVER_MAIN,
WAIT_EVENT_WAL_SENDER_MAIN,
WAIT_EVENT_WAL_WRITER_MAIN,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN
} WaitEventActivity;
/* ----------
diff --git a/src/include/postmaster/discardworker.h b/src/include/postmaster/discardworker.h
new file mode 100644
index 0000000..f00c6c4
--- /dev/null
+++ b/src/include/postmaster/discardworker.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.h
+ * Exports from postmaster/discardworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/discardworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _DISCARDWORKER_H
+#define _DISCARDWORKER_H
+
+/*
+ * This function will perform multiple actions based on need. (a) retrieve
+ * transactions which have become all-visible and truncate the associated undo
+ * logs or will increment the tail pointer. (b) drop the buffers corresponding
+ * to truncated pages.
+ */
+extern void DiscardWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void DiscardWorkerRegister(void);
+
+#endif /* _DISCARDWORKER_H */
diff --git a/src/include/postmaster/undoloop.h b/src/include/postmaster/undoloop.h
new file mode 100644
index 0000000..21d91c3
--- /dev/null
+++ b/src/include/postmaster/undoloop.h
@@ -0,0 +1,89 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoloop.h
+ * Exports from postmaster/undoloop.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ *
+ * src/include/postmaster/undoloop.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOLOOP_H
+#define _UNDOLOOP_H
+
+#include "access/undoinsert.h"
+#include "utils/hsearch.h"
+#include "utils/relcache.h"
+
+
+/* Various options while executing the undo actions for the page. */
+#define UNDO_ACTION_UPDATE_TPD 0x0001
+
+/* Remembers the last seen RecentGlobalXmin */
+TransactionId latestRecentGlobalXmin;
+
+/*
+ * This function will read the undo records starting from the undo
+ * from_urecptr till to_urecptr and if to_urecptr is invalid then till the
+ * first undo location of transaction. This also discards the buffers by
+ * calling DropUndoBuffers for which undo log is removed. This function
+ * can be used by RollbackToSavePoint, by Rollback, by undoworker to complete
+ * the work of errored out transactions or when there is an error in single
+ * user mode.
+ */
+extern void execute_undo_actions(UndoRecPtr from_urecptr,
+ UndoRecPtr to_urecptr, bool nopartial, bool rewind, bool rellock);
+extern void process_and_execute_undo_actions_page(UndoRecPtr from_urecptr,
+ Relation rel, Buffer buffer, uint32 epoch,
+ TransactionId xid, int slot_no);
+
+/*
+ * This function will be responsible to truncate the undo logs
+ * for transactions that become all-visible after RecentGlobalXmin has
+ * advanced (value is different than latestRecentGlobalXmin). The easiest
+ * way could be to traverse the undo log array that contains least transaction
+ * id for that undo log and see if it precedes RecentGlobalXmin, then start
+ * discarding the undo log for that transaction (moving the tail pointer of
+ * undo log) till it finds the transaction which is not all-visible. This also
+ * discards the buffers by calling ForgetBuffer for which undo log is
+ * removed. This function can be invoked by undoworker or after commit in
+ * single user mode.
+ */
+extern void recover_undo_pages();
+
+/*
+ * To increase the efficiency of the zheap system, we create a hash table for
+ * the rollbacks. All the rollback requests exceeding certain threshold, are
+ * pushed to this table. Undo worker starts reading the entries from this hash
+ * table one at a time, performs undo actions related to the respective xid and
+ * removes them from the hash table. This way backend is free from performing the
+ * undo actions in case of heavy rollbacks. The data structures and the routines
+ * required for this infrastructure are as follows.
+ */
+
+/* This is the data structure for each hash table entry for rollbacks. */
+typedef struct RollbackHashEntry
+{
+ UndoRecPtr start_urec_ptr;
+ UndoRecPtr end_urec_ptr;
+ Oid dbid;
+} RollbackHashEntry;
+
+extern bool RollbackHTIsFull(void);
+
+/* To push the rollback requests from backend to the respective hash table */
+extern bool PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr,
+ Oid dbid);
+
+/* To perform the undo actions reading from the hash table */
+extern void RollbackFromHT(Oid dbid);
+/* To calculate the size of the hash table size for rollabcks. */
+extern int RollbackHTSize(void);
+
+/* To initialize the hash table in shared memory for rollbacks. */
+extern void InitRollbackHashTable(void);
+extern List *RollbackHTGetDBList(void);
+extern bool ConditionTransactionUndoActionLock(TransactionId xid);
+extern void TransactionUndoActionLockRelease(TransactionId xid);
+#endif /* _UNDOLOOP_H */
diff --git a/src/include/postmaster/undoworker.h b/src/include/postmaster/undoworker.h
new file mode 100644
index 0000000..c277e01
--- /dev/null
+++ b/src/include/postmaster/undoworker.h
@@ -0,0 +1,39 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.h
+ * Exports from postmaster/undoworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/undoworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOWORKER_H
+#define _UNDOWORKER_H
+
+/* GUC options */
+/* undo worker sleep time between rounds */
+extern int UndoWorkerDelay;
+
+/*
+ * This function will perform multiple actions based on need. (a) retreive
+ * transaction and its corresponding undopoiter from shared memory queue and
+ * call undoloop to perform undo actions. After applying all the undo records
+ * for a particular transaction, it will increment the tail pointer in undo log.
+ * (b) it needs to retrieve transactions which have become all-visible and truncate
+ * the associated undo logs or will increment the tail pointer. (c) udjust the
+ * number of undo workers based on the work required to perform undo actions
+ * (it could be size of shared memory queue containing transactions that needs
+ * aborts). (d) drop the buffers corresponding to truncated pages (e) Sleep for
+ * UndoWorkerDelay, if there is no more work.
+ */
+extern void UndoWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void UndoLauncherRegister(void);
+extern void UndoLauncherShmemInit(void);
+extern Size UndoLauncherShmemSize(void);
+extern void UndoLauncherMain(Datum main_arg);
+extern void UndoWorkerMain(Datum main_arg);
+
+#endif /* _UNDOWORKER_H */
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index a37fda7..675761c 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -150,6 +150,8 @@ typedef enum LockTagType
LOCKTAG_VIRTUALTRANSACTION, /* virtual transaction (ditto) */
/* ID info for a virtual transaction is its VirtualTransactionId */
LOCKTAG_SPECULATIVE_TOKEN, /* speculative insertion Xid and token */
+ /* ID info for an transaction undoaction is transaction id */
+ LOCKTAG_TRANSACTION_UNDOACTION, /* transaction (waiting for undoaction) */
/* ID info for a transaction is its TransactionId */
LOCKTAG_OBJECT, /* non-relation database object */
/* ID info for an object is DB OID + CLASS OID + OBJECT OID + SUBID */
@@ -246,6 +248,14 @@ typedef struct LOCKTAG
(locktag).locktag_type = LOCKTAG_SPECULATIVE_TOKEN, \
(locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+#define SET_LOCKTAG_TRANSACTION_UNDOACTION(locktag,xid) \
+ ((locktag).locktag_field1 = (xid), \
+ (locktag).locktag_field2 = 0, \
+ (locktag).locktag_field3 = 0, \
+ (locktag).locktag_field4 = 0, \
+ (locktag).locktag_type = LOCKTAG_TRANSACTION_UNDOACTION, \
+ (locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+
#define SET_LOCKTAG_OBJECT(locktag,dboid,classoid,objoid,objsubid) \
((locktag).locktag_field1 = (dboid), \
(locktag).locktag_field2 = (classoid), \
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index cb613c8..f6b9d98 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -270,6 +270,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Oldest transaction id which is having undo. */
+ pg_atomic_uint64 oldestXidWithEpochHavingUndo;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
1.8.3.1
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: Undo worker and transaction rollback
2018-11-05 11:40 Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-05 12:32 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
@ 2018-11-30 15:13 ` Dmitry Dolgov <[email protected]>
2018-12-03 12:44 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Dmitry Dolgov @ 2018-11-30 15:13 UTC (permalink / raw)
To: [email protected]; +Cc: pgsql-hackers
> On Mon, Nov 5, 2018 at 1:32 PM Dilip Kumar <[email protected]> wrote:
>
> Updated patch, include defect fix from Kuntal posted on [1].
>
> [1] https://www.postgresql.org/message-id/CAGz5QCKpCG6egFAdazC%2BJgyk7YSE1OBN9h-QpwCkg-NnSWN5AQ%40mail.g...
Thanks for working on this feature. Unfortunately, looks like the current
version of patch has some conflicts with the current master, could you please
post an updated version again?
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: Undo worker and transaction rollback
2018-11-05 11:40 Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-05 12:32 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-30 15:13 ` Re: Undo worker and transaction rollback Dmitry Dolgov <[email protected]>
@ 2018-12-03 12:44 ` Dilip Kumar <[email protected]>
2018-12-04 09:43 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Dilip Kumar @ 2018-12-03 12:44 UTC (permalink / raw)
To: [email protected]; +Cc: pgsql-hackers
On Fri, 30 Nov 2018, 20:42 Dmitry Dolgov <[email protected] wrote:
> > On Mon, Nov 5, 2018 at 1:32 PM Dilip Kumar <[email protected]>
> wrote:
> >
> > Updated patch, include defect fix from Kuntal posted on [1].
> >
> > [1]
> https://www.postgresql.org/message-id/CAGz5QCKpCG6egFAdazC%2BJgyk7YSE1OBN9h-QpwCkg-NnSWN5AQ%40mail.g...
>
> Thanks for working on this feature. Unfortunately, looks like the current
> version of patch has some conflicts with the current master, could you
> please
> post an updated version again?
>
Currently, I am in process rebasing undolog patch set[1]. Post that I will
rebase this patch on top of those patch set.
[1]
https://www.postgresql.org/message-id/CAFiTN-syRxU3jTpkOxHQsEqKC95LGd86JTdZ2stozyXWDUSffg%40mail.gma...
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: Undo worker and transaction rollback
2018-11-05 11:40 Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-05 12:32 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-30 15:13 ` Re: Undo worker and transaction rollback Dmitry Dolgov <[email protected]>
2018-12-03 12:44 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
@ 2018-12-04 09:43 ` Dilip Kumar <[email protected]>
2018-12-31 05:34 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Dilip Kumar @ 2018-12-04 09:43 UTC (permalink / raw)
To: [email protected]; +Cc: pgsql-hackers
On Mon, Dec 3, 2018 at 6:14 PM Dilip Kumar <[email protected]> wrote:
>
> On Fri, 30 Nov 2018, 20:42 Dmitry Dolgov <[email protected] wrote:
>>
>> > On Mon, Nov 5, 2018 at 1:32 PM Dilip Kumar <[email protected]> wrote:
>> >
>> > Updated patch, include defect fix from Kuntal posted on [1].
>> >
>> > [1] https://www.postgresql.org/message-id/CAGz5QCKpCG6egFAdazC%2BJgyk7YSE1OBN9h-QpwCkg-NnSWN5AQ%40mail.g...
>>
>> Thanks for working on this feature. Unfortunately, looks like the current
>> version of patch has some conflicts with the current master, could you please
>> post an updated version again?
>
>
> Currently, I am in process rebasing undolog patch set[1]. Post that I will rebase this patch on top of those patch set.
>
> [1] https://www.postgresql.org/message-id/CAFiTN-syRxU3jTpkOxHQsEqKC95LGd86JTdZ2stozyXWDUSffg%40mail.gma...
Updated patch. This will apply on top of undolog and undo interface
patch set[1].
[1] https://www.postgresql.org/message-id/CAFiTN-s112o6KLi_gVX%3Db-vO9t_2_ufxxRxG6Ff1iDx5Knvohw%40mail.g...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/x-patch] 0001-Undo-worker-and-transaction-rollback_v4.patch (134.5K, ../../CAFiTN-tKWfS2jAmyLrqtSb7VLA7FqrQLBFWadNnSrjV4-OS4VA@mail.gmail.com/2-0001-Undo-worker-and-transaction-rollback_v4.patch)
download | inline diff:
From 050dd60e9c8b1e2b9447d13ff9359b2a2e8efa8a Mon Sep 17 00:00:00 2001
From: Dilip Kumar <[email protected]>
Date: Tue, 4 Dec 2018 14:06:32 +0530
Subject: [PATCH] Undo worker and transaction rollback
Patch provides mechanism for discarding the older undo when they
are no longer relavent. It also providea a mechanism for rolling
back the effect of the aborted transactiona. It handles the rollbacks
directly from backends as well as from undo worker for the larger
transaction.
Dilip Kumar, with help from Rafia Sabih, Amit Kapila, Mithun.CY, Thomas Munro and Kuntal Ghosh
---
src/backend/access/rmgrdesc/Makefile | 3 +-
src/backend/access/rmgrdesc/undoactiondesc.c | 64 +++
src/backend/access/rmgrdesc/xlogdesc.c | 4 +-
src/backend/access/transam/rmgr.c | 5 +-
src/backend/access/transam/twophase.c | 45 +-
src/backend/access/transam/varsup.c | 12 +
src/backend/access/transam/xact.c | 441 ++++++++++++++++-
src/backend/access/transam/xlog.c | 20 +
src/backend/access/undo/Makefile | 2 +-
src/backend/access/undo/undoaction.c | 683 ++++++++++++++++++++++++++
src/backend/access/undo/undoactionxlog.c | 49 ++
src/backend/access/undo/undodiscard.c | 458 +++++++++++++++++
src/backend/access/undo/undoinsert.c | 50 +-
src/backend/access/undo/undorecord.c | 3 +
src/backend/commands/vacuum.c | 11 +
src/backend/postmaster/Makefile | 4 +-
src/backend/postmaster/bgworker.c | 11 +
src/backend/postmaster/discardworker.c | 170 +++++++
src/backend/postmaster/pgstat.c | 7 +-
src/backend/postmaster/postmaster.c | 7 +
src/backend/postmaster/undoworker.c | 664 +++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 4 +
src/backend/storage/ipc/ipci.c | 6 +
src/backend/storage/lmgr/lwlocknames.txt | 2 +
src/backend/storage/lmgr/proc.c | 2 +
src/backend/utils/adt/lockfuncs.c | 1 +
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc.c | 11 +
src/backend/utils/misc/pg_controldata.c | 9 +-
src/backend/utils/misc/postgresql.conf.sample | 7 +
src/bin/pg_controldata/pg_controldata.c | 2 +
src/bin/pg_resetwal/pg_resetwal.c | 7 +
src/bin/pg_rewind/parsexlog.c | 2 +-
src/bin/pg_waldump/rmgrdesc.c | 3 +-
src/include/access/rmgr.h | 2 +-
src/include/access/rmgrlist.h | 47 +-
src/include/access/transam.h | 4 +
src/include/access/twophase.h | 2 +-
src/include/access/undoaction.h | 28 ++
src/include/access/undoaction_xlog.h | 74 +++
src/include/access/undodiscard.h | 31 ++
src/include/access/undoinsert.h | 2 +
src/include/access/undorecord.h | 13 +
src/include/access/xact.h | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlog_internal.h | 9 +-
src/include/catalog/pg_control.h | 7 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +
src/include/postmaster/discardworker.h | 25 +
src/include/postmaster/undoloop.h | 89 ++++
src/include/postmaster/undoworker.h | 39 ++
src/include/storage/lock.h | 10 +
src/include/storage/proc.h | 2 +
54 files changed, 3117 insertions(+), 45 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/undoactiondesc.c
create mode 100644 src/backend/access/undo/undoaction.c
create mode 100644 src/backend/access/undo/undoactionxlog.c
create mode 100644 src/backend/access/undo/undodiscard.c
create mode 100644 src/backend/postmaster/discardworker.c
create mode 100644 src/backend/postmaster/undoworker.c
create mode 100644 src/include/access/undoaction.h
create mode 100644 src/include/access/undoaction_xlog.h
create mode 100644 src/include/access/undodiscard.h
create mode 100644 src/include/postmaster/discardworker.h
create mode 100644 src/include/postmaster/undoloop.h
create mode 100644 src/include/postmaster/undoworker.h
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index 91ad1ef..640d37f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = brindesc.o clogdesc.o committsdesc.o dbasedesc.o genericdesc.o \
gindesc.o gistdesc.o hashdesc.o heapdesc.o logicalmsgdesc.o \
mxactdesc.o nbtdesc.o relmapdesc.o replorigindesc.o seqdesc.o \
- smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undologdesc.o xactdesc.o xlogdesc.o
+ smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undoactiondesc.o \
+ undologdesc.o xactdesc.o xlogdesc.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/rmgrdesc/undoactiondesc.c b/src/backend/access/rmgrdesc/undoactiondesc.c
new file mode 100644
index 0000000..89343b8
--- /dev/null
+++ b/src/backend/access/rmgrdesc/undoactiondesc.c
@@ -0,0 +1,64 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactiondesc.c
+ * rmgr descriptor routines for access/undo/undoactionxlog.c
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/access/rmgrdesc/undoactiondesc.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+
+void
+undoaction_desc(StringInfo buf, XLogReaderState *record)
+{
+ char *rec = XLogRecGetData(record);
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ if (info == XLOG_UNDO_PAGE)
+ {
+ uint8 *flags = (uint8 *) rec;
+
+ appendStringInfo(buf, "page_contains_tpd_slot: %c ",
+ (*flags & XLU_PAGE_CONTAINS_TPD_SLOT) ? 'T' : 'F');
+ appendStringInfo(buf, "is_page_initialized: %c ",
+ (*flags & XLU_INIT_PAGE) ? 'T' : 'F');
+ if (*flags & XLU_PAGE_CONTAINS_TPD_SLOT)
+ {
+ xl_undoaction_page *xlrec =
+ (xl_undoaction_page *) ((char *) flags + sizeof(uint8));
+
+ appendStringInfo(buf, "urec_ptr %lu xid %u trans_slot_id %u",
+ xlrec->urec_ptr, xlrec->xid, xlrec->trans_slot_id);
+ }
+ }
+ else if (info == XLOG_UNDO_APPLY_PROGRESS)
+ {
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) rec;
+
+ appendStringInfo(buf, "urec_ptr %lu progress %u",
+ xlrec->urec_ptr, xlrec->progress);
+ }
+}
+
+const char *
+undoaction_identify(uint8 info)
+{
+ const char *id = NULL;
+
+ switch (info & ~XLR_INFO_MASK)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ id = "UNDO APPLY PROGRESS";
+ break;
+ }
+
+ return id;
+}
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 00741c7..987e39c 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -47,7 +47,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
"tli %u; prev tli %u; fpw %s; xid %u:%u; oid %u; multi %u; offset %u; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
- "oldest running xid %u; %s",
+ "oldest running xid %u; "
+ "oldest xid with epoch having undo " UINT64_FORMAT "; %s",
(uint32) (checkpoint->redo >> 32), (uint32) checkpoint->redo,
checkpoint->ThisTimeLineID,
checkpoint->PrevTimeLineID,
@@ -63,6 +64,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
checkpoint->oldestCommitTsXid,
checkpoint->newestCommitTsXid,
checkpoint->oldestActiveXid,
+ checkpoint->oldestXidWithEpochHavingUndo,
(info == XLOG_CHECKPOINT_SHUTDOWN) ? "shutdown" : "online");
}
else if (info == XLOG_NEXTOID)
diff --git a/src/backend/access/transam/rmgr.c b/src/backend/access/transam/rmgr.c
index 8b05374..6238240 100644
--- a/src/backend/access/transam/rmgr.c
+++ b/src/backend/access/transam/rmgr.c
@@ -18,6 +18,7 @@
#include "access/multixact.h"
#include "access/nbtxlog.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -31,8 +32,8 @@
#include "utils/relmapper.h"
/* must be kept in sync with RmgrData definition in xlog_internal.h */
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
- { name, redo, desc, identify, startup, cleanup, mask },
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
+ { name, redo, desc, identify, startup, cleanup, mask, undo, undo_desc },
const RmgrData RmgrTable[RM_MAX_ID + 1] = {
#include "access/rmgrlist.h"
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index e65dccc..2c640f9 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -93,6 +93,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/origin.h"
#include "replication/syncrep.h"
#include "replication/walsender.h"
@@ -915,6 +916,12 @@ typedef struct TwoPhaseFileHeader
uint16 gidlen; /* length of the GID - GID follows the header */
XLogRecPtr origin_lsn; /* lsn of this record at origin node */
TimestampTz origin_timestamp; /* time of prepare at origin node */
+ /*
+ * We need the locations of start and end undo record pointers when rollbacks
+ * are to be performed for prepared transactions using zheap relations.
+ */
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
} TwoPhaseFileHeader;
/*
@@ -989,7 +996,8 @@ save_state_data(const void *data, uint32 len)
* Initializes data structure and inserts the 2PC file header record.
*/
void
-StartPrepare(GlobalTransaction gxact)
+StartPrepare(GlobalTransaction gxact, UndoRecPtr *start_urec_ptr,
+ UndoRecPtr *end_urec_ptr)
{
PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
PGXACT *pgxact = &ProcGlobal->allPgXact[gxact->pgprocno];
@@ -1027,6 +1035,10 @@ StartPrepare(GlobalTransaction gxact)
&hdr.initfileinval);
hdr.gidlen = strlen(gxact->gid) + 1; /* Include '\0' */
+ /* save the start and end undo record pointers */
+ memcpy(hdr.start_urec_ptr, start_urec_ptr, sizeof(hdr.start_urec_ptr));
+ memcpy(hdr.end_urec_ptr, end_urec_ptr, sizeof(hdr.end_urec_ptr));
+
save_state_data(&hdr, sizeof(TwoPhaseFileHeader));
save_state_data(gxact->gid, hdr.gidlen);
@@ -1452,6 +1464,9 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
RelFileNode *delrels;
int ndelrels;
SharedInvalidationMessage *invalmsgs;
+ int i;
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
/*
* Validate the GID, and lock the GXACT to ensure that two backends do not
@@ -1489,6 +1504,34 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
invalmsgs = (SharedInvalidationMessage *) bufptr;
bufptr += MAXALIGN(hdr->ninvalmsgs * sizeof(SharedInvalidationMessage));
+ /* save the start and end undo record pointers */
+ memcpy(start_urec_ptr, hdr->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(end_urec_ptr, hdr->end_urec_ptr, sizeof(end_urec_ptr));
+
+ /*
+ * Perform undo actions, if there are undologs for this transaction.
+ * We need to perform undo actions while we are still in transaction.
+ * Never push rollbacks of temp tables to undo worker.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (end_urec_ptr[i] != InvalidUndoRecPtr && !isCommit)
+ {
+ bool result = false;
+ uint64 rollback_size = 0;
+
+ if (i != UNDO_TEMP)
+ rollback_size = end_urec_ptr[i] - start_urec_ptr[i];
+
+ if (rollback_size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(end_urec_ptr[i], start_urec_ptr[i], InvalidOid);
+
+ if (!result)
+ execute_undo_actions(end_urec_ptr[i], start_urec_ptr[i], true,
+ true, false);
+ }
+ }
+
/* compute latestXid among all children */
latestXid = TransactionIdLatest(xid, hdr->nsubxacts, children);
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index a5eb29e..31bea0e 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -284,10 +284,22 @@ SetTransactionIdLimit(TransactionId oldest_datfrozenxid, Oid oldest_datoid)
TransactionId xidStopLimit;
TransactionId xidWrapLimit;
TransactionId curXid;
+ TransactionId oldestXidHavingUndo;
Assert(TransactionIdIsNormal(oldest_datfrozenxid));
/*
+ * To determine the last safe xid that can be allocated, we need to
+ * consider oldestXidHavingUndo. The oldestXidHavingUndo will be only
+ * valid for zheap storage engine, so it won't impact any other storage
+ * engine.
+ */
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ oldest_datfrozenxid = Min(oldest_datfrozenxid, oldestXidHavingUndo);
+
+ /*
* The place where we actually get into deep trouble is halfway around
* from the oldest potentially-existing XID. (This calculation is
* probably off by one or two counts, because the special XIDs reduce the
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 6060013..66d1685 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -42,6 +42,7 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/origin.h"
@@ -280,6 +281,20 @@ typedef struct SubXactCallbackItem
static SubXactCallbackItem *SubXact_callbacks = NULL;
+/* Location in undo log from where to start applying the undo actions. */
+static UndoRecPtr UndoActionStartPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+/* Location in undo log up to which undo actions need to be applied. */
+static UndoRecPtr UndoActionEndPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+/* Do we need to perform any undo actions? */
+static bool PerformUndoActions = false;
/* local function prototypes */
static void AssignTransactionId(TransactionState s);
@@ -1826,6 +1841,7 @@ StartTransaction(void)
{
TransactionState s;
VirtualTransactionId vxid;
+ int i;
/*
* Let's just make sure the state stack is empty
@@ -1904,6 +1920,13 @@ StartTransaction(void)
nUnreportedXids = 0;
s->didLogXid = false;
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
/*
* must initialize resource-management stuff first
*/
@@ -2213,7 +2236,7 @@ CommitTransaction(void)
* NB: if you change this routine, better look at CommitTransaction too!
*/
static void
-PrepareTransaction(void)
+PrepareTransaction(UndoRecPtr *start_urec_ptr, UndoRecPtr *end_urec_ptr)
{
TransactionState s = CurrentTransactionState;
TransactionId xid = GetCurrentTransactionId();
@@ -2361,7 +2384,7 @@ PrepareTransaction(void)
* PREPARED; in particular, pay attention to whether things should happen
* before or after releasing the transaction's locks.
*/
- StartPrepare(gxact);
+ StartPrepare(gxact, start_urec_ptr, end_urec_ptr);
AtPrepare_Notify();
AtPrepare_Locks();
@@ -2484,6 +2507,65 @@ PrepareTransaction(void)
RESUME_INTERRUPTS();
}
+static void
+AtAbort_Rollback(void)
+{
+ TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i;
+
+ /* XXX: TODO: check this logic, which was moved out of UserAbortTransactionBlock */
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (CurrentTransactionState->state == TRANS_INPROGRESS)
+ {
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (latest_urec_ptr[i])
+ {
+ if (i == UNDO_TEMP)
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ false, true, true);
+ else
+ {
+ uint64 size = latest_urec_ptr[i] - s->start_urec_ptr[i];
+ bool result = false;
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions
+ * later.
+ */
+ if (size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ true, true, true);
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
+}
/*
* AbortTransaction
@@ -2585,6 +2667,7 @@ AbortTransaction(void)
*/
AfterTriggerEndXact(false); /* 'false' means it's abort */
AtAbort_Portals();
+ AtAbort_Rollback();
AtEOXact_LargeObject(false);
AtAbort_Notify();
AtEOXact_RelationMap(false, is_parallel_worker);
@@ -2794,6 +2877,12 @@ void
CommitTransactionCommand(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i;
+
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(end_urec_ptr, s->latest_urec_ptr, sizeof(end_urec_ptr));
switch (s->blockState)
{
@@ -2883,7 +2972,7 @@ CommitTransactionCommand(void)
* return to the idle state.
*/
case TBLOCK_PREPARE:
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, end_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
break;
@@ -2929,6 +3018,23 @@ CommitTransactionCommand(void)
{
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
+
+ /*
+ * Update the end undo record pointer if it's not valid with
+ * the currently popped transaction's end undo record pointer.
+ * This is particularly required when the first command of
+ * the transaction is of type which does not require an undo,
+ * e.g. savepoint x.
+ * Accordingly, update the start undo record pointer.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(end_urec_ptr[i]))
+ end_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(s->start_urec_ptr[i]))
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
} while (s->blockState == TBLOCK_SUBCOMMIT);
/* If we had a COMMIT command, finish off the main xact too */
if (s->blockState == TBLOCK_END)
@@ -2940,7 +3046,7 @@ CommitTransactionCommand(void)
else if (s->blockState == TBLOCK_PREPARE)
{
Assert(s->parent == NULL);
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, end_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
}
else
@@ -3034,6 +3140,18 @@ void
AbortCurrentTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
+
+ /*
+ * The undo actions are allowed to be executed at the end of statement
+ * execution when we are not in transaction block, otherwise they are
+ * executed when user explicitly ends the transaction.
+ *
+ * So if we are in a transaction block don't set the PerformUndoActions
+ * because this flag will be set when user explicitly issue rollback or
+ * rollback to savepoint.
+ */
+ PerformUndoActions = false;
switch (s->blockState)
{
@@ -3068,6 +3186,16 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * We are outside the transaction block so remember the required
+ * information to perform undo actions and also set the
+ * PerformUndoActions so that we execute it before completing this
+ * command.
+ */
+ PerformUndoActions = true;
+ memcpy (UndoActionStartPtr, s->latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
break;
/*
@@ -3104,6 +3232,9 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3123,6 +3254,9 @@ AbortCurrentTransaction(void)
case TBLOCK_ABORT_END:
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3133,6 +3267,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Failed while executing the rollback command, need perform any
+ * pending undo actions.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3144,6 +3284,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Perform any pending actions if failed while preparing the
+ * transaction.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3166,6 +3312,17 @@ AbortCurrentTransaction(void)
case TBLOCK_SUBCOMMIT:
case TBLOCK_SUBABORT_PENDING:
case TBLOCK_SUBRESTART:
+ /*
+ * If we are here and still UndoActionStartPtr is valid that means
+ * the subtransaction failed while executing the undo action, so
+ * store its undo action start point in parent so that parent can
+ * start its undo action from this point.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ s->parent->latest_urec_ptr[i] = UndoActionStartPtr[i];
+ }
AbortSubTransaction();
CleanupSubTransaction();
AbortCurrentTransaction();
@@ -3183,6 +3340,109 @@ AbortCurrentTransaction(void)
}
/*
+ * XactPerformUndoActionsIfPending - Execute pending undo actions.
+ *
+ * If the parent transaction state is valid (when there is an error in the
+ * subtransaction and rollback to savepoint is executed), then allow to
+ * perform undo actions in it, otherwise perform them in a new transaction.
+ */
+void
+XactPerformUndoActionsIfPending()
+{
+ TransactionState s = CurrentTransactionState;
+ uint64 rollback_size = 0;
+ bool new_xact = true, result = false, no_pending_action = true;
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ if (!PerformUndoActions)
+ return;
+
+ /* If there is no undo log for any persistence level, then return. */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ {
+ no_pending_action = false;
+ break;
+ }
+ }
+
+ if (no_pending_action)
+ {
+ PerformUndoActions = false;
+ return;
+ }
+
+ /*
+ * Execute undo actions under parent transaction, if any. Otherwise start
+ * a new transaction.
+ */
+ if (GetTopTransactionIdIfAny() != InvalidTransactionId)
+ {
+ memcpy(parent_latest_urec_ptr, s->latest_urec_ptr,
+ sizeof (parent_latest_urec_ptr));
+ new_xact = false;
+ }
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions later.
+ * Never push the rollbacks for temp tables.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ continue;
+
+ if (i == UNDO_TEMP)
+ goto perform_rollback;
+ else
+ rollback_size = UndoActionStartPtr[i] - UndoActionEndPtr[i];
+
+ if (new_xact && rollback_size > rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+perform_rollback:
+ if (new_xact)
+ {
+ TransactionState xact;
+
+ /* Start a new transaction for performing the rollback */
+ StartTransactionCommand();
+ xact = CurrentTransactionState;
+
+ /*
+ * Store the previous transactions start and end undo record
+ * pointers into this transaction's state so that if there is
+ * some error while performing undo actions we can restart
+ * from begining.
+ */
+ memcpy(xact->start_urec_ptr, UndoActionEndPtr,
+ sizeof(UndoActionEndPtr));
+ memcpy(xact->latest_urec_ptr, UndoActionStartPtr,
+ sizeof(UndoActionStartPtr));
+ }
+
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ new_xact, true, true);
+
+ if (new_xact)
+ CommitTransactionCommand();
+ else
+ {
+ /* Restore parent's state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+ }
+
+ PerformUndoActions = false;
+}
+
+/*
* PreventInTransactionBlock
*
* This routine is to be called by statements that must not run inside
@@ -3583,6 +3843,10 @@ EndTransactionBlock(void)
{
TransactionState s = CurrentTransactionState;
bool result = false;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i ;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
switch (s->blockState)
{
@@ -3628,6 +3892,16 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+
+ /*
+ * We are calculating latest_urec_ptr, even though its a commit
+ * case. This is to handle any error during the commit path.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_END;
@@ -3653,6 +3927,11 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if(!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_ABORT_PENDING;
@@ -3705,6 +3984,18 @@ EndTransactionBlock(void)
break;
}
+ /*
+ * We need to perform undo actions if the transaction is failed. Remember
+ * the required information to perform undo actions at the end of
+ * statement execution.
+ */
+ if (!result)
+ PerformUndoActions = true;
+
+ memcpy(UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy(UndoActionEndPtr, TopTransactionStateData.start_urec_ptr,
+ sizeof(UndoActionEndPtr));
+
return result;
}
@@ -3962,6 +4253,12 @@ ReleaseSavepoint(const char *name)
TransactionState s = CurrentTransactionState;
TransactionState target,
xact;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4055,8 +4352,32 @@ ReleaseSavepoint(const char *name)
if (xact == target)
break;
xact = xact->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = xact->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(xact->start_urec_ptr[i]))
+ start_urec_ptr[i] = xact->start_urec_ptr[i];
+ }
+
Assert(PointerIsValid(xact));
}
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction. Also set the start_urec_ptr if
+ * parent start_urec_ptr is not valid.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ xact->parent->latest_urec_ptr[i] = latest_urec_ptr[i];
+ if (!UndoRecPtrIsValid(xact->parent->start_urec_ptr[i]))
+ xact->parent->start_urec_ptr[i] = start_urec_ptr[i];
+ }
}
/*
@@ -4267,6 +4588,7 @@ void
ReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4285,6 +4607,22 @@ ReleaseCurrentSubTransaction(void)
BlockStateAsString(s->blockState));
Assert(s->state == TRANS_INPROGRESS);
MemoryContextSwitchTo(CurTransactionContext);
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ s->parent->latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (!UndoRecPtrIsValid(s->parent->start_urec_ptr[i]))
+ s->parent->start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
Assert(s->state == TRANS_INPROGRESS);
@@ -4301,6 +4639,14 @@ void
RollbackAndReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i;
/*
* Unlike ReleaseCurrentSubTransaction(), this is nominally permitted
@@ -4347,6 +4693,19 @@ RollbackAndReleaseCurrentSubTransaction(void)
if (s->blockState == TBLOCK_SUBINPROGRESS)
AbortSubTransaction();
+ /*
+ * Remember the required information to perform undo actions before
+ * cleaning up the subtransaction state.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ {
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+ }
+
/* And clean it up, too */
CleanupSubTransaction();
@@ -4355,6 +4714,30 @@ RollbackAndReleaseCurrentSubTransaction(void)
s->blockState == TBLOCK_INPROGRESS ||
s->blockState == TBLOCK_IMPLICIT_INPROGRESS ||
s->blockState == TBLOCK_STARTED);
+
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ parent_latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ /*
+ * Store the undo action start point in the parent state so that
+ * we can apply undo actions these undos also during rollback of
+ * parent transaction in case of error while applying the undo
+ * actions.
+ */
+ s->latest_urec_ptr[i] = latest_urec_ptr[i];
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false,
+ true, true);
+
+ /* Restore parent state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+
+ /* Successfully performed undo actions so reset the flag. */
+ PerformUndoActions = false;
}
/*
@@ -4568,6 +4951,7 @@ static void
StartSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
if (s->state != TRANS_DEFAULT)
elog(WARNING, "StartSubTransaction while in %s state",
@@ -4585,6 +4969,13 @@ StartSubTransaction(void)
AtSubStart_Notify();
AfterTriggerBeginSubXact();
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
s->state = TRANS_INPROGRESS;
/*
@@ -4708,6 +5099,47 @@ CommitSubTransaction(void)
PopTransaction();
}
+static void
+AtSubAbort_Rollback(TransactionState s)
+{
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ /* XXX: TODO: Check this logic, which was moved out of RollbackToSavepoint() */
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (s->state == TRANS_INPROGRESS)
+ {
+ for ( i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false, true, false);
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
+}
+
/*
* AbortSubTransaction
*/
@@ -4802,6 +5234,7 @@ AbortSubTransaction(void)
s->parent->subTransactionId,
s->curTransactionOwner,
s->parent->curTransactionOwner);
+ AtSubAbort_Rollback(s);
AtEOSubXact_LargeObject(false, s->subTransactionId,
s->parent->subTransactionId);
AtSubAbort_Notify();
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 01815a6..e575322 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -5170,6 +5170,7 @@ BootStrapXLOG(void)
checkPoint.newestCommitTsXid = InvalidTransactionId;
checkPoint.time = (pg_time_t) time(NULL);
checkPoint.oldestActiveXid = InvalidTransactionId;
+ checkPoint.oldestXidWithEpochHavingUndo = InvalidTransactionId;
ShmemVariableCache->nextXid = checkPoint.nextXid;
ShmemVariableCache->nextOid = checkPoint.nextOid;
@@ -6604,6 +6605,10 @@ StartupXLOG(void)
(errmsg_internal("commit timestamp Xid oldest/newest: %u/%u",
checkPoint.oldestCommitTsXid,
checkPoint.newestCommitTsXid)));
+ ereport(DEBUG1,
+ (errmsg_internal("oldest xid with epoch having undo: " UINT64_FORMAT,
+ checkPoint.oldestXidWithEpochHavingUndo)));
+
if (!TransactionIdIsNormal(checkPoint.nextXid))
ereport(PANIC,
(errmsg("invalid next transaction ID")));
@@ -6621,6 +6626,10 @@ StartupXLOG(void)
XLogCtl->ckptXidEpoch = checkPoint.nextXidEpoch;
XLogCtl->ckptXid = checkPoint.nextXid;
+ /* Read oldest xid having undo from checkpoint and set in proc global. */
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* Initialize replication slots, before there's a chance to remove
* required resources.
@@ -8792,6 +8801,9 @@ CreateCheckPoint(int flags)
checkPoint.nextOid += ShmemVariableCache->oidCount;
LWLockRelease(OidGenLock);
+ checkPoint.oldestXidWithEpochHavingUndo =
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo);
+
MultiXactGetCheckptMulti(shutdown,
&checkPoint.nextMulti,
&checkPoint.nextMultiOffset,
@@ -9702,6 +9714,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceOldest(checkPoint.oldestMulti,
checkPoint.oldestMultiDB);
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* No need to set oldestClogXid here as well; it'll be set when we
* redo an xl_clog_truncate if it changed since initialization.
@@ -9760,6 +9775,8 @@ xlog_redo(XLogReaderState *record)
/* ControlFile->checkPointCopy always tracks the latest ckpt XID */
ControlFile->checkPointCopy.nextXidEpoch = checkPoint.nextXidEpoch;
ControlFile->checkPointCopy.nextXid = checkPoint.nextXid;
+ ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo =
+ checkPoint.oldestXidWithEpochHavingUndo;
/* Update shared-memory copy of checkpoint XID/epoch */
SpinLockAcquire(&XLogCtl->info_lck);
@@ -9809,6 +9826,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceNextMXact(checkPoint.nextMulti,
checkPoint.nextMultiOffset);
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* NB: This may perform multixact truncation when replaying WAL
* generated by an older primary.
diff --git a/src/backend/access/undo/Makefile b/src/backend/access/undo/Makefile
index f41e8f7..fdf7f7d 100644
--- a/src/backend/access/undo/Makefile
+++ b/src/backend/access/undo/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/access/undo
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = undoinsert.o undolog.o undorecord.o
+OBJS = undoaction.o undoactionxlog.o undodiscard.o undoinsert.o undolog.o undorecord.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/undo/undoaction.c b/src/backend/access/undo/undoaction.c
new file mode 100644
index 0000000..562630e
--- /dev/null
+++ b/src/backend/access/undo/undoaction.c
@@ -0,0 +1,683 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.c
+ * execute undo actions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undoaction.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/undoaction.h"
+#include "access/undoaction_xlog.h"
+#include "access/undolog.h"
+#include "access/undorecord.h"
+#include "access/visibilitymap.h"
+#include "access/xact.h"
+#include "access/xloginsert.h"
+#include "access/xlog_internal.h"
+#include "nodes/pg_list.h"
+#include "pgstat.h"
+#include "postmaster/undoloop.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "utils/rel.h"
+#include "utils/relfilenodemap.h"
+#include "miscadmin.h"
+#include "storage/shmem.h"
+#include "access/undodiscard.h"
+
+#define ROLLBACK_HT_SIZE 1024
+
+static bool execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr,
+ Oid reloid, TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool norellock, int options);
+static void RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr);
+
+/* This is the hash table to store all the rollabck requests. */
+static HTAB *RollbackHT;
+
+/*
+ * execute_undo_actions - Execute the undo actions
+ *
+ * from_urecptr - undo record pointer from where to start applying undo action.
+ * to_urecptr - undo record pointer upto which point apply undo action.
+ * nopartial - true if rollback is for complete transaction.
+ * rewind - whether to rewind the insert location of the undo log or not.
+ * Only the backend executed the transaction can rewind, but
+ * any other process e.g. undo worker should not rewind it.
+ * Because, if the backend have already inserted new undo records
+ * for the next transaction and if we rewind then we will loose
+ * the undo record inserted for the new transaction.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ */
+void
+execute_undo_actions(UndoRecPtr from_urecptr, UndoRecPtr to_urecptr,
+ bool nopartial, bool rewind, bool rellock)
+{
+ UnpackedUndoRecord *uur = NULL;
+ UndoRecPtr urec_ptr, prev_urec_ptr;
+ UndoRecPtr save_urec_ptr;
+ Oid prev_reloid = InvalidOid;
+ ForkNumber prev_fork = InvalidForkNumber;
+ BlockNumber prev_block = InvalidBlockNumber;
+ List *luinfo = NIL;
+ bool more_undo;
+ int options = 0;
+ TransactionId xid = InvalidTransactionId;
+ UndoRecInfo *urec_info;
+
+ Assert(from_urecptr != InvalidUndoRecPtr);
+ Assert(UndoRecPtrGetLogNo(from_urecptr) != UndoRecPtrGetLogNo(to_urecptr) ||
+ from_urecptr >= to_urecptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ * FIXME: this won't work if undolog crossed the limit of 1TB, because
+ * then from_urecptr and to_urecptr will be from different lognos.
+ */
+ if (to_urecptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(from_urecptr);
+ to_urecptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ save_urec_ptr = urec_ptr = from_urecptr;
+
+ if (nopartial)
+ {
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber, InvalidOffsetNumber,
+ InvalidTransactionId, NULL, NULL);
+ if (uur == NULL)
+ return;
+
+ xid = uur->uur_xid;
+ UndoRecordRelease(uur);
+ uur = NULL;
+
+ /*
+ * Grab the undo action apply lock before start applying the undo action
+ * this will prevent applying undo actions concurrently. If we do not
+ * get the lock that mean its already being applied concurrently or the
+ * discard worker might be pushing its request to the rollback hash
+ * table
+ */
+ if (!ConditionTransactionUndoActionLock(xid))
+ return;
+ }
+
+ prev_urec_ptr = InvalidUndoRecPtr;
+ while (prev_urec_ptr != to_urecptr)
+ {
+ Oid reloid = InvalidOid;
+ uint16 urec_prevlen;
+ bool non_page;
+
+ more_undo = true;
+
+ prev_urec_ptr = urec_ptr;
+
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId, NULL, NULL);
+
+ /* If there is no info block, this is not a page-based undo record. */
+ non_page = uur && !(uur->uur_info & UREC_INFO_BLOCK);
+
+ if (uur != NULL && !non_page)
+ reloid = uur->uur_reloid;
+
+ xid = uur->uur_xid;
+
+ if (non_page)
+ {
+ prev_reloid = InvalidOid;
+ urec_prevlen = uur->uur_prevlen;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /*
+ * Execute individual undo actions not associated with a page
+ * immediately.
+ */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->uur = uur;
+ urec_info->urp = urec_ptr;
+ luinfo = lappend(luinfo, urec_info);
+ execute_undo_actions_page(luinfo, urec_ptr, reloid, xid,
+ InvalidBlockNumber, false, rellock,
+ 0);
+ pfree(urec_info);
+ urec_info = NULL;
+ list_free(luinfo);
+ luinfo = NIL;
+ UndoRecordRelease(uur);
+
+ /* Follow undo chain until to_urecptr. */
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ /*
+ * If the record is already discarded by undo worker or if the relation
+ * is dropped or truncated, then we cannot fetch record successfully.
+ * Hence, skip quietly.
+ *
+ * Note: reloid remains InvalidOid for a discarded record.
+ */
+ else if (!OidIsValid(reloid))
+ {
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ urec_prevlen = uur->uur_prevlen;
+
+ /* Release the just-fetched record */
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else if (!OidIsValid(prev_reloid) ||
+ (prev_reloid == reloid &&
+ prev_fork == uur->uur_fork &&
+ prev_block == uur->uur_block))
+ {
+ /* Collect the undo records that belong to the same page. */
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+
+ luinfo = lappend(luinfo, urec_info);
+ urec_prevlen = uur->uur_prevlen;
+ save_urec_ptr = uur->uur_blkprev;
+
+ if (uur->uur_info & UREC_INFO_PAYLOAD_CONTAINS_SLOT)
+ options |= UNDO_ACTION_UPDATE_TPD;
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else
+ {
+ more_undo = true;
+ }
+
+ /*
+ * If no more undo is left to be processed and we are rolling back the
+ * complete transaction, then we can consider that the undo chain for a
+ * block is complete.
+ * If the previous undo pointer in the page is invalid, then also the
+ * undo chain for the current block is completed.
+ */
+ if (luinfo &&
+ ((!more_undo && nopartial) || !UndoRecPtrIsValid(save_urec_ptr)))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, true, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+ else if (luinfo)
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, false, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ /*
+ * There are still more records to process, so keep moving backwards
+ * in the chain.
+ */
+ if (more_undo)
+ {
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+ luinfo = lappend(luinfo, urec_info);
+
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+ save_urec_ptr = uur->uur_blkprev;
+
+ if (uur->uur_info & UREC_INFO_PAYLOAD_CONTAINS_SLOT)
+ options |= UNDO_ACTION_UPDATE_TPD;
+
+ /*
+ * Continue to process the records if this is not the last undo
+ * record in chain.
+ */
+ urec_prevlen = uur->uur_prevlen;
+ if (urec_prevlen > 0 && urec_ptr != to_urecptr)
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen);
+ else
+ break;
+ }
+ else
+ break;
+ }
+
+ /* Apply the undo actions for the remaining records. */
+ if (list_length(luinfo))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, nopartial ? true : false,
+ rellock, options);
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+ }
+
+ if (rewind)
+ {
+ /* Read the prevlen from the first record of this transaction. */
+ uur = UndoFetchRecord(to_urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If undo is already discarded before we rewind, then do nothing.
+ */
+ if (uur == NULL)
+ return;
+
+
+ /*
+ * Rewind the insert location to start of this transaction. This is
+ * to avoid reapplying some intermediate undo. We do not need to wal
+ * log this information here, because if the system crash before we
+ * rewind the insert pointer then after recovery we can identify
+ * whether the undo is already applied or not from the slot undo record
+ * pointer. Also set the correct prevlen value (what we have fetched
+ * from the undo).
+ */
+ UndoLogRewind(to_urecptr, uur->uur_prevlen);
+
+ UndoRecordRelease(uur);
+ }
+
+ if (nopartial)
+ {
+ /*
+ * Set undo action apply completed in the transaction header if this is
+ * a main transaction and we have not rewound its undo.
+ */
+ if (!rewind)
+ {
+ /*
+ * Undo action is applied so delete the hash table entry and release
+ * the undo action lock.
+ */
+ RollbackHTRemoveEntry(from_urecptr);
+
+ /*
+ * Prepare and update the progress of the undo action apply in the
+ * transaction header.
+ */
+ PrepareUpdateUndoActionProgress(to_urecptr, 1);
+
+ START_CRIT_SECTION();
+
+ /* Update the progress in the transaction header. */
+ UndoRecordUpdateTransInfo();
+
+ /* WAL log the undo apply progress. */
+ {
+ xl_undoapply_progress xlrec;
+
+ xlrec.urec_ptr = to_urecptr;
+ xlrec.progress = 1;
+
+ /*
+ * FIXME : We need to register undo buffers and set LSN for them
+ * that will be required for FPW of the undo buffers.
+ */
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, sizeof(xlrec));
+
+ (void) XLogInsert(RM_UNDOACTION_ID, XLOG_UNDO_APPLY_PROGRESS);
+ }
+
+ END_CRIT_SECTION();
+ UnlockReleaseUndoBuffers();
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+}
+
+/*
+ * execute_undo_actions_page - Execute the undo actions for a page
+ *
+ * After applying all the undo actions for a page, we clear the transaction
+ * slot on a page if the undo chain for block is complete, otherwise just
+ * rewind the undo pointer to the last record for that block that precedes
+ * the last undo record for which action is replayed.
+ *
+ * luinfo - list of undo records (along with their location) for which undo
+ * action needs to be replayed.
+ * urec_ptr - undo record pointer to which we need to rewind.
+ * reloid - OID of relation on which undo actions needs to be applied.
+ * blkno - block number on which undo actions needs to be applied.
+ * blk_chain_complete - indicates whether the undo chain for block is
+ * complete.
+ * nopartial - true if rollback is for complete transaction. If we are not
+ * rolling back the complete transaction then we need to apply the
+ * undo action for UNDO_INVALID_XACT_SLOT also because in such
+ * case we will rewind the insert undo location.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ * options - options for executing undo actions.
+ *
+ * returns true, if successfully applied the undo actions, otherwise, false.
+ */
+static bool
+execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock, int options)
+{
+ UndoRecInfo *first;
+
+ /*
+ * All records passed to us are for the same RMGR, so we just use the
+ * first record to dispatch.
+ */
+ Assert(luinfo != NIL);
+ first = (UndoRecInfo *) linitial(luinfo);
+
+ return RmgrTable[first->uur->uur_rmid].rm_undo(luinfo, urec_ptr, reloid,
+ xid, blkno,
+ blk_chain_complete, rellock,
+ options);
+}
+
+/*
+ * To return the size of the hash-table for rollbacks.
+ */
+int
+RollbackHTSize(void)
+{
+ return hash_estimate_size(ROLLBACK_HT_SIZE, sizeof(RollbackHashEntry));
+}
+
+/*
+ * To initialize the hash-table for rollbacks in shared memory
+ * for the given size.
+ */
+void
+InitRollbackHashTable(void)
+{
+ HASHCTL info;
+ MemSet(&info, 0, sizeof(info));
+
+ info.keysize = sizeof(UndoRecPtr);
+ info.entrysize = sizeof(RollbackHashEntry);
+ info.hash = tag_hash;
+
+ RollbackHT = ShmemInitHash("Undo actions Lookup Table",
+ ROLLBACK_HT_SIZE, ROLLBACK_HT_SIZE, &info,
+ HASH_ELEM | HASH_FUNCTION | HASH_FIXED_SIZE);
+}
+
+/*
+ * To push the rollback requests from backend to the hash-table.
+ * Return true if the request is successfully added, else false
+ * and the caller may execute undo actions itself.
+ */
+bool
+PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr, Oid dbid)
+{
+ bool found = false;
+ RollbackHashEntry *rh;
+
+ Assert(UndoRecPtrGetLogNo(start_urec_ptr) != UndoRecPtrGetLogNo(end_urec_ptr) ||
+ start_urec_ptr >= end_urec_ptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ */
+ if (start_urec_ptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(end_urec_ptr);
+ start_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ Assert(UndoRecPtrIsValid(start_urec_ptr));
+
+ /* If there is no space to accomodate new request, then we can't proceed. */
+ if (RollbackHTIsFull())
+ return false;
+
+ if(!UndoRecPtrIsValid(end_urec_ptr))
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(start_urec_ptr);
+ end_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ rh = (RollbackHashEntry *) hash_search(RollbackHT, &start_urec_ptr,
+ HASH_ENTER_NULL, &found);
+ if (!rh)
+ {
+ LWLockRelease(RollbackHTLock);
+ return false;
+ }
+ /* We shouldn't try to push the same rollback request again. */
+ if (!found)
+ {
+ rh->start_urec_ptr = start_urec_ptr;
+ rh->end_urec_ptr = end_urec_ptr;
+ rh->dbid = (dbid == InvalidOid) ? MyDatabaseId : dbid;
+ }
+ LWLockRelease(RollbackHTLock);
+
+ return true;
+}
+
+/*
+ * To perform the undo actions for the transactions whose rollback
+ * requests are in hash table. Sequentially, scan the hash-table
+ * and perform the undo-actions for the respective transactions.
+ * Once, the undo-actions are applied, remove the entry from the
+ * hash table.
+ */
+void
+RollbackFromHT(Oid dbid)
+{
+ UndoRecPtr start[ROLLBACK_HT_SIZE];
+ UndoRecPtr end[ROLLBACK_HT_SIZE];
+ RollbackHashEntry *rh;
+ HASH_SEQ_STATUS status;
+ int i = 0;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ Assert(hash_get_num_entries(RollbackHT) <= ROLLBACK_HT_SIZE);
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ {
+ if (rh->dbid == dbid)
+ {
+ start[i] = rh->start_urec_ptr;
+ end[i] = rh->end_urec_ptr;
+ i++;
+ }
+ }
+
+ LWLockRelease(RollbackHTLock);
+
+ /* Execute the rollback requests */
+ while(--i >= 0)
+ {
+ Assert(UndoRecPtrIsValid(start[i]));
+ Assert(UndoRecPtrIsValid(end[i]));
+
+ StartTransactionCommand();
+ execute_undo_actions(start[i], end[i], true, false, true);
+ CommitTransactionCommand();
+ }
+}
+
+/*
+ * Remove the rollback request entry from the rollback hash table.
+ */
+static void
+RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr)
+{
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ hash_search(RollbackHT, &start_urec_ptr, HASH_REMOVE, NULL);
+
+ LWLockRelease(RollbackHTLock);
+}
+
+/*
+ * To check if the rollback requests in the hash table are all
+ * completed or not. This is required because we don't not want to
+ * expose RollbackHT in xact.c, where it is required to ensure
+ * that we push the resuests only when there is some space in
+ * the hash-table.
+ */
+bool
+RollbackHTIsFull(void)
+{
+ bool result = false;
+
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ if (hash_get_num_entries(RollbackHT) >= ROLLBACK_HT_SIZE)
+ result = true;
+
+ LWLockRelease(RollbackHTLock);
+
+ return result;
+}
+
+/*
+ * Get database list from the rollback hash table.
+ */
+List *
+RollbackHTGetDBList()
+{
+ HASH_SEQ_STATUS status;
+ RollbackHashEntry *rh;
+ List *dblist = NIL;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ dblist = list_append_unique_oid(dblist, rh->dbid);
+
+ LWLockRelease(RollbackHTLock);
+
+ return dblist;
+}
+
+/*
+ * ConditionTransactionUndoActionLock
+ *
+ * Insert a lock showing that the undo action for given transaction is in
+ * progress. This is only done for the main transaction not for the
+ * sub-transaction.
+ */
+bool
+ConditionTransactionUndoActionLock(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ if (LOCKACQUIRE_NOT_AVAIL == LockAcquire(&tag, ExclusiveLock, false, true))
+ return false;
+ else
+ return true;
+}
+
+/*
+ * TransactionUndoActionLockRelease
+ *
+ * Delete the lock showing that the undo action given transaction ID is in
+ * progress.
+ */
+void
+TransactionUndoActionLockRelease(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ LockRelease(&tag, ExclusiveLock, false);
+}
diff --git a/src/backend/access/undo/undoactionxlog.c b/src/backend/access/undo/undoactionxlog.c
new file mode 100644
index 0000000..546f43d
--- /dev/null
+++ b/src/backend/access/undo/undoactionxlog.c
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactionxlog.c
+ * WAL replay logic for undo actions.
+ *
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/undo/undoactionxlog.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+#include "access/undoinsert.h"
+#include "access/visibilitymap.h"
+#include "access/xlog.h"
+#include "access/xlogutils.h"
+
+/*
+ * Replay of undo apply progress.
+ */
+static void
+undo_xlog_apply_progress(XLogReaderState *record)
+{
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) XLogRecGetData(record);
+
+ /* Update the progress in the transaction header. */
+ PrepareUpdateUndoActionProgress(xlrec->urec_ptr, xlrec->progress);
+ UndoRecordUpdateTransInfo();
+}
+
+void
+undoaction_redo(XLogReaderState *record)
+{
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ switch (info)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ undo_xlog_apply_progress(record);
+ break;
+ default:
+ elog(PANIC, "undoaction_redo: unknown op code %u", info);
+ }
+}
diff --git a/src/backend/access/undo/undodiscard.c b/src/backend/access/undo/undodiscard.c
new file mode 100644
index 0000000..9d55588
--- /dev/null
+++ b/src/backend/access/undo/undodiscard.c
@@ -0,0 +1,458 @@
+/*-------------------------------------------------------------------------
+ *
+ * undodiscard.c
+ * discard undo records
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undodiscard.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/xlog.h"
+#include "access/xact.h"
+#include "access/undolog.h"
+#include "access/undodiscard.h"
+#include "catalog/pg_tablespace.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "storage/shmem.h"
+#include "storage/proc.h"
+#include "utils/resowner.h"
+#include "postmaster/undoloop.h"
+
+static UndoRecPtr FetchLatestUndoPtrForXid(UndoRecPtr urecptr,
+ UnpackedUndoRecord *uur_start,
+ UndoLogControl *log);
+
+/*
+ * Discard the undo for the log
+ *
+ * Search the undo log, get the start record for each transaction until we get
+ * the transaction with xid >= xmin or an invalid xid. Then call undolog
+ * routine to discard upto that point and update the memory structure for the
+ * log slot. We set the hibernate flag if we do not have any undo logs, this
+ * flag is passed to the undo worker wherein it determines if system is idle
+ * and it should sleep for sometime.
+ *
+ * Return the oldest xid remaining in this undo log (which should be >= xmin,
+ * since we'll discard everything older). Return InvalidTransactionId if the
+ * undo log is empty.
+ */
+static TransactionId
+UndoDiscardOneLog(UndoLogControl *log, TransactionId xmin, bool *hibernate)
+{
+ UndoRecPtr undo_recptr, next_insert, from_urecptr;
+ UndoRecPtr next_urecptr = InvalidUndoRecPtr;
+ UnpackedUndoRecord *uur = NULL;
+ bool need_discard = false;
+ bool log_complete = false;
+ TransactionId undoxid = InvalidTransactionId;
+ TransactionId xid = log->oldest_xid;
+ TransactionId latest_discardxid = InvalidTransactionId;
+ uint32 epoch = 0;
+
+ undo_recptr = log->oldest_data;
+
+ /* There might not be any undo log and hibernation might be needed. */
+ *hibernate = true;
+
+ /* Loop until we run out of discardable transactions. */
+ do
+ {
+ bool pending_abort = false;
+
+ next_insert = UndoLogGetNextInsertPtr(log->logno, xid);
+
+ /*
+ * If the next insert location in the undo log is same as the oldest
+ * data for the log then there is nothing more to discard in this log
+ * so discard upto this point.
+ */
+ if (next_insert == undo_recptr)
+ {
+ /*
+ * If the discard location and the insert location is same then
+ * there is nothing to discard.
+ */
+ if (undo_recptr == log->oldest_data)
+ break;
+ else
+ log_complete = true;
+ }
+ else
+ {
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+
+ Assert(uur != NULL);
+
+ if (!TransactionIdDidCommit(uur->uur_xid) &&
+ TransactionIdPrecedes(uur->uur_xid, xmin) &&
+ uur->uur_progress == 0)
+ {
+ /*
+ * At the time of recovery, we might not have a valid next undo
+ * record pointer and in that case we'll calculate the location
+ * of from pointer using the last record of next insert
+ * location.
+ */
+ if (ConditionTransactionUndoActionLock(uur->uur_xid))
+ {
+ TransactionId xid = uur->uur_xid;
+ UndoLogControl *log = NULL;
+ UndoLogNumber logno;
+
+ logno = UndoRecPtrGetLogNo(undo_recptr);
+ log = UndoLogGet(logno, false);
+
+ /*
+ * If the corresponding log got rewinded to a location
+ * prior to undo_recptr, the undo actions are already
+ * applied.
+ */
+ if (log->meta.insert > undo_recptr)
+ {
+ UndoRecordRelease(uur);
+
+ /* Fetch the undo record under undo action lock. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the undo actions for the aborted transaction is
+ * already applied then continue discarding the undo log
+ * otherwise discard till current point and stop processing
+ * this undo log.
+ * Also, check this is indeed the transaction id we're
+ * looking for. It is possible that after rewinding
+ * some other transaction has inserted an undo record.
+ */
+ if (uur->uur_xid == xid && uur->uur_progress == 0)
+ {
+ from_urecptr = FetchLatestUndoPtrForXid(undo_recptr, uur, log);
+ (void)PushRollbackReq(from_urecptr, undo_recptr, uur->uur_dbid);
+ pending_abort = true;
+ }
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+ else
+ pending_abort = true;
+ }
+
+ next_urecptr = uur->uur_next;
+ undoxid = uur->uur_xid;
+ xid = undoxid;
+ epoch = uur->uur_xidepoch;
+ }
+
+ /* we can discard upto this point. */
+ if (TransactionIdFollowsOrEquals(undoxid, xmin) ||
+ next_urecptr == SpecialUndoRecPtr ||
+ UndoRecPtrGetLogNo(next_urecptr) != log->logno ||
+ log_complete || pending_abort)
+ {
+ /* Hey, I got some undo log to discard, can not hibernate now. */
+ *hibernate = false;
+
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /*
+ * If Transaction id is smaller than the xmin that means this must
+ * be the last transaction in this undo log, so we need to get the
+ * last insert point in this undo log and discard till that point.
+ * Also, if the transaction has pending abort, we stop discarding
+ * undo from the same location.
+ */
+ if (TransactionIdPrecedes(undoxid, xmin) && !pending_abort)
+ {
+ UndoRecPtr next_insert = InvalidUndoRecPtr;
+
+ /*
+ * Get the last insert location for this transaction Id, if it
+ * returns invalid pointer that means there is new transaction
+ * has started for this undolog. So we need to refetch the undo
+ * and continue the process.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, undoxid);
+ if (!UndoRecPtrIsValid(next_insert))
+ continue;
+
+ undo_recptr = next_insert;
+ need_discard = true;
+ epoch = 0;
+ latest_discardxid = undoxid;
+ undoxid = InvalidTransactionId;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_EXCLUSIVE);
+
+ /*
+ * If no more pending undo logs then set the oldest transaction to
+ * InvalidTransactionId.
+ */
+ if (log_complete)
+ {
+ log->oldest_xid = InvalidTransactionId;
+ log->oldest_xidepoch = 0;
+ }
+ else
+ {
+ log->oldest_xid = undoxid;
+ log->oldest_xidepoch = epoch;
+ }
+
+ log->oldest_data = undo_recptr;
+ LWLockRelease(&log->discard_lock);
+
+ if (need_discard)
+ UndoLogDiscard(undo_recptr, latest_discardxid);
+
+ break;
+ }
+
+ /*
+ * This transaction is smaller than the xmin so lets jump to the next
+ * transaction.
+ */
+ undo_recptr = next_urecptr;
+ latest_discardxid = undoxid;
+
+ if(uur != NULL)
+ {
+ UndoRecordRelease(uur);
+ uur = NULL;
+ }
+
+ need_discard = true;
+ } while (true);
+
+ return undoxid;
+}
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+void
+UndoDiscard(TransactionId oldestXmin, bool *hibernate)
+{
+ TransactionId oldestXidHavingUndo = oldestXmin;
+ uint64 epoch = GetEpochForXid(oldestXmin);
+ UndoLogControl *log = NULL;
+
+ /*
+ * TODO: Ideally we'd arrange undo logs so that we can efficiently find
+ * those with oldest_xid < oldestXmin, but for now we'll just scan all of
+ * them.
+ */
+ while ((log = UndoLogNext(log)))
+ {
+ TransactionId oldest_xid = InvalidTransactionId;
+
+ /* We can't process temporary undo logs. */
+ if (log->meta.persistence == UNDO_TEMP)
+ continue;
+
+ /*
+ * If the first xid of the undo log is smaller than the xmin the try
+ * to discard the undo log.
+ */
+ if (TransactionIdPrecedes(log->oldest_xid, oldestXmin))
+ {
+ /*
+ * If the XID in the discard entry is invalid then start scanning
+ * from the first valid undorecord in the log.
+ */
+ if (!TransactionIdIsValid(log->oldest_xid))
+ {
+ bool full = false;
+ UndoRecPtr urp = UndoLogGetFirstValidRecord(log, &full);
+
+ if (!UndoRecPtrIsValid(urp))
+ {
+ /*
+ * There is nothing to be discarded. If there is also no
+ * more free space, then a call to UndoLogDiscard() will
+ * discard it the undo log completely and free up the
+ * UndoLogControl slot.
+ */
+ if (full)
+ UndoLogDiscard(MakeUndoRecPtr(log->meta.logno,
+ log->meta.discard),
+ InvalidTransactionId);
+ continue;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_SHARED);
+ log->oldest_data = urp;
+ LWLockRelease(&log->discard_lock);
+ }
+
+ /* Process the undo log. */
+ oldest_xid = UndoDiscardOneLog(log, oldestXmin, hibernate);
+ }
+
+ if (TransactionIdIsValid(oldest_xid) &&
+ TransactionIdPrecedes(oldest_xid, oldestXidHavingUndo))
+ {
+ oldestXidHavingUndo = oldest_xid;
+ epoch = GetEpochForXid(oldest_xid);
+ }
+ }
+
+ /*
+ * Update the oldestXidWithEpochHavingUndo in the shared memory.
+ *
+ * XXX In future if multiple worker can perform discard then we may need
+ * to use compare and swap for updating the shared memory value.
+ */
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ MakeEpochXid(epoch, oldestXidHavingUndo));
+}
+
+/*
+ * Fetch the latest urec pointer for the transaction.
+ */
+UndoRecPtr
+FetchLatestUndoPtrForXid(UndoRecPtr urecptr, UnpackedUndoRecord *uur_start,
+ UndoLogControl *log)
+{
+ UndoRecPtr next_urecptr, from_urecptr;
+ uint16 prevlen;
+ UndoLogOffset next_insert;
+ UnpackedUndoRecord *uur;
+ bool refetch = false;
+
+ uur = uur_start;
+
+ while (true)
+ {
+ /* fetch the undo record again if required. */
+ if (refetch)
+ {
+ uur = UndoFetchRecord(urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ refetch = false;
+ }
+
+ next_urecptr = uur->uur_next;
+ prevlen = UndoLogGetPrevLen(log->logno);
+
+ /*
+ * If this is the last transaction in the log then calculate the latest
+ * urec pointer using next insert location of the undo log. Otherwise,
+ * calculate using next transaction's start pointer.
+ */
+ if (uur->uur_next == SpecialUndoRecPtr)
+ {
+ /*
+ * While fetching the next insert location if the new transaction
+ * has already started in this log then lets re-fetch the undo
+ * record.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+ if (!UndoRecPtrIsValid(next_insert))
+ {
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ refetch = true;
+ continue;
+ }
+
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen);
+ break;
+ }
+ else if ((UndoRecPtrGetLogNo(next_urecptr) != log->logno) &&
+ UndoLogIsDiscarded(next_urecptr))
+ {
+ /*
+ * If next_urecptr is in different undolog and its already discarded
+ * that means the undo actions for this transaction which are in the
+ * next log has already been executed and we only need to execute
+ * which are remaining in this log.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+
+ Assert(UndoRecPtrIsValid(next_insert));
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen);
+ break;
+ }
+ else
+ {
+ UnpackedUndoRecord *next_uur;
+
+ next_uur = UndoFetchRecord(next_urecptr,
+ InvalidBlockNumber,
+ InvalidOffsetNumber,
+ InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the next_urecptr is in the same log then calculate the
+ * from pointer using prevlen.
+ */
+ if (UndoRecPtrGetLogNo(next_urecptr) == log->logno)
+ {
+ from_urecptr =
+ UndoGetPrevUndoRecptr(next_urecptr, next_uur->uur_prevlen);
+ UndoRecordRelease(next_uur);
+ break;
+ }
+ else
+ {
+ /*
+ * The transaction is overflowed to the next log, so restart
+ * the processing from then next log.
+ */
+ log = UndoLogGet(UndoRecPtrGetLogNo(next_urecptr), false);
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ uur = next_uur;
+ continue;
+ }
+
+ UndoRecordRelease(next_uur);
+ }
+ }
+
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+
+ return from_urecptr;
+}
+
+/*
+ * Discard the undo logs for temp tables.
+ */
+void
+TempUndoDiscard(UndoLogNumber logno)
+{
+ UndoLogControl *log = UndoLogGet(logno, false);
+
+ /*
+ * Discard the undo log for temp table only. Ensure that there is
+ * something to be discarded there.
+ */
+ Assert (log->meta.persistence == UNDO_TEMP);
+
+ /* Process the undo log. */
+ UndoLogDiscard(MakeUndoRecPtr(log->logno, log->meta.insert),
+ InvalidTransactionId);
+}
diff --git a/src/backend/access/undo/undoinsert.c b/src/backend/access/undo/undoinsert.c
index 7c7e4ff..16016b8 100644
--- a/src/backend/access/undo/undoinsert.c
+++ b/src/backend/access/undo/undoinsert.c
@@ -150,7 +150,6 @@ static UnpackedUndoRecord *UndoGetOneRecord(UnpackedUndoRecord *urec,
UndoPersistence persistence);
static void UndoRecordPrepareTransInfo(UndoRecPtr urecptr,
bool log_switched);
-static void UndoRecordUpdateTransInfo(void);
static int UndoGetBufferSlot(RelFileNode rnode, BlockNumber blk,
ReadBufferMode rbm,
UndoPersistence persistence);
@@ -293,12 +292,59 @@ UndoRecordPrepareTransInfo(UndoRecPtr urecptr, bool log_switched)
}
/*
+ * Update the progress of the undo record in the transaction header.
+ */
+void
+PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress)
+{
+ Buffer buffer = InvalidBuffer;
+ BlockNumber cur_blk;
+ RelFileNode rnode;
+ UndoLogNumber logno = UndoRecPtrGetLogNo(urecptr);
+ UndoLogControl *log;
+ Page page;
+ int already_decoded = 0;
+ int starting_byte;
+ int bufidx;
+ int index = 0;
+
+ log = UndoLogGet(logno, false);
+
+ if (log->meta.persistence == UNDO_TEMP)
+ return;
+
+ UndoRecPtrAssignRelFileNode(rnode, urecptr);
+ cur_blk = UndoRecPtrGetBlockNum(urecptr);
+ starting_byte = UndoRecPtrGetPageOffset(urecptr);
+
+ while (true)
+ {
+ bufidx = UndoGetBufferSlot(rnode, cur_blk,
+ RBM_NORMAL,
+ log->meta.persistence);
+ xact_urec_info.idx_undo_buffers[index++] = bufidx;
+ buffer = undo_buffer[bufidx].buf;
+ page = BufferGetPage(buffer);
+
+ if (UnpackUndoRecord(&xact_urec_info.uur, page, starting_byte,
+ &already_decoded, true))
+ break;
+
+ starting_byte = UndoLogBlockHeaderSize;
+ cur_blk++;
+ }
+
+ xact_urec_info.urecptr = urecptr;
+ xact_urec_info.uur.uur_progress = progress;
+}
+
+/*
* Overwrite the first undo record of the previous transaction to update its
* next pointer. This will just insert the already prepared record by
* UndoRecordPrepareTransInfo. This must be called under the critical section.
* This will just overwrite the undo header not the data.
*/
-static void
+void
UndoRecordUpdateTransInfo(void)
{
UndoLogNumber logno = UndoRecPtrGetLogNo(xact_urec_info.urecptr);
diff --git a/src/backend/access/undo/undorecord.c b/src/backend/access/undo/undorecord.c
index 73076dc..3cd4853 100644
--- a/src/backend/access/undo/undorecord.c
+++ b/src/backend/access/undo/undorecord.c
@@ -90,6 +90,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
*/
if (*already_written == 0)
{
+ work_hdr.urec_rmid = uur->uur_rmid;
work_hdr.urec_type = uur->uur_type;
work_hdr.urec_info = uur->uur_info;
work_hdr.urec_prevlen = uur->uur_prevlen;
@@ -114,6 +115,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
* We should have been passed the same record descriptor as before, or
* caller has messed up.
*/
+ Assert(work_hdr.urec_rmid == uur->uur_rmid);
Assert(work_hdr.urec_type == uur->uur_type);
Assert(work_hdr.urec_info == uur->uur_info);
Assert(work_hdr.urec_prevlen == uur->uur_prevlen);
@@ -276,6 +278,7 @@ UnpackUndoRecord(UnpackedUndoRecord *uur, Page page, int starting_byte,
&my_bytes_decoded, already_decoded, false))
return false;
+ uur->uur_rmid = work_hdr.urec_rmid;
uur->uur_type = work_hdr.urec_type;
uur->uur_info = work_hdr.urec_info;
uur->uur_prevlen = work_hdr.urec_prevlen;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 25b3b03..8d98d3f 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1381,6 +1381,7 @@ vac_truncate_clog(TransactionId frozenXID,
MultiXactId lastSaneMinMulti)
{
TransactionId nextXID = ReadNewTransactionId();
+ TransactionId oldestXidHavingUndo;
Relation relation;
HeapScanDesc scan;
HeapTuple tuple;
@@ -1475,6 +1476,16 @@ vac_truncate_clog(TransactionId frozenXID,
return;
/*
+ * We can't truncate the clog for transactions that still have undo. The
+ * oldestXidHavingUndo will be only valid for zheap storage engine, so it
+ * won't impact any other storage engine.
+ */
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ frozenXID = Min(frozenXID, oldestXidHavingUndo);
+
+ /*
* Advance the oldest value for commit timestamps before truncating, so
* that if a user requests a timestamp for a transaction we're truncating
* away right after this point, they get NULL instead of an ugly "file not
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 71c2321..9ce6ff0 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -12,7 +12,7 @@ subdir = src/backend/postmaster
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
- pgarch.o pgstat.o postmaster.o startup.o syslogger.o walwriter.o
+OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o discardworker.o fork_process.o \
+ pgarch.o pgstat.o postmaster.o startup.o syslogger.o undoworker.o walwriter.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index d2b695e..49df516 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -20,7 +20,9 @@
#include "pgstat.h"
#include "port/atomics.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "storage/dsm.h"
@@ -129,6 +131,15 @@ static const struct
},
{
"ApplyWorkerMain", ApplyWorkerMain
+ },
+ {
+ "UndoLauncherMain", UndoLauncherMain
+ },
+ {
+ "UndoWorkerMain", UndoWorkerMain
+ },
+ {
+ "DiscardWorkerMain", DiscardWorkerMain
}
};
diff --git a/src/backend/postmaster/discardworker.c b/src/backend/postmaster/discardworker.c
new file mode 100644
index 0000000..e4c6719
--- /dev/null
+++ b/src/backend/postmaster/discardworker.c
@@ -0,0 +1,170 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.c
+ * The undo discard worker for asynchronous undo management.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/discardworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include <unistd.h>
+
+/* These are always necessary for a bgworker. */
+#include "access/transam.h"
+#include "miscadmin.h"
+#include "postmaster/bgworker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/lwlock.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+#include "access/undodiscard.h"
+#include "pgstat.h"
+#include "postmaster/discardworker.h"
+#include "storage/procarray.h"
+#include "tcop/tcopprot.h"
+#include "utils/guc.h"
+#include "utils/resowner.h"
+
+static void undoworker_sigterm_handler(SIGNAL_ARGS);
+
+/* max sleep time between cycles (100 milliseconds) */
+#define MIN_NAPTIME_PER_CYCLE 100L
+#define DELAYED_NAPTIME 10 * MIN_NAPTIME_PER_CYCLE
+#define MAX_NAPTIME_PER_CYCLE 100 * MIN_NAPTIME_PER_CYCLE
+
+static bool got_SIGTERM = false;
+static bool hibernate = false;
+static long wait_time = MIN_NAPTIME_PER_CYCLE;
+
+/* SIGTERM: set flag to exit at next convenient time */
+static void
+undoworker_sigterm_handler(SIGNAL_ARGS)
+{
+ got_SIGTERM = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+}
+
+/*
+ * DiscardWorkerRegister -- Register a undo discard worker.
+ */
+void
+DiscardWorkerRegister(void)
+{
+ BackgroundWorker bgw;
+
+ /* TODO: This should be configurable. */
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "discard worker");
+ sprintf(bgw.bgw_library_name, "postgres");
+ sprintf(bgw.bgw_function_name, "DiscardWorkerMain");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * DiscardWorkerMain -- Main loop for the undo discard worker.
+ */
+void
+DiscardWorkerMain(Datum main_arg)
+{
+ ereport(LOG,
+ (errmsg("discard worker started")));
+
+ /* Establish signal handlers. */
+ pqsignal(SIGTERM, undoworker_sigterm_handler);
+ BackgroundWorkerUnblockSignals();
+
+ /* Make it easy to identify our processes. */
+ SetConfigOption("application_name", MyBgworkerEntry->bgw_name,
+ PGC_USERSET, PGC_S_SESSION);
+
+ /*
+ * Create resource owner for discard worker as it need to read the undo
+ * records outside the transaction blocks which intern access buffer read
+ * routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ /* Enter main loop */
+ while (!got_SIGTERM)
+ {
+ int rc;
+ TransactionId OldestXmin, oldestXidHavingUndo;
+
+ OldestXmin = GetOldestXmin(NULL, PROCARRAY_FLAGS_DEFAULT);
+
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+
+ /*
+ * Call the discard routine if there oldestXidHavingUndo is lagging
+ * behind OldestXmin.
+ */
+ if (OldestXmin != InvalidTransactionId &&
+ TransactionIdPrecedes(oldestXidHavingUndo, OldestXmin))
+ {
+ UndoDiscard(OldestXmin, &hibernate);
+
+ /*
+ * If we got some undo logs to discard or discarded something,
+ * then reset the wait_time as we have got work to do.
+ * Note that if there are some undologs that cannot be discarded,
+ * then above condition will remain unsatisified till oldestXmin
+ * remains unchanged and the wait_time will not reset in that case.
+ */
+ if (!hibernate)
+ wait_time = MIN_NAPTIME_PER_CYCLE;
+ }
+
+ /* Wait for more work. */
+ rc = WaitLatch(&MyProc->procLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wait_time,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN);
+
+ ResetLatch(&MyProc->procLatch);
+
+ /*
+ * Increase the wait_time based on the length of inactivity. If wait_time
+ * is within one second, then increment it by 100 ms at a time. Henceforth,
+ * increment it one second at a time, till it reaches ten seconds. Never
+ * increase the wait_time more than ten seconds, it will be too much of
+ * waiting otherwise.
+ */
+ if (rc & WL_TIMEOUT && hibernate)
+ {
+ wait_time += (wait_time < DELAYED_NAPTIME ?
+ MIN_NAPTIME_PER_CYCLE : DELAYED_NAPTIME);
+ if (wait_time > MAX_NAPTIME_PER_CYCLE)
+ wait_time = MAX_NAPTIME_PER_CYCLE;
+ }
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+
+ ReleaseAuxProcessResources(true);
+
+ /* we're done */
+ ereport(LOG,
+ (errmsg("discard worker shutting down")));
+
+ proc_exit(0);
+}
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 9d717d9..81f0698 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -3506,6 +3506,12 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_SYSLOGGER_MAIN:
event_name = "SysLoggerMain";
break;
+ case WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN:
+ event_name = "UndoDiscardWorkerMain";
+ break;
+ case WAIT_EVENT_UNDO_LAUNCHER_MAIN:
+ event_name = "UndoLauncherMain";
+ break;
case WAIT_EVENT_WAL_RECEIVER_MAIN:
event_name = "WalReceiverMain";
break;
@@ -3918,7 +3924,6 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_UNDO_FILE_SYNC:
event_name = "UndoFileSync";
break;
-
case WAIT_EVENT_WALSENDER_TIMELINE_HISTORY_READ:
event_name = "WALSenderTimelineHistoryRead";
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a33a131..9954b21 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -111,10 +111,12 @@
#include "port/pg_bswap.h"
#include "postmaster/autovacuum.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/walsender.h"
#include "storage/fd.h"
@@ -991,6 +993,11 @@ PostmasterMain(int argc, char *argv[])
*/
ApplyLauncherRegister();
+ UndoLauncherRegister();
+
+ /* Register the Undo Discard worker. */
+ DiscardWorkerRegister();
+
/*
* process any libraries that should be preloaded at postmaster start
*/
diff --git a/src/backend/postmaster/undoworker.c b/src/backend/postmaster/undoworker.c
new file mode 100644
index 0000000..bf72203
--- /dev/null
+++ b/src/backend/postmaster/undoworker.c
@@ -0,0 +1,664 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.c
+ * undo launcher and undo worker process.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/undoworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "funcapi.h"
+#include "miscadmin.h"
+#include "pgstat.h"
+
+#include "access/heapam.h"
+#include "access/htup.h"
+#include "access/htup_details.h"
+#include "access/sysattr.h"
+#include "access/xact.h"
+
+#include "catalog/indexing.h"
+#include "catalog/pg_database.h"
+
+#include "libpq/pqsignal.h"
+
+#include "postmaster/bgworker.h"
+#include "postmaster/fork_process.h"
+#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
+
+#include "replication/slot.h"
+#include "replication/worker_internal.h"
+
+#include "storage/ipc.h"
+#include "storage/lmgr.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+
+#include "tcop/tcopprot.h"
+
+#include "utils/fmgroids.h"
+#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/resowner.h"
+
+/* max sleep time between cycles (100 milliseconds) */
+#define DEFAULT_NAPTIME_PER_CYCLE 100L
+#define DEFAULT_RETRY_NAPTIME 50L
+
+int max_undo_workers = 5;
+
+typedef struct UndoApplyWorker
+{
+ /* Indicates if this slot is used or free. */
+ bool in_use;
+
+ /* Increased everytime the slot is taken by new worker. */
+ uint16 generation;
+
+ /* Pointer to proc array. NULL if not running. */
+ PGPROC *proc;
+
+ /* Database id to connect to. */
+ Oid dbid;
+} UndoApplyWorker;
+
+UndoApplyWorker *MyUndoWorker = NULL;
+
+typedef struct UndoApplyCtxStruct
+{
+ /* Supervisor process. */
+ pid_t launcher_pid;
+
+ /* Background workers. */
+ UndoApplyWorker workers[FLEXIBLE_ARRAY_MEMBER];
+} UndoApplyCtxStruct;
+
+UndoApplyCtxStruct *UndoApplyCtx;
+
+static void undo_worker_onexit(int code, Datum arg);
+static void undo_worker_cleanup(UndoApplyWorker *worker);
+
+static volatile sig_atomic_t got_SIGHUP = false;
+
+/*
+ * Wait for a background worker to start up and attach to the shmem context.
+ *
+ * This is only needed for cleaning up the shared memory in case the worker
+ * fails to attach.
+ */
+static void
+WaitForUndoWorkerAttach(UndoApplyWorker *worker,
+ uint16 generation,
+ BackgroundWorkerHandle *handle)
+{
+ BgwHandleStatus status;
+ int rc;
+
+ for (;;)
+ {
+ pid_t pid;
+
+ CHECK_FOR_INTERRUPTS();
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+
+ /* Worker either died or has started; no need to do anything. */
+ if (!worker->in_use || worker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ LWLockRelease(UndoWorkerLock);
+
+ /* Check if worker has died before attaching, and clean up after it. */
+ status = GetBackgroundWorkerPid(handle, &pid);
+
+ if (status == BGWH_STOPPED)
+ {
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ /* Ensure that this was indeed the worker we waited for. */
+ if (generation == worker->generation)
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ /*
+ * We need timeout because we generally don't get notified via latch
+ * about the worker attach. But we don't expect to have to wait long.
+ */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 10L, WAIT_EVENT_BGWORKER_STARTUP);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+ }
+
+ return;
+}
+
+/*
+ * Get dbid from the worker slot.
+ */
+static Oid
+slot_get_dbid(int slot)
+{
+ Oid dbid;
+
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty,",
+ slot)));
+ }
+
+ dbid = MyUndoWorker->dbid;
+
+ LWLockRelease(UndoWorkerLock);
+
+ return dbid;
+}
+
+/*
+ * Attach to a slot.
+ */
+static void
+undo_worker_attach(int slot)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty, cannot attach",
+ slot)));
+ }
+
+ if (MyUndoWorker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is already used by "
+ "another worker, cannot attach", slot)));
+ }
+
+ MyUndoWorker->proc = MyProc;
+ before_shmem_exit(undo_worker_onexit, (Datum) 0);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Walks the workers array and searches for one that matches given
+ * dbid.
+ */
+static UndoApplyWorker *
+undo_worker_find(Oid dbid)
+{
+ int i;
+ UndoApplyWorker *res = NULL;
+
+ Assert(LWLockHeldByMe(UndoWorkerLock));
+
+ /* Search for attached worker for a given db id. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (w->in_use && w->dbid == dbid)
+ {
+ res = w;
+ break;
+ }
+ }
+
+ return res;
+}
+
+/*
+ * Check whether the dbid exist or not.
+ *
+ * Refer comments from GetDatabaseTupleByOid.
+ * FIXME: Should we expose GetDatabaseTupleByOid and directly use it.
+ */
+static bool
+dbid_exist(Oid dboid)
+{
+ HeapTuple tuple;
+ Relation relation;
+ SysScanDesc scan;
+ ScanKeyData key[1];
+ bool result = false;
+ /*
+ * form a scan key
+ */
+ ScanKeyInit(&key[0],
+ Anum_pg_database_oid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dboid));
+
+ relation = heap_open(DatabaseRelationId, AccessShareLock);
+ scan = systable_beginscan(relation, DatabaseOidIndexId,
+ criticalSharedRelcachesBuilt,
+ NULL,
+ 1, key);
+
+ tuple = systable_getnext(scan);
+
+ if (HeapTupleIsValid(tuple))
+ result = true;
+
+ /* all done */
+ systable_endscan(scan);
+ heap_close(relation, AccessShareLock);
+
+ return result;
+}
+
+/*
+ * Start new undo apply background worker, if possible otherwise return false.
+ */
+static bool
+undo_worker_launch(Oid dbid)
+{
+ BackgroundWorker bgw;
+ BackgroundWorkerHandle *bgw_handle;
+ uint16 generation;
+ int i;
+ int slot = 0;
+ UndoApplyWorker *worker = NULL;
+
+ /*
+ * We need to do the modification of the shared memory under lock so that
+ * we have consistent view.
+ */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ /* Find unused worker slot. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (!w->in_use)
+ {
+ worker = w;
+ slot = i;
+ break;
+ }
+ }
+
+ /* There are no more free worker slots */
+ if (worker == NULL)
+ return false;
+
+ /* Prepare the worker slot. */
+ worker->in_use = true;
+ worker->proc = NULL;
+ worker->dbid = dbid;
+ worker->generation++;
+
+ generation = worker->generation;
+ LWLockRelease(UndoWorkerLock);
+
+ /* Register the new dynamic worker. */
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoWorkerMain");
+ snprintf(bgw.bgw_type, BGW_MAXLEN, "undo apply worker");
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "undo apply worker");
+
+ bgw.bgw_restart_time = BGW_NEVER_RESTART;
+ bgw.bgw_notify_pid = MyProcPid;
+ bgw.bgw_main_arg = Int32GetDatum(slot);
+
+ StartTransactionCommand();
+ /* Check the database exists or not. */
+ if (!dbid_exist(dbid))
+ {
+ CommitTransactionCommand();
+
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return true;
+ }
+
+ /*
+ * Acquire database object lock before launching the worker so that it
+ * doesn't get dropped while worker is connecting to the database.
+ */
+ LockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+
+ /* Recheck whether database still exists or not. */
+ if (!dbid_exist(dbid))
+ {
+ CommitTransactionCommand();
+
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return true;
+ }
+
+ if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle))
+ {
+ /* Failed to start worker, so clean up the worker slot. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+
+ UnlockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+ CommitTransactionCommand();
+
+ return false;
+ }
+
+ /* Now wait until it attaches. */
+ WaitForUndoWorkerAttach(worker, generation, bgw_handle);
+
+ /*
+ * By this point the undo-worker has already connected to the database so we
+ * can release the database lock.
+ */
+ UnlockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+ CommitTransactionCommand();
+
+ return true;
+}
+
+/*
+ * Detach the worker (cleans up the worker info).
+ */
+static void
+undo_worker_detach(void)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ undo_worker_cleanup(MyUndoWorker);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Clean up worker info.
+ */
+static void
+undo_worker_cleanup(UndoApplyWorker *worker)
+{
+ Assert(LWLockHeldByMeInMode(UndoWorkerLock, LW_EXCLUSIVE));
+
+ worker->in_use = false;
+ worker->proc = NULL;
+ worker->dbid = InvalidOid;
+}
+
+/*
+ * Cleanup function for undo worker launcher.
+ *
+ * Called on undo worker launcher exit.
+ */
+static void
+undo_launcher_onexit(int code, Datum arg)
+{
+ UndoApplyCtx->launcher_pid = 0;
+}
+
+/* SIGHUP: set flag to reload configuration at next convenient time */
+static void
+undo_launcher_sighup(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGHUP = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+undo_worker_onexit(int code, Datum arg)
+{
+ undo_worker_detach();
+}
+
+/*
+ * UndoLauncherShmemSize
+ * Compute space needed for undo launcher shared memory
+ */
+Size
+UndoLauncherShmemSize(void)
+{
+ Size size;
+
+ /*
+ * Need the fixed struct and the array of LogicalRepWorker.
+ */
+ size = sizeof(UndoApplyCtxStruct);
+ size = MAXALIGN(size);
+ size = add_size(size, mul_size(max_undo_workers,
+ sizeof(UndoApplyWorker)));
+ return size;
+}
+
+/*
+ * UndoLauncherRegister
+ * Register a background worker running the undo worker launcher.
+ */
+void
+UndoLauncherRegister(void)
+{
+ BackgroundWorker bgw;
+
+ if (max_undo_workers == 0)
+ return;
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoLauncherMain");
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "undo worker launcher");
+ snprintf(bgw.bgw_type, BGW_MAXLEN,
+ "undo worker launcher");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * UndoLauncherShmemInit
+ * Allocate and initialize undo worker launcher shared memory
+ */
+void
+UndoLauncherShmemInit(void)
+{
+ bool found;
+
+ UndoApplyCtx = (UndoApplyCtxStruct *)
+ ShmemInitStruct("Undo Worker Launcher Data",
+ UndoLauncherShmemSize(),
+ &found);
+
+ if (!found)
+ memset(UndoApplyCtx, 0, UndoLauncherShmemSize());
+}
+
+/*
+ * Main loop for the undo worker launcher process.
+ */
+void
+UndoLauncherMain(Datum main_arg)
+{
+ MemoryContext tmpctx;
+ MemoryContext oldctx;
+
+ ereport(DEBUG1,
+ (errmsg("undo launcher started")));
+
+ before_shmem_exit(undo_launcher_onexit, (Datum) 0);
+
+ Assert(UndoApplyCtx->launcher_pid == 0);
+ UndoApplyCtx->launcher_pid = MyProcPid;
+
+ /* Establish signal handlers. */
+ pqsignal(SIGHUP, undo_launcher_sighup);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * Establish connection to nailed catalogs (we only ever access
+ * pg_subscription).
+ */
+ BackgroundWorkerInitializeConnection(NULL, NULL, 0);
+
+ /* Use temporary context for the database list and worker info. */
+ tmpctx = AllocSetContextCreate(TopMemoryContext,
+ "Undo worker Launcher context",
+ ALLOCSET_DEFAULT_SIZES);
+ /* Enter main loop */
+ for (;;)
+ {
+ int rc;
+ List *dblist;
+ ListCell *l;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* switch to the temp context. */
+ oldctx = MemoryContextSwitchTo(tmpctx);
+ dblist = RollbackHTGetDBList();
+
+ foreach(l, dblist)
+ {
+ UndoApplyWorker *w;
+ Oid dbid = lfirst_oid(l);
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+ w = undo_worker_find(dbid);
+ LWLockRelease(UndoWorkerLock);
+
+ if (w == NULL)
+ {
+retry:
+ if (!undo_worker_launch(dbid))
+ {
+ /* Could not launch the worker, retry after sometime, */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_RETRY_NAPTIME,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+ goto retry;
+ }
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+
+ /* Clean the temporary memory. */
+ MemoryContextReset(tmpctx);
+
+ /* Wait for more work. */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_NAPTIME_PER_CYCLE,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ if (got_SIGHUP)
+ {
+ got_SIGHUP = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+}
+
+/*
+ * UndoWorkerMain -- Main loop for the undo apply worker.
+ */
+void
+UndoWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ Oid dbid;
+
+ dbid = slot_get_dbid(worker_slot);
+
+ /* Setup signal handling */
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /* Connect to the database. */
+ BackgroundWorkerInitializeConnectionByOid(dbid, 0, 0);
+
+ /* Attach to slot */
+ undo_worker_attach(worker_slot);
+
+ /*
+ * Create resource owner for undo worker. Undo worker need this as it
+ * need to read the undo records outside the transaction blocks which
+ * intern access buffer read routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ RollbackFromHT(dbid);
+
+ ReleaseAuxProcessResources(true);
+
+ proc_exit(0);
+}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 1a7a381..b9c0e18 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -159,6 +159,10 @@ LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *recor
ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(record),
buf.origptr);
break;
+ case RM_UNDOACTION_ID:
+ /* Logical decoding is not yet implemented for undoactions. */
+ Assert(0);
+ break;
case RM_NEXT_ID:
elog(ERROR, "unexpected RM_NEXT_ID rmgr_id: %u", (RmgrIds) XLogRecGetRmid(buf.record));
}
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 4725cbe..1470041 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -29,6 +29,8 @@
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
#include "replication/walreceiver.h"
@@ -152,6 +154,8 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
size = add_size(size, BackendRandomShmemSize());
+ size = add_size(size, RollbackHTSize());
+ size = add_size(size, UndoLauncherShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -226,6 +230,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
SUBTRANSShmemInit();
MultiXactShmemInit();
InitBufferPool();
+ InitRollbackHashTable();
/*
* Set up lock manager
@@ -264,6 +269,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
WalSndShmemInit();
WalRcvShmemInit();
ApplyLauncherShmemInit();
+ UndoLauncherShmemInit();
/*
* Set up other modules that need some shared memory space
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 554af46..52f6959 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -51,3 +51,5 @@ BackendRandomLock 43
LogicalRepWorkerLock 44
CLogTruncationLock 45
UndoLogLock 46
+RollbackHTLock 47
+UndoWorkerLock 48
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 33387fb..69c7b6a 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -286,6 +286,8 @@ InitProcGlobal(void)
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemAlloc(sizeof(slock_t));
SpinLockInit(ProcStructLock);
+
+ pg_atomic_init_u64(&ProcGlobal->oldestXidWithEpochHavingUndo, 0);
}
/*
diff --git a/src/backend/utils/adt/lockfuncs.c b/src/backend/utils/adt/lockfuncs.c
index 525decb..8b2ad64 100644
--- a/src/backend/utils/adt/lockfuncs.c
+++ b/src/backend/utils/adt/lockfuncs.c
@@ -32,6 +32,7 @@ const char *const LockTagTypeNames[] = {
"virtualxid",
"speculative token",
"object",
+ "undoaction",
"userlock",
"advisory"
};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index c693977..12e7704 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -121,6 +121,7 @@ bool allowSystemTableMods = false;
int work_mem = 1024;
int maintenance_work_mem = 16384;
int max_parallel_maintenance_workers = 2;
+int rollback_overflow_size = 64;
/*
* Primary determinants of sizes of shared-memory structures.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 8b0ade6..48ce9df 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2862,6 +2862,17 @@ static struct config_int ConfigureNamesInt[] =
},
{
+ {"rollback_overflow_size", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Rollbacks greater than this size are done lazily"),
+ NULL,
+ GUC_UNIT_MB
+ },
+ &rollback_overflow_size,
+ 64, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
{"wal_segment_size", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the size of write ahead log segments."),
NULL,
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index a376875..8ebb9c2 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -78,8 +78,8 @@ pg_control_system(PG_FUNCTION_ARGS)
Datum
pg_control_checkpoint(PG_FUNCTION_ARGS)
{
- Datum values[19];
- bool nulls[19];
+ Datum values[20];
+ bool nulls[20];
TupleDesc tupdesc;
HeapTuple htup;
ControlFileData *ControlFile;
@@ -128,6 +128,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
XIDOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 18, "checkpoint_time",
TIMESTAMPTZOID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 19, "oldest_xid_with_epoch_having_undo",
+ TIMESTAMPTZOID, -1, 0);
tupdesc = BlessTupleDesc(tupdesc);
/* Read the control file. */
@@ -202,6 +204,9 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
time_t_to_timestamptz(ControlFile->checkPointCopy.time));
nulls[17] = false;
+ values[18] = Int64GetDatum(ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo);
+ nulls[18] = false;
+
htup = heap_form_tuple(tupdesc, values, nulls);
PG_RETURN_DATUM(HeapTupleGetDatum(htup));
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1fa02d2..82b9cf6 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -737,4 +737,11 @@
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
+# If often there are large transactions requiring rollbacks, then we can push
+# them to undo-workers for better performance. The size specifeid by the
+# parameter below, determines the minimum size of the rollback requests to be
+# sent to the undo-worker.
+#
+#rollback_overflow_size = 64
+
# Add settings for extensions here
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 895a51f..753929e 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -278,6 +278,8 @@ main(int argc, char *argv[])
ControlFile->checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile->checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo);
printf(_("Time of latest checkpoint: %s\n"),
ckpttime_str);
printf(_("Fake LSN counter for unlogged rels: %X/%X\n"),
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 6fb403a..72714dd 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -448,6 +448,7 @@ main(int argc, char *argv[])
if (ControlFile.checkPointCopy.oldestXid < FirstNormalTransactionId)
ControlFile.checkPointCopy.oldestXid += FirstNormalTransactionId;
ControlFile.checkPointCopy.oldestXidDB = InvalidOid;
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo = 0;
}
if (set_oldest_commit_ts_xid != 0)
@@ -716,6 +717,8 @@ GuessControlValues(void)
ControlFile.checkPointCopy.oldestMultiDB = InvalidOid;
ControlFile.checkPointCopy.time = (pg_time_t) time(NULL);
ControlFile.checkPointCopy.oldestActiveXid = InvalidTransactionId;
+ ControlFile.checkPointCopy.nextXid = 0;
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo = 0;
ControlFile.state = DB_SHUTDOWNED;
ControlFile.time = (pg_time_t) time(NULL);
@@ -808,6 +811,8 @@ PrintControlValues(bool guessed)
ControlFile.checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile.checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo);
printf(_("Maximum data alignment: %u\n"),
ControlFile.maxAlign);
/* we don't print floatFormat since can't say much useful about it */
@@ -884,6 +889,8 @@ PrintNewControlValues(void)
ControlFile.checkPointCopy.oldestXid);
printf(_("OldestXID's DB: %u\n"),
ControlFile.checkPointCopy.oldestXidDB);
+ printf(_("OldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo);
}
if (set_xid_epoch != -1)
diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c
index 4002847..e71a71e 100644
--- a/src/bin/pg_rewind/parsexlog.c
+++ b/src/bin/pg_rewind/parsexlog.c
@@ -29,7 +29,7 @@
* RmgrNames is an array of resource manager names, to make error messages
* a bit nicer.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
name,
static const char *RmgrNames[RM_MAX_ID + 1] = {
diff --git a/src/bin/pg_waldump/rmgrdesc.c b/src/bin/pg_waldump/rmgrdesc.c
index 938150d..3ef0a60 100644
--- a/src/bin/pg_waldump/rmgrdesc.c
+++ b/src/bin/pg_waldump/rmgrdesc.c
@@ -20,6 +20,7 @@
#include "access/nbtxlog.h"
#include "access/rmgr.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -33,7 +34,7 @@
#include "storage/standbydefs.h"
#include "utils/relmapper.h"
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
{ name, desc, identify},
const RmgrDescData RmgrDescTable[RM_MAX_ID + 1] = {
diff --git a/src/include/access/rmgr.h b/src/include/access/rmgr.h
index c9b5c56..e1fb42a 100644
--- a/src/include/access/rmgr.h
+++ b/src/include/access/rmgr.h
@@ -19,7 +19,7 @@ typedef uint8 RmgrId;
* Note: RM_MAX_ID must fit in RmgrId; widening that type will affect the XLOG
* file format.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
symname,
typedef enum RmgrIds
diff --git a/src/include/access/rmgrlist.h b/src/include/access/rmgrlist.h
index 9c6fca4..eee7835 100644
--- a/src/include/access/rmgrlist.h
+++ b/src/include/access/rmgrlist.h
@@ -25,26 +25,27 @@
*/
/* symbol name, textual name, redo, desc, identify, startup, cleanup */
-PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL)
-PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL)
-PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL)
-PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL)
-PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL)
-PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask)
-PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask)
-PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask)
-PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask)
-PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask)
-PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask)
-PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask)
-PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL)
-PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL)
-PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask)
-PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL)
-PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL)
+PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask, NULL, NULL)
+PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask, NULL, NULL)
+PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask, NULL, NULL)
+PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask, NULL, NULL)
+PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask, NULL, NULL)
+PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask, NULL, NULL)
+PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask, NULL, NULL)
+PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask, NULL, NULL)
+PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOACTION_ID, "UndoAction", undoaction_redo, undoaction_desc, undoaction_identify, NULL, NULL, NULL, NULL, NULL)
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 83ec3f1..7b983ef 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -68,6 +68,10 @@
(AssertMacro(TransactionIdIsNormal(id1) && TransactionIdIsNormal(id2)), \
(int32) ((id1) - (id2)) > 0)
+/* Extract xid from a value comprised of epoch and xid */
+#define GetXidFromEpochXid(epochxid) \
+ ((uint32) (epochxid) & 0XFFFFFFFF)
+
/* ----------
* Object ID (OID) zero is InvalidOid.
*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 0e932da..fb05c3b 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -41,7 +41,7 @@ extern GlobalTransaction MarkAsPreparing(TransactionId xid, const char *gid,
TimestampTz prepared_at,
Oid owner, Oid databaseid);
-extern void StartPrepare(GlobalTransaction gxact);
+extern void StartPrepare(GlobalTransaction gxact, UndoRecPtr *, UndoRecPtr *);
extern void EndPrepare(GlobalTransaction gxact);
extern bool StandbyTransactionIdIsPrepared(TransactionId xid);
diff --git a/src/include/access/undoaction.h b/src/include/access/undoaction.h
new file mode 100644
index 0000000..5455259
--- /dev/null
+++ b/src/include/access/undoaction.h
@@ -0,0 +1,28 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.h
+ * undo action prototypes
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_H
+#define UNDOACTION_H
+
+#include "postgres.h"
+
+#include "access/undolog.h"
+#include "access/undorecord.h"
+
+/* undo record information */
+typedef struct UndoRecInfo
+{
+ UndoRecPtr urp; /* undo recptr (undo record location). */
+ UnpackedUndoRecord *uur; /* actual undo record. */
+} UndoRecInfo;
+
+#endif
diff --git a/src/include/access/undoaction_xlog.h b/src/include/access/undoaction_xlog.h
new file mode 100644
index 0000000..bfc6418
--- /dev/null
+++ b/src/include/access/undoaction_xlog.h
@@ -0,0 +1,74 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction_xlog.h
+ * undo action XLOG definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction_xlog.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_XLOG_H
+#define UNDOACTION_XLOG_H
+
+#include "access/undolog.h"
+#include "access/xlogreader.h"
+#include "lib/stringinfo.h"
+#include "storage/off.h"
+
+/*
+ * WAL record definitions for undoactions.c's WAL operations
+ */
+#define XLOG_UNDO_PAGE 0x00
+#define XLOG_UNDO_RESET_SLOT 0x10
+#define XLOG_UNDO_APPLY_PROGRESS 0x20
+
+/*
+ * xl_undoaction_page flag values, 8 bits are available.
+ */
+#define XLU_PAGE_CONTAINS_TPD_SLOT (1<<0)
+#define XLU_PAGE_CLEAR_VISIBILITY_MAP (1<<1)
+#define XLU_CONTAINS_TPD_OFFSET_MAP (1<<2)
+#define XLU_INIT_PAGE (1<<3)
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_page
+{
+ UndoRecPtr urec_ptr;
+ TransactionId xid;
+ int trans_slot_id; /* transaction slot id */
+} xl_undoaction_page;
+
+#define SizeOfUndoActionPage (offsetof(xl_undoaction_page, trans_slot_id) + sizeof(int))
+
+/* This is what we need to know about undo apply progress */
+typedef struct xl_undoapply_progress
+{
+ UndoRecPtr urec_ptr;
+ uint32 progress;
+} xl_undoapply_progress;
+
+#define SizeOfUndoActionProgress (offsetof(xl_undoapply_progress, progress) + sizeof(uint32))
+
+/*
+ * xl_undoaction_reset_slot flag values, 8 bits are available.
+ */
+#define XLU_RESET_CONTAINS_TPD_SLOT (1<<0)
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_reset_slot
+{
+ UndoRecPtr urec_ptr;
+ int trans_slot_id; /* transaction slot id */
+ uint8 flags;
+} xl_undoaction_reset_slot;
+
+#define SizeOfUndoActionResetSlot (offsetof(xl_undoaction_reset_slot, flags) + sizeof(uint8))
+
+extern void undoaction_redo(XLogReaderState *record);
+extern void undoaction_desc(StringInfo buf, XLogReaderState *record);
+extern const char *undoaction_identify(uint8 info);
+
+#endif /* UNDOACTION_XLOG_H */
diff --git a/src/include/access/undodiscard.h b/src/include/access/undodiscard.h
new file mode 100644
index 0000000..70d6408
--- /dev/null
+++ b/src/include/access/undodiscard.h
@@ -0,0 +1,31 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoinsert.h
+ * undo discard definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undodiscard.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDODISCARD_H
+#define UNDODISCARD_H
+
+#include "access/undolog.h"
+#include "access/xlogdefs.h"
+#include "catalog/pg_class.h"
+#include "storage/lwlock.h"
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+extern void UndoDiscard(TransactionId xmin, bool *hibernate);
+
+#endif /* UNDODISCARD_H */
diff --git a/src/include/access/undoinsert.h b/src/include/access/undoinsert.h
index fe4a97e..8711432 100644
--- a/src/include/access/undoinsert.h
+++ b/src/include/access/undoinsert.h
@@ -102,6 +102,8 @@ extern void UndoSetPrepareSize(UnpackedUndoRecord *undorecords, int nrecords,
extern UndoRecPtr UndoGetPrevUndoRecptr(UndoRecPtr urp, uint16 prevlen);
extern void UndoRecordOnUndoLogChange(UndoPersistence persistence);
+extern void PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress);
+extern void UndoRecordUpdateTransInfo(void);
/* Reset globals related to undo buffers */
extern void ResetUndoBuffers(void);
diff --git a/src/include/access/undorecord.h b/src/include/access/undorecord.h
index 9ca2455..34f248f 100644
--- a/src/include/access/undorecord.h
+++ b/src/include/access/undorecord.h
@@ -20,6 +20,17 @@
#include "storage/buf.h"
#include "storage/off.h"
+typedef enum undorectype
+{
+ UNDO_INSERT,
+ UNDO_MULTI_INSERT,
+ UNDO_DELETE,
+ UNDO_INPLACE_UPDATE,
+ UNDO_UPDATE,
+ UNDO_XID_LOCK_ONLY,
+ UNDO_XID_MULTI_LOCK_ONLY,
+ UNDO_ITEMID_UNUSED
+} undorectype;
/*
* Every undo record begins with an UndoRecordHeader structure, which is
@@ -30,6 +41,7 @@
*/
typedef struct UndoRecordHeader
{
+ RmgrId urec_rmid; /* RMGR [XXX:TODO: this creates an alignment hole?] */
uint8 urec_type; /* record type code */
uint8 urec_info; /* flag bits */
uint16 urec_prevlen; /* length of previous record in bytes */
@@ -149,6 +161,7 @@ typedef struct UndoRecordPayload
*/
typedef struct UnpackedUndoRecord
{
+ RmgrId uur_rmid; /* rmgr ID */
uint8 uur_type; /* record type code */
uint8 uur_info; /* flag bits */
uint16 uur_prevlen; /* length of previous record */
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 73394c5..ad8cabe 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -419,6 +419,7 @@ extern XLogRecPtr XactLogAbortRecord(TimestampTz abort_time,
int xactflags, TransactionId twophase_xid,
const char *twophase_gid);
extern void xact_redo(XLogReaderState *record);
+extern void XactPerformUndoActionsIfPending(void);
/* xactdesc.c */
extern void xact_desc(StringInfo buf, XLogReaderState *record);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d4e742f..1091fe7 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -225,6 +225,9 @@ extern bool XLOG_DEBUG;
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+/* Generate a 64-bit xid by using epoch and 32-bit xid. */
+#define MakeEpochXid(epoch, xid) \
+ ((epoch << 32) | (xid))
/* Checkpoint statistics */
typedef struct CheckpointStatsData
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 30610b3..9226cff 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -21,8 +21,10 @@
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
+#include "access/undorecord.h"
#include "datatype/timestamp.h"
#include "lib/stringinfo.h"
+#include "nodes/pg_list.h"
#include "pgtime.h"
#include "storage/block.h"
#include "storage/relfilenode.h"
@@ -294,9 +296,14 @@ typedef struct RmgrData
void (*rm_startup) (void);
void (*rm_cleanup) (void);
void (*rm_mask) (char *pagedata, BlockNumber blkno);
+ bool (*rm_undo) (List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock,
+ int options);
+ void (*rm_undo_desc) (StringInfo buf, UnpackedUndoRecord *record);
} RmgrData;
-extern const RmgrData RmgrTable[];
+extern PGDLLIMPORT const RmgrData RmgrTable[];
/*
* Exported to support xlog switching from checkpointer
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 773d9e6..6918efc 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -61,6 +61,13 @@ typedef struct CheckPoint
* set to InvalidTransactionId.
*/
TransactionId oldestActiveXid;
+
+ /*
+ * Oldest transaction id with epoc which is having undo. Include this value
+ * in the checkpoint record so that whenever server starts we get proper
+ * value.
+ */
+ uint64 oldestXidWithEpochHavingUndo;
} CheckPoint;
/* XLOG info values for XLOG rmgr */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index d6b32c0..549bb38 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -245,6 +245,7 @@ extern PGDLLIMPORT bool allowSystemTableMods;
extern PGDLLIMPORT int work_mem;
extern PGDLLIMPORT int maintenance_work_mem;
extern PGDLLIMPORT int max_parallel_maintenance_workers;
+extern PGDLLIMPORT int rollback_overflow_size;
extern int VacuumCostPageHit;
extern int VacuumCostPageMiss;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 763379e..7c5f05e 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -749,6 +749,7 @@ typedef enum BackendState
#define PG_WAIT_TIMEOUT 0x09000000U
#define PG_WAIT_IO 0x0A000000U
#define PG_WAIT_PAGE_TRANS_SLOT 0x0B000000U
+#define PG_WAIT_ROLLBACK_HT 0x0C000000U
/* ----------
* Wait Events - Activity
@@ -774,6 +775,8 @@ typedef enum
WAIT_EVENT_WAL_RECEIVER_MAIN,
WAIT_EVENT_WAL_SENDER_MAIN,
WAIT_EVENT_WAL_WRITER_MAIN,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN
} WaitEventActivity;
/* ----------
diff --git a/src/include/postmaster/discardworker.h b/src/include/postmaster/discardworker.h
new file mode 100644
index 0000000..f00c6c4
--- /dev/null
+++ b/src/include/postmaster/discardworker.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.h
+ * Exports from postmaster/discardworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/discardworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _DISCARDWORKER_H
+#define _DISCARDWORKER_H
+
+/*
+ * This function will perform multiple actions based on need. (a) retrieve
+ * transactions which have become all-visible and truncate the associated undo
+ * logs or will increment the tail pointer. (b) drop the buffers corresponding
+ * to truncated pages.
+ */
+extern void DiscardWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void DiscardWorkerRegister(void);
+
+#endif /* _DISCARDWORKER_H */
diff --git a/src/include/postmaster/undoloop.h b/src/include/postmaster/undoloop.h
new file mode 100644
index 0000000..21d91c3
--- /dev/null
+++ b/src/include/postmaster/undoloop.h
@@ -0,0 +1,89 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoloop.h
+ * Exports from postmaster/undoloop.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ *
+ * src/include/postmaster/undoloop.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOLOOP_H
+#define _UNDOLOOP_H
+
+#include "access/undoinsert.h"
+#include "utils/hsearch.h"
+#include "utils/relcache.h"
+
+
+/* Various options while executing the undo actions for the page. */
+#define UNDO_ACTION_UPDATE_TPD 0x0001
+
+/* Remembers the last seen RecentGlobalXmin */
+TransactionId latestRecentGlobalXmin;
+
+/*
+ * This function will read the undo records starting from the undo
+ * from_urecptr till to_urecptr and if to_urecptr is invalid then till the
+ * first undo location of transaction. This also discards the buffers by
+ * calling DropUndoBuffers for which undo log is removed. This function
+ * can be used by RollbackToSavePoint, by Rollback, by undoworker to complete
+ * the work of errored out transactions or when there is an error in single
+ * user mode.
+ */
+extern void execute_undo_actions(UndoRecPtr from_urecptr,
+ UndoRecPtr to_urecptr, bool nopartial, bool rewind, bool rellock);
+extern void process_and_execute_undo_actions_page(UndoRecPtr from_urecptr,
+ Relation rel, Buffer buffer, uint32 epoch,
+ TransactionId xid, int slot_no);
+
+/*
+ * This function will be responsible to truncate the undo logs
+ * for transactions that become all-visible after RecentGlobalXmin has
+ * advanced (value is different than latestRecentGlobalXmin). The easiest
+ * way could be to traverse the undo log array that contains least transaction
+ * id for that undo log and see if it precedes RecentGlobalXmin, then start
+ * discarding the undo log for that transaction (moving the tail pointer of
+ * undo log) till it finds the transaction which is not all-visible. This also
+ * discards the buffers by calling ForgetBuffer for which undo log is
+ * removed. This function can be invoked by undoworker or after commit in
+ * single user mode.
+ */
+extern void recover_undo_pages();
+
+/*
+ * To increase the efficiency of the zheap system, we create a hash table for
+ * the rollbacks. All the rollback requests exceeding certain threshold, are
+ * pushed to this table. Undo worker starts reading the entries from this hash
+ * table one at a time, performs undo actions related to the respective xid and
+ * removes them from the hash table. This way backend is free from performing the
+ * undo actions in case of heavy rollbacks. The data structures and the routines
+ * required for this infrastructure are as follows.
+ */
+
+/* This is the data structure for each hash table entry for rollbacks. */
+typedef struct RollbackHashEntry
+{
+ UndoRecPtr start_urec_ptr;
+ UndoRecPtr end_urec_ptr;
+ Oid dbid;
+} RollbackHashEntry;
+
+extern bool RollbackHTIsFull(void);
+
+/* To push the rollback requests from backend to the respective hash table */
+extern bool PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr,
+ Oid dbid);
+
+/* To perform the undo actions reading from the hash table */
+extern void RollbackFromHT(Oid dbid);
+/* To calculate the size of the hash table size for rollabcks. */
+extern int RollbackHTSize(void);
+
+/* To initialize the hash table in shared memory for rollbacks. */
+extern void InitRollbackHashTable(void);
+extern List *RollbackHTGetDBList(void);
+extern bool ConditionTransactionUndoActionLock(TransactionId xid);
+extern void TransactionUndoActionLockRelease(TransactionId xid);
+#endif /* _UNDOLOOP_H */
diff --git a/src/include/postmaster/undoworker.h b/src/include/postmaster/undoworker.h
new file mode 100644
index 0000000..c277e01
--- /dev/null
+++ b/src/include/postmaster/undoworker.h
@@ -0,0 +1,39 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.h
+ * Exports from postmaster/undoworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/undoworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOWORKER_H
+#define _UNDOWORKER_H
+
+/* GUC options */
+/* undo worker sleep time between rounds */
+extern int UndoWorkerDelay;
+
+/*
+ * This function will perform multiple actions based on need. (a) retreive
+ * transaction and its corresponding undopoiter from shared memory queue and
+ * call undoloop to perform undo actions. After applying all the undo records
+ * for a particular transaction, it will increment the tail pointer in undo log.
+ * (b) it needs to retrieve transactions which have become all-visible and truncate
+ * the associated undo logs or will increment the tail pointer. (c) udjust the
+ * number of undo workers based on the work required to perform undo actions
+ * (it could be size of shared memory queue containing transactions that needs
+ * aborts). (d) drop the buffers corresponding to truncated pages (e) Sleep for
+ * UndoWorkerDelay, if there is no more work.
+ */
+extern void UndoWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void UndoLauncherRegister(void);
+extern void UndoLauncherShmemInit(void);
+extern Size UndoLauncherShmemSize(void);
+extern void UndoLauncherMain(Datum main_arg);
+extern void UndoWorkerMain(Datum main_arg);
+
+#endif /* _UNDOWORKER_H */
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index a37fda7..675761c 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -150,6 +150,8 @@ typedef enum LockTagType
LOCKTAG_VIRTUALTRANSACTION, /* virtual transaction (ditto) */
/* ID info for a virtual transaction is its VirtualTransactionId */
LOCKTAG_SPECULATIVE_TOKEN, /* speculative insertion Xid and token */
+ /* ID info for an transaction undoaction is transaction id */
+ LOCKTAG_TRANSACTION_UNDOACTION, /* transaction (waiting for undoaction) */
/* ID info for a transaction is its TransactionId */
LOCKTAG_OBJECT, /* non-relation database object */
/* ID info for an object is DB OID + CLASS OID + OBJECT OID + SUBID */
@@ -246,6 +248,14 @@ typedef struct LOCKTAG
(locktag).locktag_type = LOCKTAG_SPECULATIVE_TOKEN, \
(locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+#define SET_LOCKTAG_TRANSACTION_UNDOACTION(locktag,xid) \
+ ((locktag).locktag_field1 = (xid), \
+ (locktag).locktag_field2 = 0, \
+ (locktag).locktag_field3 = 0, \
+ (locktag).locktag_field4 = 0, \
+ (locktag).locktag_type = LOCKTAG_TRANSACTION_UNDOACTION, \
+ (locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+
#define SET_LOCKTAG_OBJECT(locktag,dboid,classoid,objoid,objsubid) \
((locktag).locktag_field1 = (dboid), \
(locktag).locktag_field2 = (classoid), \
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index cb613c8..f6b9d98 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -270,6 +270,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Oldest transaction id which is having undo. */
+ pg_atomic_uint64 oldestXidWithEpochHavingUndo;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
1.8.3.1
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: Undo worker and transaction rollback
2018-11-05 11:40 Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-05 12:32 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-30 15:13 ` Re: Undo worker and transaction rollback Dmitry Dolgov <[email protected]>
2018-12-03 12:44 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-12-04 09:43 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
@ 2018-12-31 05:34 ` Dilip Kumar <[email protected]>
2019-01-11 04:09 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Dilip Kumar @ 2018-12-31 05:34 UTC (permalink / raw)
To: Dmitry Dolgov <[email protected]>; +Cc: pgsql-hackers
On Tue, Dec 4, 2018 at 3:13 PM Dilip Kumar <[email protected]> wrote:
After the latest change in undo interface patch[1], this patch need to
be rebased. Attaching the rebased version of the patch.
[1]https://www.postgresql.org/message-id/CAFiTN-tfquvm6tnWFaJNYYz-vSY%3D%2BSQTMv%2BFv1jPozTrW4mtKw%40ma...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] 0001-Undo-worker-and-transaction-rollback_v5.patch (134.9K, ../../CAFiTN-sGnEsz+VYfzqxoaHahP5WvXSsRyoORwkaZGn=7K-6mRQ@mail.gmail.com/2-0001-Undo-worker-and-transaction-rollback_v5.patch)
download | inline diff:
From 78a3d422773bdbeb0e08c66817698659843e3c79 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <[email protected]>
Date: Sun, 30 Dec 2018 13:36:59 +0530
Subject: [PATCH] Undo worker and transaction rollback
Patch provides mechanism for discarding the older undo when they
are no longer relavent. It also providea a mechanism for rolling
back the effect of the aborted transactiona. It handles the rollbacks
directly from backends as well as from undo worker for the larger
transaction.
Dilip Kumar, with help from Rafia Sabih, Amit Kapila, Mithun.CY, Thomas Munro and Kuntal Ghosh
---
src/backend/access/rmgrdesc/Makefile | 3 +-
src/backend/access/rmgrdesc/undoactiondesc.c | 64 +++
src/backend/access/rmgrdesc/xlogdesc.c | 4 +-
src/backend/access/transam/rmgr.c | 5 +-
src/backend/access/transam/twophase.c | 45 +-
src/backend/access/transam/varsup.c | 12 +
src/backend/access/transam/xact.c | 441 ++++++++++++++++-
src/backend/access/transam/xlog.c | 20 +
src/backend/access/undo/Makefile | 2 +-
src/backend/access/undo/undoaction.c | 688 ++++++++++++++++++++++++++
src/backend/access/undo/undoactionxlog.c | 49 ++
src/backend/access/undo/undodiscard.c | 461 +++++++++++++++++
src/backend/access/undo/undoinsert.c | 52 +-
src/backend/access/undo/undorecord.c | 3 +
src/backend/commands/vacuum.c | 11 +
src/backend/postmaster/Makefile | 4 +-
src/backend/postmaster/bgworker.c | 11 +
src/backend/postmaster/discardworker.c | 170 +++++++
src/backend/postmaster/pgstat.c | 7 +-
src/backend/postmaster/postmaster.c | 7 +
src/backend/postmaster/undoworker.c | 664 +++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 4 +
src/backend/storage/ipc/ipci.c | 6 +
src/backend/storage/lmgr/lwlocknames.txt | 2 +
src/backend/storage/lmgr/proc.c | 2 +
src/backend/utils/adt/lockfuncs.c | 1 +
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc.c | 11 +
src/backend/utils/misc/pg_controldata.c | 9 +-
src/backend/utils/misc/postgresql.conf.sample | 7 +
src/bin/pg_controldata/pg_controldata.c | 2 +
src/bin/pg_resetwal/pg_resetwal.c | 7 +
src/bin/pg_rewind/parsexlog.c | 2 +-
src/bin/pg_waldump/rmgrdesc.c | 3 +-
src/include/access/rmgr.h | 2 +-
src/include/access/rmgrlist.h | 47 +-
src/include/access/transam.h | 4 +
src/include/access/twophase.h | 2 +-
src/include/access/undoaction.h | 28 ++
src/include/access/undoaction_xlog.h | 74 +++
src/include/access/undodiscard.h | 31 ++
src/include/access/undoinsert.h | 2 +
src/include/access/undorecord.h | 13 +
src/include/access/xact.h | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlog_internal.h | 9 +-
src/include/catalog/pg_control.h | 7 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +
src/include/postmaster/discardworker.h | 25 +
src/include/postmaster/undoloop.h | 89 ++++
src/include/postmaster/undoworker.h | 39 ++
src/include/storage/lock.h | 10 +
src/include/storage/proc.h | 2 +
54 files changed, 3127 insertions(+), 45 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/undoactiondesc.c
create mode 100644 src/backend/access/undo/undoaction.c
create mode 100644 src/backend/access/undo/undoactionxlog.c
create mode 100644 src/backend/access/undo/undodiscard.c
create mode 100644 src/backend/postmaster/discardworker.c
create mode 100644 src/backend/postmaster/undoworker.c
create mode 100644 src/include/access/undoaction.h
create mode 100644 src/include/access/undoaction_xlog.h
create mode 100644 src/include/access/undodiscard.h
create mode 100644 src/include/postmaster/discardworker.h
create mode 100644 src/include/postmaster/undoloop.h
create mode 100644 src/include/postmaster/undoworker.h
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index 91ad1ef..640d37f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = brindesc.o clogdesc.o committsdesc.o dbasedesc.o genericdesc.o \
gindesc.o gistdesc.o hashdesc.o heapdesc.o logicalmsgdesc.o \
mxactdesc.o nbtdesc.o relmapdesc.o replorigindesc.o seqdesc.o \
- smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undologdesc.o xactdesc.o xlogdesc.o
+ smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undoactiondesc.o \
+ undologdesc.o xactdesc.o xlogdesc.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/rmgrdesc/undoactiondesc.c b/src/backend/access/rmgrdesc/undoactiondesc.c
new file mode 100644
index 0000000..89343b8
--- /dev/null
+++ b/src/backend/access/rmgrdesc/undoactiondesc.c
@@ -0,0 +1,64 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactiondesc.c
+ * rmgr descriptor routines for access/undo/undoactionxlog.c
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/access/rmgrdesc/undoactiondesc.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+
+void
+undoaction_desc(StringInfo buf, XLogReaderState *record)
+{
+ char *rec = XLogRecGetData(record);
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ if (info == XLOG_UNDO_PAGE)
+ {
+ uint8 *flags = (uint8 *) rec;
+
+ appendStringInfo(buf, "page_contains_tpd_slot: %c ",
+ (*flags & XLU_PAGE_CONTAINS_TPD_SLOT) ? 'T' : 'F');
+ appendStringInfo(buf, "is_page_initialized: %c ",
+ (*flags & XLU_INIT_PAGE) ? 'T' : 'F');
+ if (*flags & XLU_PAGE_CONTAINS_TPD_SLOT)
+ {
+ xl_undoaction_page *xlrec =
+ (xl_undoaction_page *) ((char *) flags + sizeof(uint8));
+
+ appendStringInfo(buf, "urec_ptr %lu xid %u trans_slot_id %u",
+ xlrec->urec_ptr, xlrec->xid, xlrec->trans_slot_id);
+ }
+ }
+ else if (info == XLOG_UNDO_APPLY_PROGRESS)
+ {
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) rec;
+
+ appendStringInfo(buf, "urec_ptr %lu progress %u",
+ xlrec->urec_ptr, xlrec->progress);
+ }
+}
+
+const char *
+undoaction_identify(uint8 info)
+{
+ const char *id = NULL;
+
+ switch (info & ~XLR_INFO_MASK)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ id = "UNDO APPLY PROGRESS";
+ break;
+ }
+
+ return id;
+}
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 00741c7..987e39c 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -47,7 +47,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
"tli %u; prev tli %u; fpw %s; xid %u:%u; oid %u; multi %u; offset %u; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
- "oldest running xid %u; %s",
+ "oldest running xid %u; "
+ "oldest xid with epoch having undo " UINT64_FORMAT "; %s",
(uint32) (checkpoint->redo >> 32), (uint32) checkpoint->redo,
checkpoint->ThisTimeLineID,
checkpoint->PrevTimeLineID,
@@ -63,6 +64,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
checkpoint->oldestCommitTsXid,
checkpoint->newestCommitTsXid,
checkpoint->oldestActiveXid,
+ checkpoint->oldestXidWithEpochHavingUndo,
(info == XLOG_CHECKPOINT_SHUTDOWN) ? "shutdown" : "online");
}
else if (info == XLOG_NEXTOID)
diff --git a/src/backend/access/transam/rmgr.c b/src/backend/access/transam/rmgr.c
index 8b05374..6238240 100644
--- a/src/backend/access/transam/rmgr.c
+++ b/src/backend/access/transam/rmgr.c
@@ -18,6 +18,7 @@
#include "access/multixact.h"
#include "access/nbtxlog.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -31,8 +32,8 @@
#include "utils/relmapper.h"
/* must be kept in sync with RmgrData definition in xlog_internal.h */
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
- { name, redo, desc, identify, startup, cleanup, mask },
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
+ { name, redo, desc, identify, startup, cleanup, mask, undo, undo_desc },
const RmgrData RmgrTable[RM_MAX_ID + 1] = {
#include "access/rmgrlist.h"
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index e65dccc..2c640f9 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -93,6 +93,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/origin.h"
#include "replication/syncrep.h"
#include "replication/walsender.h"
@@ -915,6 +916,12 @@ typedef struct TwoPhaseFileHeader
uint16 gidlen; /* length of the GID - GID follows the header */
XLogRecPtr origin_lsn; /* lsn of this record at origin node */
TimestampTz origin_timestamp; /* time of prepare at origin node */
+ /*
+ * We need the locations of start and end undo record pointers when rollbacks
+ * are to be performed for prepared transactions using zheap relations.
+ */
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
} TwoPhaseFileHeader;
/*
@@ -989,7 +996,8 @@ save_state_data(const void *data, uint32 len)
* Initializes data structure and inserts the 2PC file header record.
*/
void
-StartPrepare(GlobalTransaction gxact)
+StartPrepare(GlobalTransaction gxact, UndoRecPtr *start_urec_ptr,
+ UndoRecPtr *end_urec_ptr)
{
PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
PGXACT *pgxact = &ProcGlobal->allPgXact[gxact->pgprocno];
@@ -1027,6 +1035,10 @@ StartPrepare(GlobalTransaction gxact)
&hdr.initfileinval);
hdr.gidlen = strlen(gxact->gid) + 1; /* Include '\0' */
+ /* save the start and end undo record pointers */
+ memcpy(hdr.start_urec_ptr, start_urec_ptr, sizeof(hdr.start_urec_ptr));
+ memcpy(hdr.end_urec_ptr, end_urec_ptr, sizeof(hdr.end_urec_ptr));
+
save_state_data(&hdr, sizeof(TwoPhaseFileHeader));
save_state_data(gxact->gid, hdr.gidlen);
@@ -1452,6 +1464,9 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
RelFileNode *delrels;
int ndelrels;
SharedInvalidationMessage *invalmsgs;
+ int i;
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
/*
* Validate the GID, and lock the GXACT to ensure that two backends do not
@@ -1489,6 +1504,34 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
invalmsgs = (SharedInvalidationMessage *) bufptr;
bufptr += MAXALIGN(hdr->ninvalmsgs * sizeof(SharedInvalidationMessage));
+ /* save the start and end undo record pointers */
+ memcpy(start_urec_ptr, hdr->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(end_urec_ptr, hdr->end_urec_ptr, sizeof(end_urec_ptr));
+
+ /*
+ * Perform undo actions, if there are undologs for this transaction.
+ * We need to perform undo actions while we are still in transaction.
+ * Never push rollbacks of temp tables to undo worker.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (end_urec_ptr[i] != InvalidUndoRecPtr && !isCommit)
+ {
+ bool result = false;
+ uint64 rollback_size = 0;
+
+ if (i != UNDO_TEMP)
+ rollback_size = end_urec_ptr[i] - start_urec_ptr[i];
+
+ if (rollback_size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(end_urec_ptr[i], start_urec_ptr[i], InvalidOid);
+
+ if (!result)
+ execute_undo_actions(end_urec_ptr[i], start_urec_ptr[i], true,
+ true, false);
+ }
+ }
+
/* compute latestXid among all children */
latestXid = TransactionIdLatest(xid, hdr->nsubxacts, children);
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index a5eb29e..31bea0e 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -284,10 +284,22 @@ SetTransactionIdLimit(TransactionId oldest_datfrozenxid, Oid oldest_datoid)
TransactionId xidStopLimit;
TransactionId xidWrapLimit;
TransactionId curXid;
+ TransactionId oldestXidHavingUndo;
Assert(TransactionIdIsNormal(oldest_datfrozenxid));
/*
+ * To determine the last safe xid that can be allocated, we need to
+ * consider oldestXidHavingUndo. The oldestXidHavingUndo will be only
+ * valid for zheap storage engine, so it won't impact any other storage
+ * engine.
+ */
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ oldest_datfrozenxid = Min(oldest_datfrozenxid, oldestXidHavingUndo);
+
+ /*
* The place where we actually get into deep trouble is halfway around
* from the oldest potentially-existing XID. (This calculation is
* probably off by one or two counts, because the special XIDs reduce the
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 6060013..66d1685 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -42,6 +42,7 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/origin.h"
@@ -280,6 +281,20 @@ typedef struct SubXactCallbackItem
static SubXactCallbackItem *SubXact_callbacks = NULL;
+/* Location in undo log from where to start applying the undo actions. */
+static UndoRecPtr UndoActionStartPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+/* Location in undo log up to which undo actions need to be applied. */
+static UndoRecPtr UndoActionEndPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+/* Do we need to perform any undo actions? */
+static bool PerformUndoActions = false;
/* local function prototypes */
static void AssignTransactionId(TransactionState s);
@@ -1826,6 +1841,7 @@ StartTransaction(void)
{
TransactionState s;
VirtualTransactionId vxid;
+ int i;
/*
* Let's just make sure the state stack is empty
@@ -1904,6 +1920,13 @@ StartTransaction(void)
nUnreportedXids = 0;
s->didLogXid = false;
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
/*
* must initialize resource-management stuff first
*/
@@ -2213,7 +2236,7 @@ CommitTransaction(void)
* NB: if you change this routine, better look at CommitTransaction too!
*/
static void
-PrepareTransaction(void)
+PrepareTransaction(UndoRecPtr *start_urec_ptr, UndoRecPtr *end_urec_ptr)
{
TransactionState s = CurrentTransactionState;
TransactionId xid = GetCurrentTransactionId();
@@ -2361,7 +2384,7 @@ PrepareTransaction(void)
* PREPARED; in particular, pay attention to whether things should happen
* before or after releasing the transaction's locks.
*/
- StartPrepare(gxact);
+ StartPrepare(gxact, start_urec_ptr, end_urec_ptr);
AtPrepare_Notify();
AtPrepare_Locks();
@@ -2484,6 +2507,65 @@ PrepareTransaction(void)
RESUME_INTERRUPTS();
}
+static void
+AtAbort_Rollback(void)
+{
+ TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i;
+
+ /* XXX: TODO: check this logic, which was moved out of UserAbortTransactionBlock */
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (CurrentTransactionState->state == TRANS_INPROGRESS)
+ {
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (latest_urec_ptr[i])
+ {
+ if (i == UNDO_TEMP)
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ false, true, true);
+ else
+ {
+ uint64 size = latest_urec_ptr[i] - s->start_urec_ptr[i];
+ bool result = false;
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions
+ * later.
+ */
+ if (size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ true, true, true);
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
+}
/*
* AbortTransaction
@@ -2585,6 +2667,7 @@ AbortTransaction(void)
*/
AfterTriggerEndXact(false); /* 'false' means it's abort */
AtAbort_Portals();
+ AtAbort_Rollback();
AtEOXact_LargeObject(false);
AtAbort_Notify();
AtEOXact_RelationMap(false, is_parallel_worker);
@@ -2794,6 +2877,12 @@ void
CommitTransactionCommand(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i;
+
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(end_urec_ptr, s->latest_urec_ptr, sizeof(end_urec_ptr));
switch (s->blockState)
{
@@ -2883,7 +2972,7 @@ CommitTransactionCommand(void)
* return to the idle state.
*/
case TBLOCK_PREPARE:
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, end_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
break;
@@ -2929,6 +3018,23 @@ CommitTransactionCommand(void)
{
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
+
+ /*
+ * Update the end undo record pointer if it's not valid with
+ * the currently popped transaction's end undo record pointer.
+ * This is particularly required when the first command of
+ * the transaction is of type which does not require an undo,
+ * e.g. savepoint x.
+ * Accordingly, update the start undo record pointer.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(end_urec_ptr[i]))
+ end_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(s->start_urec_ptr[i]))
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
} while (s->blockState == TBLOCK_SUBCOMMIT);
/* If we had a COMMIT command, finish off the main xact too */
if (s->blockState == TBLOCK_END)
@@ -2940,7 +3046,7 @@ CommitTransactionCommand(void)
else if (s->blockState == TBLOCK_PREPARE)
{
Assert(s->parent == NULL);
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, end_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
}
else
@@ -3034,6 +3140,18 @@ void
AbortCurrentTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
+
+ /*
+ * The undo actions are allowed to be executed at the end of statement
+ * execution when we are not in transaction block, otherwise they are
+ * executed when user explicitly ends the transaction.
+ *
+ * So if we are in a transaction block don't set the PerformUndoActions
+ * because this flag will be set when user explicitly issue rollback or
+ * rollback to savepoint.
+ */
+ PerformUndoActions = false;
switch (s->blockState)
{
@@ -3068,6 +3186,16 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * We are outside the transaction block so remember the required
+ * information to perform undo actions and also set the
+ * PerformUndoActions so that we execute it before completing this
+ * command.
+ */
+ PerformUndoActions = true;
+ memcpy (UndoActionStartPtr, s->latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
break;
/*
@@ -3104,6 +3232,9 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3123,6 +3254,9 @@ AbortCurrentTransaction(void)
case TBLOCK_ABORT_END:
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3133,6 +3267,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Failed while executing the rollback command, need perform any
+ * pending undo actions.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3144,6 +3284,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Perform any pending actions if failed while preparing the
+ * transaction.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3166,6 +3312,17 @@ AbortCurrentTransaction(void)
case TBLOCK_SUBCOMMIT:
case TBLOCK_SUBABORT_PENDING:
case TBLOCK_SUBRESTART:
+ /*
+ * If we are here and still UndoActionStartPtr is valid that means
+ * the subtransaction failed while executing the undo action, so
+ * store its undo action start point in parent so that parent can
+ * start its undo action from this point.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ s->parent->latest_urec_ptr[i] = UndoActionStartPtr[i];
+ }
AbortSubTransaction();
CleanupSubTransaction();
AbortCurrentTransaction();
@@ -3183,6 +3340,109 @@ AbortCurrentTransaction(void)
}
/*
+ * XactPerformUndoActionsIfPending - Execute pending undo actions.
+ *
+ * If the parent transaction state is valid (when there is an error in the
+ * subtransaction and rollback to savepoint is executed), then allow to
+ * perform undo actions in it, otherwise perform them in a new transaction.
+ */
+void
+XactPerformUndoActionsIfPending()
+{
+ TransactionState s = CurrentTransactionState;
+ uint64 rollback_size = 0;
+ bool new_xact = true, result = false, no_pending_action = true;
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ if (!PerformUndoActions)
+ return;
+
+ /* If there is no undo log for any persistence level, then return. */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ {
+ no_pending_action = false;
+ break;
+ }
+ }
+
+ if (no_pending_action)
+ {
+ PerformUndoActions = false;
+ return;
+ }
+
+ /*
+ * Execute undo actions under parent transaction, if any. Otherwise start
+ * a new transaction.
+ */
+ if (GetTopTransactionIdIfAny() != InvalidTransactionId)
+ {
+ memcpy(parent_latest_urec_ptr, s->latest_urec_ptr,
+ sizeof (parent_latest_urec_ptr));
+ new_xact = false;
+ }
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions later.
+ * Never push the rollbacks for temp tables.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ continue;
+
+ if (i == UNDO_TEMP)
+ goto perform_rollback;
+ else
+ rollback_size = UndoActionStartPtr[i] - UndoActionEndPtr[i];
+
+ if (new_xact && rollback_size > rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+perform_rollback:
+ if (new_xact)
+ {
+ TransactionState xact;
+
+ /* Start a new transaction for performing the rollback */
+ StartTransactionCommand();
+ xact = CurrentTransactionState;
+
+ /*
+ * Store the previous transactions start and end undo record
+ * pointers into this transaction's state so that if there is
+ * some error while performing undo actions we can restart
+ * from begining.
+ */
+ memcpy(xact->start_urec_ptr, UndoActionEndPtr,
+ sizeof(UndoActionEndPtr));
+ memcpy(xact->latest_urec_ptr, UndoActionStartPtr,
+ sizeof(UndoActionStartPtr));
+ }
+
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ new_xact, true, true);
+
+ if (new_xact)
+ CommitTransactionCommand();
+ else
+ {
+ /* Restore parent's state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+ }
+
+ PerformUndoActions = false;
+}
+
+/*
* PreventInTransactionBlock
*
* This routine is to be called by statements that must not run inside
@@ -3583,6 +3843,10 @@ EndTransactionBlock(void)
{
TransactionState s = CurrentTransactionState;
bool result = false;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i ;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
switch (s->blockState)
{
@@ -3628,6 +3892,16 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+
+ /*
+ * We are calculating latest_urec_ptr, even though its a commit
+ * case. This is to handle any error during the commit path.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_END;
@@ -3653,6 +3927,11 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if(!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_ABORT_PENDING;
@@ -3705,6 +3984,18 @@ EndTransactionBlock(void)
break;
}
+ /*
+ * We need to perform undo actions if the transaction is failed. Remember
+ * the required information to perform undo actions at the end of
+ * statement execution.
+ */
+ if (!result)
+ PerformUndoActions = true;
+
+ memcpy(UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy(UndoActionEndPtr, TopTransactionStateData.start_urec_ptr,
+ sizeof(UndoActionEndPtr));
+
return result;
}
@@ -3962,6 +4253,12 @@ ReleaseSavepoint(const char *name)
TransactionState s = CurrentTransactionState;
TransactionState target,
xact;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4055,8 +4352,32 @@ ReleaseSavepoint(const char *name)
if (xact == target)
break;
xact = xact->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = xact->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(xact->start_urec_ptr[i]))
+ start_urec_ptr[i] = xact->start_urec_ptr[i];
+ }
+
Assert(PointerIsValid(xact));
}
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction. Also set the start_urec_ptr if
+ * parent start_urec_ptr is not valid.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ xact->parent->latest_urec_ptr[i] = latest_urec_ptr[i];
+ if (!UndoRecPtrIsValid(xact->parent->start_urec_ptr[i]))
+ xact->parent->start_urec_ptr[i] = start_urec_ptr[i];
+ }
}
/*
@@ -4267,6 +4588,7 @@ void
ReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4285,6 +4607,22 @@ ReleaseCurrentSubTransaction(void)
BlockStateAsString(s->blockState));
Assert(s->state == TRANS_INPROGRESS);
MemoryContextSwitchTo(CurTransactionContext);
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ s->parent->latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (!UndoRecPtrIsValid(s->parent->start_urec_ptr[i]))
+ s->parent->start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
Assert(s->state == TRANS_INPROGRESS);
@@ -4301,6 +4639,14 @@ void
RollbackAndReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i;
/*
* Unlike ReleaseCurrentSubTransaction(), this is nominally permitted
@@ -4347,6 +4693,19 @@ RollbackAndReleaseCurrentSubTransaction(void)
if (s->blockState == TBLOCK_SUBINPROGRESS)
AbortSubTransaction();
+ /*
+ * Remember the required information to perform undo actions before
+ * cleaning up the subtransaction state.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ {
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+ }
+
/* And clean it up, too */
CleanupSubTransaction();
@@ -4355,6 +4714,30 @@ RollbackAndReleaseCurrentSubTransaction(void)
s->blockState == TBLOCK_INPROGRESS ||
s->blockState == TBLOCK_IMPLICIT_INPROGRESS ||
s->blockState == TBLOCK_STARTED);
+
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ parent_latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ /*
+ * Store the undo action start point in the parent state so that
+ * we can apply undo actions these undos also during rollback of
+ * parent transaction in case of error while applying the undo
+ * actions.
+ */
+ s->latest_urec_ptr[i] = latest_urec_ptr[i];
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false,
+ true, true);
+
+ /* Restore parent state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+
+ /* Successfully performed undo actions so reset the flag. */
+ PerformUndoActions = false;
}
/*
@@ -4568,6 +4951,7 @@ static void
StartSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
if (s->state != TRANS_DEFAULT)
elog(WARNING, "StartSubTransaction while in %s state",
@@ -4585,6 +4969,13 @@ StartSubTransaction(void)
AtSubStart_Notify();
AfterTriggerBeginSubXact();
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
s->state = TRANS_INPROGRESS;
/*
@@ -4708,6 +5099,47 @@ CommitSubTransaction(void)
PopTransaction();
}
+static void
+AtSubAbort_Rollback(TransactionState s)
+{
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ /* XXX: TODO: Check this logic, which was moved out of RollbackToSavepoint() */
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (s->state == TRANS_INPROGRESS)
+ {
+ for ( i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false, true, false);
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
+}
+
/*
* AbortSubTransaction
*/
@@ -4802,6 +5234,7 @@ AbortSubTransaction(void)
s->parent->subTransactionId,
s->curTransactionOwner,
s->parent->curTransactionOwner);
+ AtSubAbort_Rollback(s);
AtEOSubXact_LargeObject(false, s->subTransactionId,
s->parent->subTransactionId);
AtSubAbort_Notify();
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 01815a6..e575322 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -5170,6 +5170,7 @@ BootStrapXLOG(void)
checkPoint.newestCommitTsXid = InvalidTransactionId;
checkPoint.time = (pg_time_t) time(NULL);
checkPoint.oldestActiveXid = InvalidTransactionId;
+ checkPoint.oldestXidWithEpochHavingUndo = InvalidTransactionId;
ShmemVariableCache->nextXid = checkPoint.nextXid;
ShmemVariableCache->nextOid = checkPoint.nextOid;
@@ -6604,6 +6605,10 @@ StartupXLOG(void)
(errmsg_internal("commit timestamp Xid oldest/newest: %u/%u",
checkPoint.oldestCommitTsXid,
checkPoint.newestCommitTsXid)));
+ ereport(DEBUG1,
+ (errmsg_internal("oldest xid with epoch having undo: " UINT64_FORMAT,
+ checkPoint.oldestXidWithEpochHavingUndo)));
+
if (!TransactionIdIsNormal(checkPoint.nextXid))
ereport(PANIC,
(errmsg("invalid next transaction ID")));
@@ -6621,6 +6626,10 @@ StartupXLOG(void)
XLogCtl->ckptXidEpoch = checkPoint.nextXidEpoch;
XLogCtl->ckptXid = checkPoint.nextXid;
+ /* Read oldest xid having undo from checkpoint and set in proc global. */
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* Initialize replication slots, before there's a chance to remove
* required resources.
@@ -8792,6 +8801,9 @@ CreateCheckPoint(int flags)
checkPoint.nextOid += ShmemVariableCache->oidCount;
LWLockRelease(OidGenLock);
+ checkPoint.oldestXidWithEpochHavingUndo =
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo);
+
MultiXactGetCheckptMulti(shutdown,
&checkPoint.nextMulti,
&checkPoint.nextMultiOffset,
@@ -9702,6 +9714,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceOldest(checkPoint.oldestMulti,
checkPoint.oldestMultiDB);
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* No need to set oldestClogXid here as well; it'll be set when we
* redo an xl_clog_truncate if it changed since initialization.
@@ -9760,6 +9775,8 @@ xlog_redo(XLogReaderState *record)
/* ControlFile->checkPointCopy always tracks the latest ckpt XID */
ControlFile->checkPointCopy.nextXidEpoch = checkPoint.nextXidEpoch;
ControlFile->checkPointCopy.nextXid = checkPoint.nextXid;
+ ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo =
+ checkPoint.oldestXidWithEpochHavingUndo;
/* Update shared-memory copy of checkpoint XID/epoch */
SpinLockAcquire(&XLogCtl->info_lck);
@@ -9809,6 +9826,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceNextMXact(checkPoint.nextMulti,
checkPoint.nextMultiOffset);
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ checkPoint.oldestXidWithEpochHavingUndo);
+
/*
* NB: This may perform multixact truncation when replaying WAL
* generated by an older primary.
diff --git a/src/backend/access/undo/Makefile b/src/backend/access/undo/Makefile
index f41e8f7..fdf7f7d 100644
--- a/src/backend/access/undo/Makefile
+++ b/src/backend/access/undo/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/access/undo
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = undoinsert.o undolog.o undorecord.o
+OBJS = undoaction.o undoactionxlog.o undodiscard.o undoinsert.o undolog.o undorecord.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/undo/undoaction.c b/src/backend/access/undo/undoaction.c
new file mode 100644
index 0000000..230c12a
--- /dev/null
+++ b/src/backend/access/undo/undoaction.c
@@ -0,0 +1,688 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.c
+ * execute undo actions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undoaction.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/undoaction.h"
+#include "access/undoaction_xlog.h"
+#include "access/undolog.h"
+#include "access/undorecord.h"
+#include "access/visibilitymap.h"
+#include "access/xact.h"
+#include "access/xloginsert.h"
+#include "access/xlog_internal.h"
+#include "nodes/pg_list.h"
+#include "pgstat.h"
+#include "postmaster/undoloop.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "utils/rel.h"
+#include "utils/relfilenodemap.h"
+#include "miscadmin.h"
+#include "storage/shmem.h"
+#include "access/undodiscard.h"
+
+#define ROLLBACK_HT_SIZE 1024
+
+static bool execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr,
+ Oid reloid, TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool norellock, int options);
+static void RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr);
+
+/* This is the hash table to store all the rollabck requests. */
+static HTAB *RollbackHT;
+
+/*
+ * execute_undo_actions - Execute the undo actions
+ *
+ * from_urecptr - undo record pointer from where to start applying undo action.
+ * to_urecptr - undo record pointer upto which point apply undo action.
+ * nopartial - true if rollback is for complete transaction.
+ * rewind - whether to rewind the insert location of the undo log or not.
+ * Only the backend executed the transaction can rewind, but
+ * any other process e.g. undo worker should not rewind it.
+ * Because, if the backend have already inserted new undo records
+ * for the next transaction and if we rewind then we will loose
+ * the undo record inserted for the new transaction.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ */
+void
+execute_undo_actions(UndoRecPtr from_urecptr, UndoRecPtr to_urecptr,
+ bool nopartial, bool rewind, bool rellock)
+{
+ UnpackedUndoRecord *uur = NULL;
+ UndoRecPtr urec_ptr, prev_urec_ptr;
+ UndoRecPtr save_urec_ptr;
+ Oid prev_reloid = InvalidOid;
+ ForkNumber prev_fork = InvalidForkNumber;
+ BlockNumber prev_block = InvalidBlockNumber;
+ List *luinfo = NIL;
+ bool more_undo;
+ int options = 0;
+ TransactionId xid = InvalidTransactionId;
+ UndoRecInfo *urec_info;
+
+ Assert(from_urecptr != InvalidUndoRecPtr);
+ Assert(UndoRecPtrGetLogNo(from_urecptr) != UndoRecPtrGetLogNo(to_urecptr) ||
+ from_urecptr >= to_urecptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ * FIXME: this won't work if undolog crossed the limit of 1TB, because
+ * then from_urecptr and to_urecptr will be from different lognos.
+ */
+ if (to_urecptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(from_urecptr);
+ to_urecptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ save_urec_ptr = urec_ptr = from_urecptr;
+
+ if (nopartial)
+ {
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber, InvalidOffsetNumber,
+ InvalidTransactionId, NULL, NULL);
+ if (uur == NULL)
+ return;
+
+ xid = uur->uur_xid;
+ UndoRecordRelease(uur);
+ uur = NULL;
+
+ /*
+ * Grab the undo action apply lock before start applying the undo action
+ * this will prevent applying undo actions concurrently. If we do not
+ * get the lock that mean its already being applied concurrently or the
+ * discard worker might be pushing its request to the rollback hash
+ * table
+ */
+ if (!ConditionTransactionUndoActionLock(xid))
+ return;
+ }
+
+ prev_urec_ptr = InvalidUndoRecPtr;
+ while (prev_urec_ptr != to_urecptr)
+ {
+ Oid reloid = InvalidOid;
+ uint16 urec_prevlen;
+ UndoRecPtr urec_prevurp;
+ bool non_page;
+
+ more_undo = true;
+
+ prev_urec_ptr = urec_ptr;
+
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId, NULL, NULL);
+
+ /* If there is no info block, this is not a page-based undo record. */
+ non_page = uur && !(uur->uur_info & UREC_INFO_BLOCK);
+
+ if (uur != NULL && !non_page)
+ reloid = uur->uur_reloid;
+
+ xid = uur->uur_xid;
+
+ if (non_page)
+ {
+ prev_reloid = InvalidOid;
+ urec_prevlen = uur->uur_prevlen;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /*
+ * Execute individual undo actions not associated with a page
+ * immediately.
+ */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->uur = uur;
+ urec_info->urp = urec_ptr;
+ luinfo = lappend(luinfo, urec_info);
+ execute_undo_actions_page(luinfo, urec_ptr, reloid, xid,
+ InvalidBlockNumber, false, rellock,
+ 0);
+ pfree(urec_info);
+ urec_info = NULL;
+ list_free(luinfo);
+ luinfo = NIL;
+ UndoRecordRelease(uur);
+
+ /* Follow undo chain until to_urecptr. */
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ /*
+ * If the record is already discarded by undo worker or if the relation
+ * is dropped or truncated, then we cannot fetch record successfully.
+ * Hence, skip quietly.
+ *
+ * Note: reloid remains InvalidOid for a discarded record.
+ */
+ else if (!OidIsValid(reloid))
+ {
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ urec_prevlen = uur->uur_prevlen;
+
+ /* Release the just-fetched record */
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else if (!OidIsValid(prev_reloid) ||
+ (prev_reloid == reloid &&
+ prev_fork == uur->uur_fork &&
+ prev_block == uur->uur_block))
+ {
+ /* Collect the undo records that belong to the same page. */
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+
+ luinfo = lappend(luinfo, urec_info);
+ urec_prevlen = uur->uur_prevlen;
+ urec_prevurp = uur->uur_prevurp;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else
+ {
+ more_undo = true;
+ }
+
+ /*
+ * If no more undo is left to be processed and we are rolling back the
+ * complete transaction, then we can consider that the undo chain for a
+ * block is complete.
+ * If the previous undo pointer in the page is invalid, then also the
+ * undo chain for the current block is completed.
+ */
+ if (luinfo &&
+ ((!more_undo && nopartial) || !UndoRecPtrIsValid(save_urec_ptr)))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, true, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+ else if (luinfo)
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, false, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ /*
+ * There are still more records to process, so keep moving backwards
+ * in the chain.
+ */
+ if (more_undo)
+ {
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+ luinfo = lappend(luinfo, urec_info);
+
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /*
+ * Continue to process the records if this is not the last undo
+ * record in chain.
+ */
+ urec_prevlen = uur->uur_prevlen;
+ urec_prevurp = uur->uur_prevurp;
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ else
+ break;
+ }
+ else
+ break;
+ }
+
+ /* Apply the undo actions for the remaining records. */
+ if (list_length(luinfo))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, nopartial ? true : false,
+ rellock, options);
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+ }
+
+ if (rewind)
+ {
+ /* Read the prevlen from the first record of this transaction. */
+ uur = UndoFetchRecord(to_urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If undo is already discarded before we rewind, then do nothing.
+ */
+ if (uur == NULL)
+ return;
+
+
+ /*
+ * Rewind the insert location to start of this transaction. This is
+ * to avoid reapplying some intermediate undo. We do not need to wal
+ * log this information here, because if the system crash before we
+ * rewind the insert pointer then after recovery we can identify
+ * whether the undo is already applied or not from the slot undo record
+ * pointer. Also set the correct prevlen value (what we have fetched
+ * from the undo).
+ */
+ UndoLogRewind(to_urecptr, uur->uur_prevlen);
+
+ UndoRecordRelease(uur);
+ }
+
+ if (nopartial)
+ {
+ /*
+ * Set undo action apply completed in the transaction header if this is
+ * a main transaction and we have not rewound its undo.
+ */
+ if (!rewind)
+ {
+ /*
+ * Undo action is applied so delete the hash table entry and release
+ * the undo action lock.
+ */
+ RollbackHTRemoveEntry(from_urecptr);
+
+ /*
+ * Prepare and update the progress of the undo action apply in the
+ * transaction header.
+ */
+ PrepareUpdateUndoActionProgress(to_urecptr, 1);
+
+ START_CRIT_SECTION();
+
+ /* Update the progress in the transaction header. */
+ UndoRecordUpdateTransInfo(0);
+
+ /* WAL log the undo apply progress. */
+ {
+ xl_undoapply_progress xlrec;
+
+ xlrec.urec_ptr = to_urecptr;
+ xlrec.progress = 1;
+
+ /*
+ * FIXME : We need to register undo buffers and set LSN for them
+ * that will be required for FPW of the undo buffers.
+ */
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, sizeof(xlrec));
+
+ (void) XLogInsert(RM_UNDOACTION_ID, XLOG_UNDO_APPLY_PROGRESS);
+ }
+
+ END_CRIT_SECTION();
+ UnlockReleaseUndoBuffers();
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+}
+
+/*
+ * execute_undo_actions_page - Execute the undo actions for a page
+ *
+ * After applying all the undo actions for a page, we clear the transaction
+ * slot on a page if the undo chain for block is complete, otherwise just
+ * rewind the undo pointer to the last record for that block that precedes
+ * the last undo record for which action is replayed.
+ *
+ * luinfo - list of undo records (along with their location) for which undo
+ * action needs to be replayed.
+ * urec_ptr - undo record pointer to which we need to rewind.
+ * reloid - OID of relation on which undo actions needs to be applied.
+ * blkno - block number on which undo actions needs to be applied.
+ * blk_chain_complete - indicates whether the undo chain for block is
+ * complete.
+ * nopartial - true if rollback is for complete transaction. If we are not
+ * rolling back the complete transaction then we need to apply the
+ * undo action for UNDO_INVALID_XACT_SLOT also because in such
+ * case we will rewind the insert undo location.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ * options - options for executing undo actions.
+ *
+ * returns true, if successfully applied the undo actions, otherwise, false.
+ */
+static bool
+execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock, int options)
+{
+ UndoRecInfo *first;
+
+ /*
+ * All records passed to us are for the same RMGR, so we just use the
+ * first record to dispatch.
+ */
+ Assert(luinfo != NIL);
+ first = (UndoRecInfo *) linitial(luinfo);
+
+ return RmgrTable[first->uur->uur_rmid].rm_undo(luinfo, urec_ptr, reloid,
+ xid, blkno,
+ blk_chain_complete, rellock,
+ options);
+}
+
+/*
+ * To return the size of the hash-table for rollbacks.
+ */
+int
+RollbackHTSize(void)
+{
+ return hash_estimate_size(ROLLBACK_HT_SIZE, sizeof(RollbackHashEntry));
+}
+
+/*
+ * To initialize the hash-table for rollbacks in shared memory
+ * for the given size.
+ */
+void
+InitRollbackHashTable(void)
+{
+ HASHCTL info;
+ MemSet(&info, 0, sizeof(info));
+
+ info.keysize = sizeof(UndoRecPtr);
+ info.entrysize = sizeof(RollbackHashEntry);
+ info.hash = tag_hash;
+
+ RollbackHT = ShmemInitHash("Undo actions Lookup Table",
+ ROLLBACK_HT_SIZE, ROLLBACK_HT_SIZE, &info,
+ HASH_ELEM | HASH_FUNCTION | HASH_FIXED_SIZE);
+}
+
+/*
+ * To push the rollback requests from backend to the hash-table.
+ * Return true if the request is successfully added, else false
+ * and the caller may execute undo actions itself.
+ */
+bool
+PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr, Oid dbid)
+{
+ bool found = false;
+ RollbackHashEntry *rh;
+
+ Assert(UndoRecPtrGetLogNo(start_urec_ptr) != UndoRecPtrGetLogNo(end_urec_ptr) ||
+ start_urec_ptr >= end_urec_ptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ */
+ if (start_urec_ptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(end_urec_ptr);
+ start_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ Assert(UndoRecPtrIsValid(start_urec_ptr));
+
+ /* If there is no space to accomodate new request, then we can't proceed. */
+ if (RollbackHTIsFull())
+ return false;
+
+ if(!UndoRecPtrIsValid(end_urec_ptr))
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(start_urec_ptr);
+ end_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ rh = (RollbackHashEntry *) hash_search(RollbackHT, &start_urec_ptr,
+ HASH_ENTER_NULL, &found);
+ if (!rh)
+ {
+ LWLockRelease(RollbackHTLock);
+ return false;
+ }
+ /* We shouldn't try to push the same rollback request again. */
+ if (!found)
+ {
+ rh->start_urec_ptr = start_urec_ptr;
+ rh->end_urec_ptr = end_urec_ptr;
+ rh->dbid = (dbid == InvalidOid) ? MyDatabaseId : dbid;
+ }
+ LWLockRelease(RollbackHTLock);
+
+ return true;
+}
+
+/*
+ * To perform the undo actions for the transactions whose rollback
+ * requests are in hash table. Sequentially, scan the hash-table
+ * and perform the undo-actions for the respective transactions.
+ * Once, the undo-actions are applied, remove the entry from the
+ * hash table.
+ */
+void
+RollbackFromHT(Oid dbid)
+{
+ UndoRecPtr start[ROLLBACK_HT_SIZE];
+ UndoRecPtr end[ROLLBACK_HT_SIZE];
+ RollbackHashEntry *rh;
+ HASH_SEQ_STATUS status;
+ int i = 0;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ Assert(hash_get_num_entries(RollbackHT) <= ROLLBACK_HT_SIZE);
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ {
+ if (rh->dbid == dbid)
+ {
+ start[i] = rh->start_urec_ptr;
+ end[i] = rh->end_urec_ptr;
+ i++;
+ }
+ }
+
+ LWLockRelease(RollbackHTLock);
+
+ /* Execute the rollback requests */
+ while(--i >= 0)
+ {
+ Assert(UndoRecPtrIsValid(start[i]));
+ Assert(UndoRecPtrIsValid(end[i]));
+
+ StartTransactionCommand();
+ execute_undo_actions(start[i], end[i], true, false, true);
+ CommitTransactionCommand();
+ }
+}
+
+/*
+ * Remove the rollback request entry from the rollback hash table.
+ */
+static void
+RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr)
+{
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ hash_search(RollbackHT, &start_urec_ptr, HASH_REMOVE, NULL);
+
+ LWLockRelease(RollbackHTLock);
+}
+
+/*
+ * To check if the rollback requests in the hash table are all
+ * completed or not. This is required because we don't not want to
+ * expose RollbackHT in xact.c, where it is required to ensure
+ * that we push the resuests only when there is some space in
+ * the hash-table.
+ */
+bool
+RollbackHTIsFull(void)
+{
+ bool result = false;
+
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ if (hash_get_num_entries(RollbackHT) >= ROLLBACK_HT_SIZE)
+ result = true;
+
+ LWLockRelease(RollbackHTLock);
+
+ return result;
+}
+
+/*
+ * Get database list from the rollback hash table.
+ */
+List *
+RollbackHTGetDBList()
+{
+ HASH_SEQ_STATUS status;
+ RollbackHashEntry *rh;
+ List *dblist = NIL;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ dblist = list_append_unique_oid(dblist, rh->dbid);
+
+ LWLockRelease(RollbackHTLock);
+
+ return dblist;
+}
+
+/*
+ * ConditionTransactionUndoActionLock
+ *
+ * Insert a lock showing that the undo action for given transaction is in
+ * progress. This is only done for the main transaction not for the
+ * sub-transaction.
+ */
+bool
+ConditionTransactionUndoActionLock(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ if (LOCKACQUIRE_NOT_AVAIL == LockAcquire(&tag, ExclusiveLock, false, true))
+ return false;
+ else
+ return true;
+}
+
+/*
+ * TransactionUndoActionLockRelease
+ *
+ * Delete the lock showing that the undo action given transaction ID is in
+ * progress.
+ */
+void
+TransactionUndoActionLockRelease(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ LockRelease(&tag, ExclusiveLock, false);
+}
diff --git a/src/backend/access/undo/undoactionxlog.c b/src/backend/access/undo/undoactionxlog.c
new file mode 100644
index 0000000..253d8f1
--- /dev/null
+++ b/src/backend/access/undo/undoactionxlog.c
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactionxlog.c
+ * WAL replay logic for undo actions.
+ *
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/undo/undoactionxlog.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+#include "access/undoinsert.h"
+#include "access/visibilitymap.h"
+#include "access/xlog.h"
+#include "access/xlogutils.h"
+
+/*
+ * Replay of undo apply progress.
+ */
+static void
+undo_xlog_apply_progress(XLogReaderState *record)
+{
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) XLogRecGetData(record);
+
+ /* Update the progress in the transaction header. */
+ PrepareUpdateUndoActionProgress(xlrec->urec_ptr, xlrec->progress);
+ UndoRecordUpdateTransInfo(0);
+}
+
+void
+undoaction_redo(XLogReaderState *record)
+{
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ switch (info)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ undo_xlog_apply_progress(record);
+ break;
+ default:
+ elog(PANIC, "undoaction_redo: unknown op code %u", info);
+ }
+}
diff --git a/src/backend/access/undo/undodiscard.c b/src/backend/access/undo/undodiscard.c
new file mode 100644
index 0000000..847abe5
--- /dev/null
+++ b/src/backend/access/undo/undodiscard.c
@@ -0,0 +1,461 @@
+/*-------------------------------------------------------------------------
+ *
+ * undodiscard.c
+ * discard undo records
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undodiscard.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/xlog.h"
+#include "access/xact.h"
+#include "access/undolog.h"
+#include "access/undodiscard.h"
+#include "catalog/pg_tablespace.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "storage/shmem.h"
+#include "storage/proc.h"
+#include "utils/resowner.h"
+#include "postmaster/undoloop.h"
+
+static UndoRecPtr FetchLatestUndoPtrForXid(UndoRecPtr urecptr,
+ UnpackedUndoRecord *uur_start,
+ UndoLogControl *log);
+
+/*
+ * Discard the undo for the log
+ *
+ * Search the undo log, get the start record for each transaction until we get
+ * the transaction with xid >= xmin or an invalid xid. Then call undolog
+ * routine to discard upto that point and update the memory structure for the
+ * log slot. We set the hibernate flag if we do not have any undo logs, this
+ * flag is passed to the undo worker wherein it determines if system is idle
+ * and it should sleep for sometime.
+ *
+ * Return the oldest xid remaining in this undo log (which should be >= xmin,
+ * since we'll discard everything older). Return InvalidTransactionId if the
+ * undo log is empty.
+ */
+static TransactionId
+UndoDiscardOneLog(UndoLogControl *log, TransactionId xmin, bool *hibernate)
+{
+ UndoRecPtr undo_recptr, next_insert, from_urecptr;
+ UndoRecPtr next_urecptr = InvalidUndoRecPtr;
+ UnpackedUndoRecord *uur = NULL;
+ bool need_discard = false;
+ bool log_complete = false;
+ TransactionId undoxid = InvalidTransactionId;
+ TransactionId xid = log->oldest_xid;
+ TransactionId latest_discardxid = InvalidTransactionId;
+ uint32 epoch = 0;
+
+ undo_recptr = log->oldest_data;
+
+ /* There might not be any undo log and hibernation might be needed. */
+ *hibernate = true;
+
+ /* Loop until we run out of discardable transactions. */
+ do
+ {
+ bool pending_abort = false;
+
+ next_insert = UndoLogGetNextInsertPtr(log->logno, xid);
+
+ /*
+ * If the next insert location in the undo log is same as the oldest
+ * data for the log then there is nothing more to discard in this log
+ * so discard upto this point.
+ */
+ if (next_insert == undo_recptr)
+ {
+ /*
+ * If the discard location and the insert location is same then
+ * there is nothing to discard.
+ */
+ if (undo_recptr == log->oldest_data)
+ break;
+ else
+ log_complete = true;
+ }
+ else
+ {
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+
+ Assert(uur != NULL);
+
+ if (!TransactionIdDidCommit(uur->uur_xid) &&
+ TransactionIdPrecedes(uur->uur_xid, xmin) &&
+ uur->uur_progress == 0)
+ {
+ /*
+ * At the time of recovery, we might not have a valid next undo
+ * record pointer and in that case we'll calculate the location
+ * of from pointer using the last record of next insert
+ * location.
+ */
+ if (ConditionTransactionUndoActionLock(uur->uur_xid))
+ {
+ TransactionId xid = uur->uur_xid;
+ UndoLogControl *log = NULL;
+ UndoLogNumber logno;
+
+ logno = UndoRecPtrGetLogNo(undo_recptr);
+ log = UndoLogGet(logno, false);
+
+ /*
+ * If the corresponding log got rewinded to a location
+ * prior to undo_recptr, the undo actions are already
+ * applied.
+ */
+ if (log->meta.insert > undo_recptr)
+ {
+ UndoRecordRelease(uur);
+
+ /* Fetch the undo record under undo action lock. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the undo actions for the aborted transaction is
+ * already applied then continue discarding the undo log
+ * otherwise discard till current point and stop processing
+ * this undo log.
+ * Also, check this is indeed the transaction id we're
+ * looking for. It is possible that after rewinding
+ * some other transaction has inserted an undo record.
+ */
+ if (uur->uur_xid == xid && uur->uur_progress == 0)
+ {
+ from_urecptr = FetchLatestUndoPtrForXid(undo_recptr, uur, log);
+ (void)PushRollbackReq(from_urecptr, undo_recptr, uur->uur_dbid);
+ pending_abort = true;
+ }
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+ else
+ pending_abort = true;
+ }
+
+ next_urecptr = uur->uur_next;
+ undoxid = uur->uur_xid;
+ xid = undoxid;
+ epoch = uur->uur_xidepoch;
+ }
+
+ /* we can discard upto this point. */
+ if (TransactionIdFollowsOrEquals(undoxid, xmin) ||
+ next_urecptr == SpecialUndoRecPtr ||
+ UndoRecPtrGetLogNo(next_urecptr) != log->logno ||
+ log_complete || pending_abort)
+ {
+ /* Hey, I got some undo log to discard, can not hibernate now. */
+ *hibernate = false;
+
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /*
+ * If Transaction id is smaller than the xmin that means this must
+ * be the last transaction in this undo log, so we need to get the
+ * last insert point in this undo log and discard till that point.
+ * Also, if the transaction has pending abort, we stop discarding
+ * undo from the same location.
+ */
+ if (TransactionIdPrecedes(undoxid, xmin) && !pending_abort)
+ {
+ UndoRecPtr next_insert = InvalidUndoRecPtr;
+
+ /*
+ * Get the last insert location for this transaction Id, if it
+ * returns invalid pointer that means there is new transaction
+ * has started for this undolog. So we need to refetch the undo
+ * and continue the process.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, undoxid);
+ if (!UndoRecPtrIsValid(next_insert))
+ continue;
+
+ undo_recptr = next_insert;
+ need_discard = true;
+ epoch = 0;
+ latest_discardxid = undoxid;
+ undoxid = InvalidTransactionId;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_EXCLUSIVE);
+
+ /*
+ * If no more pending undo logs then set the oldest transaction to
+ * InvalidTransactionId.
+ */
+ if (log_complete)
+ {
+ log->oldest_xid = InvalidTransactionId;
+ log->oldest_xidepoch = 0;
+ }
+ else
+ {
+ log->oldest_xid = undoxid;
+ log->oldest_xidepoch = epoch;
+ }
+
+ log->oldest_data = undo_recptr;
+ LWLockRelease(&log->discard_lock);
+
+ if (need_discard)
+ UndoLogDiscard(undo_recptr, latest_discardxid);
+
+ break;
+ }
+
+ /*
+ * This transaction is smaller than the xmin so lets jump to the next
+ * transaction.
+ */
+ undo_recptr = next_urecptr;
+ latest_discardxid = undoxid;
+
+ if(uur != NULL)
+ {
+ UndoRecordRelease(uur);
+ uur = NULL;
+ }
+
+ need_discard = true;
+ } while (true);
+
+ return undoxid;
+}
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+void
+UndoDiscard(TransactionId oldestXmin, bool *hibernate)
+{
+ TransactionId oldestXidHavingUndo = oldestXmin;
+ uint64 epoch = GetEpochForXid(oldestXmin);
+ UndoLogControl *log = NULL;
+
+ /*
+ * TODO: Ideally we'd arrange undo logs so that we can efficiently find
+ * those with oldest_xid < oldestXmin, but for now we'll just scan all of
+ * them.
+ */
+ while ((log = UndoLogNext(log)))
+ {
+ TransactionId oldest_xid = InvalidTransactionId;
+
+ /* We can't process temporary undo logs. */
+ if (log->meta.persistence == UNDO_TEMP)
+ continue;
+
+ /*
+ * If the first xid of the undo log is smaller than the xmin the try
+ * to discard the undo log.
+ */
+ if (TransactionIdPrecedes(log->oldest_xid, oldestXmin))
+ {
+ /*
+ * If the XID in the discard entry is invalid then start scanning
+ * from the first valid undorecord in the log.
+ */
+ if (!TransactionIdIsValid(log->oldest_xid))
+ {
+ bool full = false;
+ UndoRecPtr urp = UndoLogGetFirstValidRecord(log, &full);
+
+ if (!UndoRecPtrIsValid(urp))
+ {
+ /*
+ * There is nothing to be discarded. If there is also no
+ * more free space, then a call to UndoLogDiscard() will
+ * discard it the undo log completely and free up the
+ * UndoLogControl slot.
+ */
+ if (full)
+ UndoLogDiscard(MakeUndoRecPtr(log->meta.logno,
+ log->meta.discard),
+ InvalidTransactionId);
+ continue;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_SHARED);
+ log->oldest_data = urp;
+ LWLockRelease(&log->discard_lock);
+ }
+
+ /* Process the undo log. */
+ oldest_xid = UndoDiscardOneLog(log, oldestXmin, hibernate);
+ }
+
+ if (TransactionIdIsValid(oldest_xid) &&
+ TransactionIdPrecedes(oldest_xid, oldestXidHavingUndo))
+ {
+ oldestXidHavingUndo = oldest_xid;
+ epoch = GetEpochForXid(oldest_xid);
+ }
+ }
+
+ /*
+ * Update the oldestXidWithEpochHavingUndo in the shared memory.
+ *
+ * XXX In future if multiple worker can perform discard then we may need
+ * to use compare and swap for updating the shared memory value.
+ */
+ pg_atomic_write_u64(&ProcGlobal->oldestXidWithEpochHavingUndo,
+ MakeEpochXid(epoch, oldestXidHavingUndo));
+}
+
+/*
+ * Fetch the latest urec pointer for the transaction.
+ */
+UndoRecPtr
+FetchLatestUndoPtrForXid(UndoRecPtr urecptr, UnpackedUndoRecord *uur_start,
+ UndoLogControl *log)
+{
+ UndoRecPtr next_urecptr, from_urecptr;
+ uint16 prevlen;
+ UndoLogOffset next_insert;
+ UnpackedUndoRecord *uur;
+ bool refetch = false;
+
+ uur = uur_start;
+
+ while (true)
+ {
+ /* fetch the undo record again if required. */
+ if (refetch)
+ {
+ uur = UndoFetchRecord(urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ refetch = false;
+ }
+
+ next_urecptr = uur->uur_next;
+ prevlen = UndoLogGetPrevLen(log->logno);
+
+ /*
+ * If this is the last transaction in the log then calculate the latest
+ * urec pointer using next insert location of the undo log. Otherwise,
+ * calculate using next transaction's start pointer.
+ */
+ if (uur->uur_next == SpecialUndoRecPtr)
+ {
+ /*
+ * While fetching the next insert location if the new transaction
+ * has already started in this log then lets re-fetch the undo
+ * record.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+ if (!UndoRecPtrIsValid(next_insert))
+ {
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ refetch = true;
+ continue;
+ }
+
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen,
+ InvalidUndoRecPtr);
+ break;
+ }
+ else if ((UndoRecPtrGetLogNo(next_urecptr) != log->logno) &&
+ UndoLogIsDiscarded(next_urecptr))
+ {
+ /*
+ * If next_urecptr is in different undolog and its already discarded
+ * that means the undo actions for this transaction which are in the
+ * next log has already been executed and we only need to execute
+ * which are remaining in this log.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+
+ Assert(UndoRecPtrIsValid(next_insert));
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen,
+ InvalidUndoRecPtr);
+ break;
+ }
+ else
+ {
+ UnpackedUndoRecord *next_uur;
+
+ next_uur = UndoFetchRecord(next_urecptr,
+ InvalidBlockNumber,
+ InvalidOffsetNumber,
+ InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the next_urecptr is in the same log then calculate the
+ * from pointer using prevlen.
+ */
+ if (UndoRecPtrGetLogNo(next_urecptr) == log->logno)
+ {
+ from_urecptr =
+ UndoGetPrevUndoRecptr(next_urecptr, next_uur->uur_prevlen,
+ InvalidUndoRecPtr);
+ UndoRecordRelease(next_uur);
+ break;
+ }
+ else
+ {
+ /*
+ * The transaction is overflowed to the next log, so restart
+ * the processing from then next log.
+ */
+ log = UndoLogGet(UndoRecPtrGetLogNo(next_urecptr), false);
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ uur = next_uur;
+ continue;
+ }
+
+ UndoRecordRelease(next_uur);
+ }
+ }
+
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+
+ return from_urecptr;
+}
+
+/*
+ * Discard the undo logs for temp tables.
+ */
+void
+TempUndoDiscard(UndoLogNumber logno)
+{
+ UndoLogControl *log = UndoLogGet(logno, false);
+
+ /*
+ * Discard the undo log for temp table only. Ensure that there is
+ * something to be discarded there.
+ */
+ Assert (log->meta.persistence == UNDO_TEMP);
+
+ /* Process the undo log. */
+ UndoLogDiscard(MakeUndoRecPtr(log->logno, log->meta.insert),
+ InvalidTransactionId);
+}
diff --git a/src/backend/access/undo/undoinsert.c b/src/backend/access/undo/undoinsert.c
index 68e6adb..8781bc0 100644
--- a/src/backend/access/undo/undoinsert.c
+++ b/src/backend/access/undo/undoinsert.c
@@ -164,7 +164,6 @@ static UnpackedUndoRecord *UndoGetOneRecord(UnpackedUndoRecord *urec,
UndoPersistence persistence);
static void UndoRecordPrepareTransInfo(UndoRecPtr urecptr,
UndoRecPtr xact_urp);
-static void UndoRecordUpdateTransInfo(int idx);
static int UndoGetBufferSlot(RelFileNode rnode, BlockNumber blk,
ReadBufferMode rbm,
UndoPersistence persistence);
@@ -295,12 +294,61 @@ UndoRecordPrepareTransInfo(UndoRecPtr urecptr, UndoRecPtr xact_urp)
}
/*
+ * Update the progress of the undo record in the transaction header.
+ */
+void
+PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress)
+{
+ Buffer buffer = InvalidBuffer;
+ BlockNumber cur_blk;
+ RelFileNode rnode;
+ UndoLogNumber logno = UndoRecPtrGetLogNo(urecptr);
+ UndoLogControl *log;
+ Page page;
+ int already_decoded = 0;
+ int starting_byte;
+ int bufidx;
+ int index = 0;
+
+ log = UndoLogGet(logno, false);
+
+ if (log->meta.persistence == UNDO_TEMP)
+ return;
+
+ UndoRecPtrAssignRelFileNode(rnode, urecptr);
+ cur_blk = UndoRecPtrGetBlockNum(urecptr);
+ starting_byte = UndoRecPtrGetPageOffset(urecptr);
+
+ while (true)
+ {
+ bufidx = UndoGetBufferSlot(rnode, cur_blk,
+ RBM_NORMAL,
+ log->meta.persistence);
+
+ xact_urec_info[xact_urec_info_idx].idx_undo_buffers[index++] = bufidx;
+ buffer = undo_buffer[bufidx].buf;
+ page = BufferGetPage(buffer);
+
+ if (UnpackUndoRecord(&xact_urec_info[xact_urec_info_idx].uur, page, starting_byte,
+ &already_decoded, true))
+ break;
+
+ starting_byte = UndoLogBlockHeaderSize;
+ cur_blk++;
+ }
+
+ xact_urec_info[xact_urec_info_idx].urecptr = urecptr;
+ xact_urec_info[xact_urec_info_idx].uur.uur_progress = progress;
+ xact_urec_info_idx++;
+}
+
+/*
* Overwrite the first undo record of the previous transaction to update its
* next pointer. This will just insert the already prepared record by
* UndoRecordPrepareTransInfo. This must be called under the critical section.
* This will just overwrite the undo header not the data.
*/
-static void
+void
UndoRecordUpdateTransInfo(int idx)
{
UndoLogNumber logno = UndoRecPtrGetLogNo(xact_urec_info[idx].urecptr);
diff --git a/src/backend/access/undo/undorecord.c b/src/backend/access/undo/undorecord.c
index a13abe3..dcd8a29 100644
--- a/src/backend/access/undo/undorecord.c
+++ b/src/backend/access/undo/undorecord.c
@@ -100,6 +100,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
*/
if (*already_written == 0)
{
+ work_hdr.urec_rmid = uur->uur_rmid;
work_hdr.urec_type = uur->uur_type;
work_hdr.urec_info = uur->uur_info;
work_hdr.urec_prevlen = uur->uur_prevlen;
@@ -125,6 +126,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
* We should have been passed the same record descriptor as before, or
* caller has messed up.
*/
+ Assert(work_hdr.urec_rmid == uur->uur_rmid);
Assert(work_hdr.urec_type == uur->uur_type);
Assert(work_hdr.urec_info == uur->uur_info);
Assert(work_hdr.urec_prevlen == uur->uur_prevlen);
@@ -288,6 +290,7 @@ UnpackUndoRecord(UnpackedUndoRecord *uur, Page page, int starting_byte,
&my_bytes_decoded, already_decoded, false))
return false;
+ uur->uur_rmid = work_hdr.urec_rmid;
uur->uur_type = work_hdr.urec_type;
uur->uur_info = work_hdr.urec_info;
uur->uur_prevlen = work_hdr.urec_prevlen;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 25b3b03..8d98d3f 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1381,6 +1381,7 @@ vac_truncate_clog(TransactionId frozenXID,
MultiXactId lastSaneMinMulti)
{
TransactionId nextXID = ReadNewTransactionId();
+ TransactionId oldestXidHavingUndo;
Relation relation;
HeapScanDesc scan;
HeapTuple tuple;
@@ -1475,6 +1476,16 @@ vac_truncate_clog(TransactionId frozenXID,
return;
/*
+ * We can't truncate the clog for transactions that still have undo. The
+ * oldestXidHavingUndo will be only valid for zheap storage engine, so it
+ * won't impact any other storage engine.
+ */
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ frozenXID = Min(frozenXID, oldestXidHavingUndo);
+
+ /*
* Advance the oldest value for commit timestamps before truncating, so
* that if a user requests a timestamp for a transaction we're truncating
* away right after this point, they get NULL instead of an ugly "file not
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 71c2321..9ce6ff0 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -12,7 +12,7 @@ subdir = src/backend/postmaster
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
- pgarch.o pgstat.o postmaster.o startup.o syslogger.o walwriter.o
+OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o discardworker.o fork_process.o \
+ pgarch.o pgstat.o postmaster.o startup.o syslogger.o undoworker.o walwriter.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index d2b695e..49df516 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -20,7 +20,9 @@
#include "pgstat.h"
#include "port/atomics.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "storage/dsm.h"
@@ -129,6 +131,15 @@ static const struct
},
{
"ApplyWorkerMain", ApplyWorkerMain
+ },
+ {
+ "UndoLauncherMain", UndoLauncherMain
+ },
+ {
+ "UndoWorkerMain", UndoWorkerMain
+ },
+ {
+ "DiscardWorkerMain", DiscardWorkerMain
}
};
diff --git a/src/backend/postmaster/discardworker.c b/src/backend/postmaster/discardworker.c
new file mode 100644
index 0000000..e4c6719
--- /dev/null
+++ b/src/backend/postmaster/discardworker.c
@@ -0,0 +1,170 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.c
+ * The undo discard worker for asynchronous undo management.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/discardworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include <unistd.h>
+
+/* These are always necessary for a bgworker. */
+#include "access/transam.h"
+#include "miscadmin.h"
+#include "postmaster/bgworker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/lwlock.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+#include "access/undodiscard.h"
+#include "pgstat.h"
+#include "postmaster/discardworker.h"
+#include "storage/procarray.h"
+#include "tcop/tcopprot.h"
+#include "utils/guc.h"
+#include "utils/resowner.h"
+
+static void undoworker_sigterm_handler(SIGNAL_ARGS);
+
+/* max sleep time between cycles (100 milliseconds) */
+#define MIN_NAPTIME_PER_CYCLE 100L
+#define DELAYED_NAPTIME 10 * MIN_NAPTIME_PER_CYCLE
+#define MAX_NAPTIME_PER_CYCLE 100 * MIN_NAPTIME_PER_CYCLE
+
+static bool got_SIGTERM = false;
+static bool hibernate = false;
+static long wait_time = MIN_NAPTIME_PER_CYCLE;
+
+/* SIGTERM: set flag to exit at next convenient time */
+static void
+undoworker_sigterm_handler(SIGNAL_ARGS)
+{
+ got_SIGTERM = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+}
+
+/*
+ * DiscardWorkerRegister -- Register a undo discard worker.
+ */
+void
+DiscardWorkerRegister(void)
+{
+ BackgroundWorker bgw;
+
+ /* TODO: This should be configurable. */
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "discard worker");
+ sprintf(bgw.bgw_library_name, "postgres");
+ sprintf(bgw.bgw_function_name, "DiscardWorkerMain");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * DiscardWorkerMain -- Main loop for the undo discard worker.
+ */
+void
+DiscardWorkerMain(Datum main_arg)
+{
+ ereport(LOG,
+ (errmsg("discard worker started")));
+
+ /* Establish signal handlers. */
+ pqsignal(SIGTERM, undoworker_sigterm_handler);
+ BackgroundWorkerUnblockSignals();
+
+ /* Make it easy to identify our processes. */
+ SetConfigOption("application_name", MyBgworkerEntry->bgw_name,
+ PGC_USERSET, PGC_S_SESSION);
+
+ /*
+ * Create resource owner for discard worker as it need to read the undo
+ * records outside the transaction blocks which intern access buffer read
+ * routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ /* Enter main loop */
+ while (!got_SIGTERM)
+ {
+ int rc;
+ TransactionId OldestXmin, oldestXidHavingUndo;
+
+ OldestXmin = GetOldestXmin(NULL, PROCARRAY_FLAGS_DEFAULT);
+
+ oldestXidHavingUndo = GetXidFromEpochXid(
+ pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));
+
+ /*
+ * Call the discard routine if there oldestXidHavingUndo is lagging
+ * behind OldestXmin.
+ */
+ if (OldestXmin != InvalidTransactionId &&
+ TransactionIdPrecedes(oldestXidHavingUndo, OldestXmin))
+ {
+ UndoDiscard(OldestXmin, &hibernate);
+
+ /*
+ * If we got some undo logs to discard or discarded something,
+ * then reset the wait_time as we have got work to do.
+ * Note that if there are some undologs that cannot be discarded,
+ * then above condition will remain unsatisified till oldestXmin
+ * remains unchanged and the wait_time will not reset in that case.
+ */
+ if (!hibernate)
+ wait_time = MIN_NAPTIME_PER_CYCLE;
+ }
+
+ /* Wait for more work. */
+ rc = WaitLatch(&MyProc->procLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wait_time,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN);
+
+ ResetLatch(&MyProc->procLatch);
+
+ /*
+ * Increase the wait_time based on the length of inactivity. If wait_time
+ * is within one second, then increment it by 100 ms at a time. Henceforth,
+ * increment it one second at a time, till it reaches ten seconds. Never
+ * increase the wait_time more than ten seconds, it will be too much of
+ * waiting otherwise.
+ */
+ if (rc & WL_TIMEOUT && hibernate)
+ {
+ wait_time += (wait_time < DELAYED_NAPTIME ?
+ MIN_NAPTIME_PER_CYCLE : DELAYED_NAPTIME);
+ if (wait_time > MAX_NAPTIME_PER_CYCLE)
+ wait_time = MAX_NAPTIME_PER_CYCLE;
+ }
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+
+ ReleaseAuxProcessResources(true);
+
+ /* we're done */
+ ereport(LOG,
+ (errmsg("discard worker shutting down")));
+
+ proc_exit(0);
+}
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 9d717d9..81f0698 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -3506,6 +3506,12 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_SYSLOGGER_MAIN:
event_name = "SysLoggerMain";
break;
+ case WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN:
+ event_name = "UndoDiscardWorkerMain";
+ break;
+ case WAIT_EVENT_UNDO_LAUNCHER_MAIN:
+ event_name = "UndoLauncherMain";
+ break;
case WAIT_EVENT_WAL_RECEIVER_MAIN:
event_name = "WalReceiverMain";
break;
@@ -3918,7 +3924,6 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_UNDO_FILE_SYNC:
event_name = "UndoFileSync";
break;
-
case WAIT_EVENT_WALSENDER_TIMELINE_HISTORY_READ:
event_name = "WALSenderTimelineHistoryRead";
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index eedc617..78b098c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -111,10 +111,12 @@
#include "port/pg_bswap.h"
#include "postmaster/autovacuum.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/walsender.h"
#include "storage/fd.h"
@@ -991,6 +993,11 @@ PostmasterMain(int argc, char *argv[])
*/
ApplyLauncherRegister();
+ UndoLauncherRegister();
+
+ /* Register the Undo Discard worker. */
+ DiscardWorkerRegister();
+
/*
* process any libraries that should be preloaded at postmaster start
*/
diff --git a/src/backend/postmaster/undoworker.c b/src/backend/postmaster/undoworker.c
new file mode 100644
index 0000000..bf72203
--- /dev/null
+++ b/src/backend/postmaster/undoworker.c
@@ -0,0 +1,664 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.c
+ * undo launcher and undo worker process.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/undoworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "funcapi.h"
+#include "miscadmin.h"
+#include "pgstat.h"
+
+#include "access/heapam.h"
+#include "access/htup.h"
+#include "access/htup_details.h"
+#include "access/sysattr.h"
+#include "access/xact.h"
+
+#include "catalog/indexing.h"
+#include "catalog/pg_database.h"
+
+#include "libpq/pqsignal.h"
+
+#include "postmaster/bgworker.h"
+#include "postmaster/fork_process.h"
+#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
+
+#include "replication/slot.h"
+#include "replication/worker_internal.h"
+
+#include "storage/ipc.h"
+#include "storage/lmgr.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+
+#include "tcop/tcopprot.h"
+
+#include "utils/fmgroids.h"
+#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/resowner.h"
+
+/* max sleep time between cycles (100 milliseconds) */
+#define DEFAULT_NAPTIME_PER_CYCLE 100L
+#define DEFAULT_RETRY_NAPTIME 50L
+
+int max_undo_workers = 5;
+
+typedef struct UndoApplyWorker
+{
+ /* Indicates if this slot is used or free. */
+ bool in_use;
+
+ /* Increased everytime the slot is taken by new worker. */
+ uint16 generation;
+
+ /* Pointer to proc array. NULL if not running. */
+ PGPROC *proc;
+
+ /* Database id to connect to. */
+ Oid dbid;
+} UndoApplyWorker;
+
+UndoApplyWorker *MyUndoWorker = NULL;
+
+typedef struct UndoApplyCtxStruct
+{
+ /* Supervisor process. */
+ pid_t launcher_pid;
+
+ /* Background workers. */
+ UndoApplyWorker workers[FLEXIBLE_ARRAY_MEMBER];
+} UndoApplyCtxStruct;
+
+UndoApplyCtxStruct *UndoApplyCtx;
+
+static void undo_worker_onexit(int code, Datum arg);
+static void undo_worker_cleanup(UndoApplyWorker *worker);
+
+static volatile sig_atomic_t got_SIGHUP = false;
+
+/*
+ * Wait for a background worker to start up and attach to the shmem context.
+ *
+ * This is only needed for cleaning up the shared memory in case the worker
+ * fails to attach.
+ */
+static void
+WaitForUndoWorkerAttach(UndoApplyWorker *worker,
+ uint16 generation,
+ BackgroundWorkerHandle *handle)
+{
+ BgwHandleStatus status;
+ int rc;
+
+ for (;;)
+ {
+ pid_t pid;
+
+ CHECK_FOR_INTERRUPTS();
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+
+ /* Worker either died or has started; no need to do anything. */
+ if (!worker->in_use || worker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ LWLockRelease(UndoWorkerLock);
+
+ /* Check if worker has died before attaching, and clean up after it. */
+ status = GetBackgroundWorkerPid(handle, &pid);
+
+ if (status == BGWH_STOPPED)
+ {
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ /* Ensure that this was indeed the worker we waited for. */
+ if (generation == worker->generation)
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ /*
+ * We need timeout because we generally don't get notified via latch
+ * about the worker attach. But we don't expect to have to wait long.
+ */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 10L, WAIT_EVENT_BGWORKER_STARTUP);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+ }
+
+ return;
+}
+
+/*
+ * Get dbid from the worker slot.
+ */
+static Oid
+slot_get_dbid(int slot)
+{
+ Oid dbid;
+
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty,",
+ slot)));
+ }
+
+ dbid = MyUndoWorker->dbid;
+
+ LWLockRelease(UndoWorkerLock);
+
+ return dbid;
+}
+
+/*
+ * Attach to a slot.
+ */
+static void
+undo_worker_attach(int slot)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty, cannot attach",
+ slot)));
+ }
+
+ if (MyUndoWorker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is already used by "
+ "another worker, cannot attach", slot)));
+ }
+
+ MyUndoWorker->proc = MyProc;
+ before_shmem_exit(undo_worker_onexit, (Datum) 0);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Walks the workers array and searches for one that matches given
+ * dbid.
+ */
+static UndoApplyWorker *
+undo_worker_find(Oid dbid)
+{
+ int i;
+ UndoApplyWorker *res = NULL;
+
+ Assert(LWLockHeldByMe(UndoWorkerLock));
+
+ /* Search for attached worker for a given db id. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (w->in_use && w->dbid == dbid)
+ {
+ res = w;
+ break;
+ }
+ }
+
+ return res;
+}
+
+/*
+ * Check whether the dbid exist or not.
+ *
+ * Refer comments from GetDatabaseTupleByOid.
+ * FIXME: Should we expose GetDatabaseTupleByOid and directly use it.
+ */
+static bool
+dbid_exist(Oid dboid)
+{
+ HeapTuple tuple;
+ Relation relation;
+ SysScanDesc scan;
+ ScanKeyData key[1];
+ bool result = false;
+ /*
+ * form a scan key
+ */
+ ScanKeyInit(&key[0],
+ Anum_pg_database_oid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dboid));
+
+ relation = heap_open(DatabaseRelationId, AccessShareLock);
+ scan = systable_beginscan(relation, DatabaseOidIndexId,
+ criticalSharedRelcachesBuilt,
+ NULL,
+ 1, key);
+
+ tuple = systable_getnext(scan);
+
+ if (HeapTupleIsValid(tuple))
+ result = true;
+
+ /* all done */
+ systable_endscan(scan);
+ heap_close(relation, AccessShareLock);
+
+ return result;
+}
+
+/*
+ * Start new undo apply background worker, if possible otherwise return false.
+ */
+static bool
+undo_worker_launch(Oid dbid)
+{
+ BackgroundWorker bgw;
+ BackgroundWorkerHandle *bgw_handle;
+ uint16 generation;
+ int i;
+ int slot = 0;
+ UndoApplyWorker *worker = NULL;
+
+ /*
+ * We need to do the modification of the shared memory under lock so that
+ * we have consistent view.
+ */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ /* Find unused worker slot. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (!w->in_use)
+ {
+ worker = w;
+ slot = i;
+ break;
+ }
+ }
+
+ /* There are no more free worker slots */
+ if (worker == NULL)
+ return false;
+
+ /* Prepare the worker slot. */
+ worker->in_use = true;
+ worker->proc = NULL;
+ worker->dbid = dbid;
+ worker->generation++;
+
+ generation = worker->generation;
+ LWLockRelease(UndoWorkerLock);
+
+ /* Register the new dynamic worker. */
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoWorkerMain");
+ snprintf(bgw.bgw_type, BGW_MAXLEN, "undo apply worker");
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "undo apply worker");
+
+ bgw.bgw_restart_time = BGW_NEVER_RESTART;
+ bgw.bgw_notify_pid = MyProcPid;
+ bgw.bgw_main_arg = Int32GetDatum(slot);
+
+ StartTransactionCommand();
+ /* Check the database exists or not. */
+ if (!dbid_exist(dbid))
+ {
+ CommitTransactionCommand();
+
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return true;
+ }
+
+ /*
+ * Acquire database object lock before launching the worker so that it
+ * doesn't get dropped while worker is connecting to the database.
+ */
+ LockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+
+ /* Recheck whether database still exists or not. */
+ if (!dbid_exist(dbid))
+ {
+ CommitTransactionCommand();
+
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return true;
+ }
+
+ if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle))
+ {
+ /* Failed to start worker, so clean up the worker slot. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+
+ UnlockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+ CommitTransactionCommand();
+
+ return false;
+ }
+
+ /* Now wait until it attaches. */
+ WaitForUndoWorkerAttach(worker, generation, bgw_handle);
+
+ /*
+ * By this point the undo-worker has already connected to the database so we
+ * can release the database lock.
+ */
+ UnlockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+ CommitTransactionCommand();
+
+ return true;
+}
+
+/*
+ * Detach the worker (cleans up the worker info).
+ */
+static void
+undo_worker_detach(void)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ undo_worker_cleanup(MyUndoWorker);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Clean up worker info.
+ */
+static void
+undo_worker_cleanup(UndoApplyWorker *worker)
+{
+ Assert(LWLockHeldByMeInMode(UndoWorkerLock, LW_EXCLUSIVE));
+
+ worker->in_use = false;
+ worker->proc = NULL;
+ worker->dbid = InvalidOid;
+}
+
+/*
+ * Cleanup function for undo worker launcher.
+ *
+ * Called on undo worker launcher exit.
+ */
+static void
+undo_launcher_onexit(int code, Datum arg)
+{
+ UndoApplyCtx->launcher_pid = 0;
+}
+
+/* SIGHUP: set flag to reload configuration at next convenient time */
+static void
+undo_launcher_sighup(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGHUP = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+undo_worker_onexit(int code, Datum arg)
+{
+ undo_worker_detach();
+}
+
+/*
+ * UndoLauncherShmemSize
+ * Compute space needed for undo launcher shared memory
+ */
+Size
+UndoLauncherShmemSize(void)
+{
+ Size size;
+
+ /*
+ * Need the fixed struct and the array of LogicalRepWorker.
+ */
+ size = sizeof(UndoApplyCtxStruct);
+ size = MAXALIGN(size);
+ size = add_size(size, mul_size(max_undo_workers,
+ sizeof(UndoApplyWorker)));
+ return size;
+}
+
+/*
+ * UndoLauncherRegister
+ * Register a background worker running the undo worker launcher.
+ */
+void
+UndoLauncherRegister(void)
+{
+ BackgroundWorker bgw;
+
+ if (max_undo_workers == 0)
+ return;
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoLauncherMain");
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "undo worker launcher");
+ snprintf(bgw.bgw_type, BGW_MAXLEN,
+ "undo worker launcher");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * UndoLauncherShmemInit
+ * Allocate and initialize undo worker launcher shared memory
+ */
+void
+UndoLauncherShmemInit(void)
+{
+ bool found;
+
+ UndoApplyCtx = (UndoApplyCtxStruct *)
+ ShmemInitStruct("Undo Worker Launcher Data",
+ UndoLauncherShmemSize(),
+ &found);
+
+ if (!found)
+ memset(UndoApplyCtx, 0, UndoLauncherShmemSize());
+}
+
+/*
+ * Main loop for the undo worker launcher process.
+ */
+void
+UndoLauncherMain(Datum main_arg)
+{
+ MemoryContext tmpctx;
+ MemoryContext oldctx;
+
+ ereport(DEBUG1,
+ (errmsg("undo launcher started")));
+
+ before_shmem_exit(undo_launcher_onexit, (Datum) 0);
+
+ Assert(UndoApplyCtx->launcher_pid == 0);
+ UndoApplyCtx->launcher_pid = MyProcPid;
+
+ /* Establish signal handlers. */
+ pqsignal(SIGHUP, undo_launcher_sighup);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * Establish connection to nailed catalogs (we only ever access
+ * pg_subscription).
+ */
+ BackgroundWorkerInitializeConnection(NULL, NULL, 0);
+
+ /* Use temporary context for the database list and worker info. */
+ tmpctx = AllocSetContextCreate(TopMemoryContext,
+ "Undo worker Launcher context",
+ ALLOCSET_DEFAULT_SIZES);
+ /* Enter main loop */
+ for (;;)
+ {
+ int rc;
+ List *dblist;
+ ListCell *l;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* switch to the temp context. */
+ oldctx = MemoryContextSwitchTo(tmpctx);
+ dblist = RollbackHTGetDBList();
+
+ foreach(l, dblist)
+ {
+ UndoApplyWorker *w;
+ Oid dbid = lfirst_oid(l);
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+ w = undo_worker_find(dbid);
+ LWLockRelease(UndoWorkerLock);
+
+ if (w == NULL)
+ {
+retry:
+ if (!undo_worker_launch(dbid))
+ {
+ /* Could not launch the worker, retry after sometime, */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_RETRY_NAPTIME,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+ goto retry;
+ }
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+
+ /* Clean the temporary memory. */
+ MemoryContextReset(tmpctx);
+
+ /* Wait for more work. */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_NAPTIME_PER_CYCLE,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ if (got_SIGHUP)
+ {
+ got_SIGHUP = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+}
+
+/*
+ * UndoWorkerMain -- Main loop for the undo apply worker.
+ */
+void
+UndoWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ Oid dbid;
+
+ dbid = slot_get_dbid(worker_slot);
+
+ /* Setup signal handling */
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /* Connect to the database. */
+ BackgroundWorkerInitializeConnectionByOid(dbid, 0, 0);
+
+ /* Attach to slot */
+ undo_worker_attach(worker_slot);
+
+ /*
+ * Create resource owner for undo worker. Undo worker need this as it
+ * need to read the undo records outside the transaction blocks which
+ * intern access buffer read routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ RollbackFromHT(dbid);
+
+ ReleaseAuxProcessResources(true);
+
+ proc_exit(0);
+}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 1a7a381..b9c0e18 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -159,6 +159,10 @@ LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *recor
ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(record),
buf.origptr);
break;
+ case RM_UNDOACTION_ID:
+ /* Logical decoding is not yet implemented for undoactions. */
+ Assert(0);
+ break;
case RM_NEXT_ID:
elog(ERROR, "unexpected RM_NEXT_ID rmgr_id: %u", (RmgrIds) XLogRecGetRmid(buf.record));
}
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 4725cbe..1470041 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -29,6 +29,8 @@
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
#include "replication/walreceiver.h"
@@ -152,6 +154,8 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
size = add_size(size, BackendRandomShmemSize());
+ size = add_size(size, RollbackHTSize());
+ size = add_size(size, UndoLauncherShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -226,6 +230,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
SUBTRANSShmemInit();
MultiXactShmemInit();
InitBufferPool();
+ InitRollbackHashTable();
/*
* Set up lock manager
@@ -264,6 +269,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
WalSndShmemInit();
WalRcvShmemInit();
ApplyLauncherShmemInit();
+ UndoLauncherShmemInit();
/*
* Set up other modules that need some shared memory space
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 554af46..52f6959 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -51,3 +51,5 @@ BackendRandomLock 43
LogicalRepWorkerLock 44
CLogTruncationLock 45
UndoLogLock 46
+RollbackHTLock 47
+UndoWorkerLock 48
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 33387fb..69c7b6a 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -286,6 +286,8 @@ InitProcGlobal(void)
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemAlloc(sizeof(slock_t));
SpinLockInit(ProcStructLock);
+
+ pg_atomic_init_u64(&ProcGlobal->oldestXidWithEpochHavingUndo, 0);
}
/*
diff --git a/src/backend/utils/adt/lockfuncs.c b/src/backend/utils/adt/lockfuncs.c
index 525decb..8b2ad64 100644
--- a/src/backend/utils/adt/lockfuncs.c
+++ b/src/backend/utils/adt/lockfuncs.c
@@ -32,6 +32,7 @@ const char *const LockTagTypeNames[] = {
"virtualxid",
"speculative token",
"object",
+ "undoaction",
"userlock",
"advisory"
};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index c693977..12e7704 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -121,6 +121,7 @@ bool allowSystemTableMods = false;
int work_mem = 1024;
int maintenance_work_mem = 16384;
int max_parallel_maintenance_workers = 2;
+int rollback_overflow_size = 64;
/*
* Primary determinants of sizes of shared-memory structures.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index b188657..2ec6c7c 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2862,6 +2862,17 @@ static struct config_int ConfigureNamesInt[] =
},
{
+ {"rollback_overflow_size", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Rollbacks greater than this size are done lazily"),
+ NULL,
+ GUC_UNIT_MB
+ },
+ &rollback_overflow_size,
+ 64, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
{"wal_segment_size", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the size of write ahead log segments."),
NULL,
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index a376875..8ebb9c2 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -78,8 +78,8 @@ pg_control_system(PG_FUNCTION_ARGS)
Datum
pg_control_checkpoint(PG_FUNCTION_ARGS)
{
- Datum values[19];
- bool nulls[19];
+ Datum values[20];
+ bool nulls[20];
TupleDesc tupdesc;
HeapTuple htup;
ControlFileData *ControlFile;
@@ -128,6 +128,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
XIDOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 18, "checkpoint_time",
TIMESTAMPTZOID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 19, "oldest_xid_with_epoch_having_undo",
+ TIMESTAMPTZOID, -1, 0);
tupdesc = BlessTupleDesc(tupdesc);
/* Read the control file. */
@@ -202,6 +204,9 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
time_t_to_timestamptz(ControlFile->checkPointCopy.time));
nulls[17] = false;
+ values[18] = Int64GetDatum(ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo);
+ nulls[18] = false;
+
htup = heap_form_tuple(tupdesc, values, nulls);
PG_RETURN_DATUM(HeapTupleGetDatum(htup));
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1fa02d2..82b9cf6 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -737,4 +737,11 @@
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
+# If often there are large transactions requiring rollbacks, then we can push
+# them to undo-workers for better performance. The size specifeid by the
+# parameter below, determines the minimum size of the rollback requests to be
+# sent to the undo-worker.
+#
+#rollback_overflow_size = 64
+
# Add settings for extensions here
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 895a51f..753929e 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -278,6 +278,8 @@ main(int argc, char *argv[])
ControlFile->checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile->checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile->checkPointCopy.oldestXidWithEpochHavingUndo);
printf(_("Time of latest checkpoint: %s\n"),
ckpttime_str);
printf(_("Fake LSN counter for unlogged rels: %X/%X\n"),
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 6fb403a..72714dd 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -448,6 +448,7 @@ main(int argc, char *argv[])
if (ControlFile.checkPointCopy.oldestXid < FirstNormalTransactionId)
ControlFile.checkPointCopy.oldestXid += FirstNormalTransactionId;
ControlFile.checkPointCopy.oldestXidDB = InvalidOid;
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo = 0;
}
if (set_oldest_commit_ts_xid != 0)
@@ -716,6 +717,8 @@ GuessControlValues(void)
ControlFile.checkPointCopy.oldestMultiDB = InvalidOid;
ControlFile.checkPointCopy.time = (pg_time_t) time(NULL);
ControlFile.checkPointCopy.oldestActiveXid = InvalidTransactionId;
+ ControlFile.checkPointCopy.nextXid = 0;
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo = 0;
ControlFile.state = DB_SHUTDOWNED;
ControlFile.time = (pg_time_t) time(NULL);
@@ -808,6 +811,8 @@ PrintControlValues(bool guessed)
ControlFile.checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile.checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo);
printf(_("Maximum data alignment: %u\n"),
ControlFile.maxAlign);
/* we don't print floatFormat since can't say much useful about it */
@@ -884,6 +889,8 @@ PrintNewControlValues(void)
ControlFile.checkPointCopy.oldestXid);
printf(_("OldestXID's DB: %u\n"),
ControlFile.checkPointCopy.oldestXidDB);
+ printf(_("OldestXidWithEpochHavingUndo:" UINT64_FORMAT "\n"),
+ ControlFile.checkPointCopy.oldestXidWithEpochHavingUndo);
}
if (set_xid_epoch != -1)
diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c
index 4002847..e71a71e 100644
--- a/src/bin/pg_rewind/parsexlog.c
+++ b/src/bin/pg_rewind/parsexlog.c
@@ -29,7 +29,7 @@
* RmgrNames is an array of resource manager names, to make error messages
* a bit nicer.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
name,
static const char *RmgrNames[RM_MAX_ID + 1] = {
diff --git a/src/bin/pg_waldump/rmgrdesc.c b/src/bin/pg_waldump/rmgrdesc.c
index 938150d..3ef0a60 100644
--- a/src/bin/pg_waldump/rmgrdesc.c
+++ b/src/bin/pg_waldump/rmgrdesc.c
@@ -20,6 +20,7 @@
#include "access/nbtxlog.h"
#include "access/rmgr.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -33,7 +34,7 @@
#include "storage/standbydefs.h"
#include "utils/relmapper.h"
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
{ name, desc, identify},
const RmgrDescData RmgrDescTable[RM_MAX_ID + 1] = {
diff --git a/src/include/access/rmgr.h b/src/include/access/rmgr.h
index c9b5c56..e1fb42a 100644
--- a/src/include/access/rmgr.h
+++ b/src/include/access/rmgr.h
@@ -19,7 +19,7 @@ typedef uint8 RmgrId;
* Note: RM_MAX_ID must fit in RmgrId; widening that type will affect the XLOG
* file format.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
symname,
typedef enum RmgrIds
diff --git a/src/include/access/rmgrlist.h b/src/include/access/rmgrlist.h
index 9c6fca4..eee7835 100644
--- a/src/include/access/rmgrlist.h
+++ b/src/include/access/rmgrlist.h
@@ -25,26 +25,27 @@
*/
/* symbol name, textual name, redo, desc, identify, startup, cleanup */
-PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL)
-PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL)
-PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL)
-PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL)
-PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL)
-PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask)
-PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask)
-PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask)
-PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask)
-PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask)
-PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask)
-PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask)
-PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL)
-PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL)
-PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask)
-PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL)
-PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL)
+PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask, NULL, NULL)
+PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask, NULL, NULL)
+PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask, NULL, NULL)
+PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask, NULL, NULL)
+PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask, NULL, NULL)
+PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask, NULL, NULL)
+PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask, NULL, NULL)
+PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask, NULL, NULL)
+PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOACTION_ID, "UndoAction", undoaction_redo, undoaction_desc, undoaction_identify, NULL, NULL, NULL, NULL, NULL)
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index f979a06..29ed249 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -68,6 +68,10 @@
(AssertMacro(TransactionIdIsNormal(id1) && TransactionIdIsNormal(id2)), \
(int32) ((id1) - (id2)) > 0)
+/* Extract xid from a value comprised of epoch and xid */
+#define GetXidFromEpochXid(epochxid) \
+ ((uint32) (epochxid) & 0XFFFFFFFF)
+
/* ----------
* Object ID (OID) zero is InvalidOid.
*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 0e932da..fb05c3b 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -41,7 +41,7 @@ extern GlobalTransaction MarkAsPreparing(TransactionId xid, const char *gid,
TimestampTz prepared_at,
Oid owner, Oid databaseid);
-extern void StartPrepare(GlobalTransaction gxact);
+extern void StartPrepare(GlobalTransaction gxact, UndoRecPtr *, UndoRecPtr *);
extern void EndPrepare(GlobalTransaction gxact);
extern bool StandbyTransactionIdIsPrepared(TransactionId xid);
diff --git a/src/include/access/undoaction.h b/src/include/access/undoaction.h
new file mode 100644
index 0000000..5455259
--- /dev/null
+++ b/src/include/access/undoaction.h
@@ -0,0 +1,28 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.h
+ * undo action prototypes
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_H
+#define UNDOACTION_H
+
+#include "postgres.h"
+
+#include "access/undolog.h"
+#include "access/undorecord.h"
+
+/* undo record information */
+typedef struct UndoRecInfo
+{
+ UndoRecPtr urp; /* undo recptr (undo record location). */
+ UnpackedUndoRecord *uur; /* actual undo record. */
+} UndoRecInfo;
+
+#endif
diff --git a/src/include/access/undoaction_xlog.h b/src/include/access/undoaction_xlog.h
new file mode 100644
index 0000000..bfc6418
--- /dev/null
+++ b/src/include/access/undoaction_xlog.h
@@ -0,0 +1,74 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction_xlog.h
+ * undo action XLOG definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction_xlog.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_XLOG_H
+#define UNDOACTION_XLOG_H
+
+#include "access/undolog.h"
+#include "access/xlogreader.h"
+#include "lib/stringinfo.h"
+#include "storage/off.h"
+
+/*
+ * WAL record definitions for undoactions.c's WAL operations
+ */
+#define XLOG_UNDO_PAGE 0x00
+#define XLOG_UNDO_RESET_SLOT 0x10
+#define XLOG_UNDO_APPLY_PROGRESS 0x20
+
+/*
+ * xl_undoaction_page flag values, 8 bits are available.
+ */
+#define XLU_PAGE_CONTAINS_TPD_SLOT (1<<0)
+#define XLU_PAGE_CLEAR_VISIBILITY_MAP (1<<1)
+#define XLU_CONTAINS_TPD_OFFSET_MAP (1<<2)
+#define XLU_INIT_PAGE (1<<3)
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_page
+{
+ UndoRecPtr urec_ptr;
+ TransactionId xid;
+ int trans_slot_id; /* transaction slot id */
+} xl_undoaction_page;
+
+#define SizeOfUndoActionPage (offsetof(xl_undoaction_page, trans_slot_id) + sizeof(int))
+
+/* This is what we need to know about undo apply progress */
+typedef struct xl_undoapply_progress
+{
+ UndoRecPtr urec_ptr;
+ uint32 progress;
+} xl_undoapply_progress;
+
+#define SizeOfUndoActionProgress (offsetof(xl_undoapply_progress, progress) + sizeof(uint32))
+
+/*
+ * xl_undoaction_reset_slot flag values, 8 bits are available.
+ */
+#define XLU_RESET_CONTAINS_TPD_SLOT (1<<0)
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_reset_slot
+{
+ UndoRecPtr urec_ptr;
+ int trans_slot_id; /* transaction slot id */
+ uint8 flags;
+} xl_undoaction_reset_slot;
+
+#define SizeOfUndoActionResetSlot (offsetof(xl_undoaction_reset_slot, flags) + sizeof(uint8))
+
+extern void undoaction_redo(XLogReaderState *record);
+extern void undoaction_desc(StringInfo buf, XLogReaderState *record);
+extern const char *undoaction_identify(uint8 info);
+
+#endif /* UNDOACTION_XLOG_H */
diff --git a/src/include/access/undodiscard.h b/src/include/access/undodiscard.h
new file mode 100644
index 0000000..70d6408
--- /dev/null
+++ b/src/include/access/undodiscard.h
@@ -0,0 +1,31 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoinsert.h
+ * undo discard definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undodiscard.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDODISCARD_H
+#define UNDODISCARD_H
+
+#include "access/undolog.h"
+#include "access/xlogdefs.h"
+#include "catalog/pg_class.h"
+#include "storage/lwlock.h"
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+extern void UndoDiscard(TransactionId xmin, bool *hibernate);
+
+#endif /* UNDODISCARD_H */
diff --git a/src/include/access/undoinsert.h b/src/include/access/undoinsert.h
index 41384e1..d736095 100644
--- a/src/include/access/undoinsert.h
+++ b/src/include/access/undoinsert.h
@@ -46,5 +46,7 @@ extern void UndoSetPrepareSize(UnpackedUndoRecord *undorecords, int nrecords,
TransactionId xid, UndoPersistence upersistence);
extern UndoRecPtr UndoGetPrevUndoRecptr(UndoRecPtr urp, uint16 prevlen, UndoRecPtr prevurp);
extern void ResetUndoBuffers(void);
+extern void PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress);
+extern void UndoRecordUpdateTransInfo(int idx);
#endif /* UNDOINSERT_H */
diff --git a/src/include/access/undorecord.h b/src/include/access/undorecord.h
index 9f09055..6663277 100644
--- a/src/include/access/undorecord.h
+++ b/src/include/access/undorecord.h
@@ -20,6 +20,17 @@
#include "storage/buf.h"
#include "storage/off.h"
+typedef enum undorectype
+{
+ UNDO_INSERT,
+ UNDO_MULTI_INSERT,
+ UNDO_DELETE,
+ UNDO_INPLACE_UPDATE,
+ UNDO_UPDATE,
+ UNDO_XID_LOCK_ONLY,
+ UNDO_XID_MULTI_LOCK_ONLY,
+ UNDO_ITEMID_UNUSED
+} undorectype;
/*
* Every undo record begins with an UndoRecordHeader structure, which is
@@ -30,6 +41,7 @@
*/
typedef struct UndoRecordHeader
{
+ RmgrId urec_rmid; /* RMGR [XXX:TODO: this creates an alignment hole?] */
uint8 urec_type; /* record type code */
uint8 urec_info; /* flag bits */
uint16 urec_prevlen; /* length of previous record in bytes */
@@ -162,6 +174,7 @@ typedef struct UndoRecordPayload
*/
typedef struct UnpackedUndoRecord
{
+ RmgrId uur_rmid; /* rmgr ID */
uint8 uur_type; /* record type code */
uint8 uur_info; /* flag bits */
uint16 uur_prevlen; /* length of previous record */
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 73394c5..ad8cabe 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -419,6 +419,7 @@ extern XLogRecPtr XactLogAbortRecord(TimestampTz abort_time,
int xactflags, TransactionId twophase_xid,
const char *twophase_gid);
extern void xact_redo(XLogReaderState *record);
+extern void XactPerformUndoActionsIfPending(void);
/* xactdesc.c */
extern void xact_desc(StringInfo buf, XLogReaderState *record);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d4e742f..1091fe7 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -225,6 +225,9 @@ extern bool XLOG_DEBUG;
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+/* Generate a 64-bit xid by using epoch and 32-bit xid. */
+#define MakeEpochXid(epoch, xid) \
+ ((epoch << 32) | (xid))
/* Checkpoint statistics */
typedef struct CheckpointStatsData
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 30610b3..9226cff 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -21,8 +21,10 @@
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
+#include "access/undorecord.h"
#include "datatype/timestamp.h"
#include "lib/stringinfo.h"
+#include "nodes/pg_list.h"
#include "pgtime.h"
#include "storage/block.h"
#include "storage/relfilenode.h"
@@ -294,9 +296,14 @@ typedef struct RmgrData
void (*rm_startup) (void);
void (*rm_cleanup) (void);
void (*rm_mask) (char *pagedata, BlockNumber blkno);
+ bool (*rm_undo) (List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock,
+ int options);
+ void (*rm_undo_desc) (StringInfo buf, UnpackedUndoRecord *record);
} RmgrData;
-extern const RmgrData RmgrTable[];
+extern PGDLLIMPORT const RmgrData RmgrTable[];
/*
* Exported to support xlog switching from checkpointer
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 773d9e6..6918efc 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -61,6 +61,13 @@ typedef struct CheckPoint
* set to InvalidTransactionId.
*/
TransactionId oldestActiveXid;
+
+ /*
+ * Oldest transaction id with epoc which is having undo. Include this value
+ * in the checkpoint record so that whenever server starts we get proper
+ * value.
+ */
+ uint64 oldestXidWithEpochHavingUndo;
} CheckPoint;
/* XLOG info values for XLOG rmgr */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index d6b32c0..549bb38 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -245,6 +245,7 @@ extern PGDLLIMPORT bool allowSystemTableMods;
extern PGDLLIMPORT int work_mem;
extern PGDLLIMPORT int maintenance_work_mem;
extern PGDLLIMPORT int max_parallel_maintenance_workers;
+extern PGDLLIMPORT int rollback_overflow_size;
extern int VacuumCostPageHit;
extern int VacuumCostPageMiss;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 763379e..7c5f05e 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -749,6 +749,7 @@ typedef enum BackendState
#define PG_WAIT_TIMEOUT 0x09000000U
#define PG_WAIT_IO 0x0A000000U
#define PG_WAIT_PAGE_TRANS_SLOT 0x0B000000U
+#define PG_WAIT_ROLLBACK_HT 0x0C000000U
/* ----------
* Wait Events - Activity
@@ -774,6 +775,8 @@ typedef enum
WAIT_EVENT_WAL_RECEIVER_MAIN,
WAIT_EVENT_WAL_SENDER_MAIN,
WAIT_EVENT_WAL_WRITER_MAIN,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN
} WaitEventActivity;
/* ----------
diff --git a/src/include/postmaster/discardworker.h b/src/include/postmaster/discardworker.h
new file mode 100644
index 0000000..f00c6c4
--- /dev/null
+++ b/src/include/postmaster/discardworker.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.h
+ * Exports from postmaster/discardworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/discardworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _DISCARDWORKER_H
+#define _DISCARDWORKER_H
+
+/*
+ * This function will perform multiple actions based on need. (a) retrieve
+ * transactions which have become all-visible and truncate the associated undo
+ * logs or will increment the tail pointer. (b) drop the buffers corresponding
+ * to truncated pages.
+ */
+extern void DiscardWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void DiscardWorkerRegister(void);
+
+#endif /* _DISCARDWORKER_H */
diff --git a/src/include/postmaster/undoloop.h b/src/include/postmaster/undoloop.h
new file mode 100644
index 0000000..21d91c3
--- /dev/null
+++ b/src/include/postmaster/undoloop.h
@@ -0,0 +1,89 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoloop.h
+ * Exports from postmaster/undoloop.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ *
+ * src/include/postmaster/undoloop.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOLOOP_H
+#define _UNDOLOOP_H
+
+#include "access/undoinsert.h"
+#include "utils/hsearch.h"
+#include "utils/relcache.h"
+
+
+/* Various options while executing the undo actions for the page. */
+#define UNDO_ACTION_UPDATE_TPD 0x0001
+
+/* Remembers the last seen RecentGlobalXmin */
+TransactionId latestRecentGlobalXmin;
+
+/*
+ * This function will read the undo records starting from the undo
+ * from_urecptr till to_urecptr and if to_urecptr is invalid then till the
+ * first undo location of transaction. This also discards the buffers by
+ * calling DropUndoBuffers for which undo log is removed. This function
+ * can be used by RollbackToSavePoint, by Rollback, by undoworker to complete
+ * the work of errored out transactions or when there is an error in single
+ * user mode.
+ */
+extern void execute_undo_actions(UndoRecPtr from_urecptr,
+ UndoRecPtr to_urecptr, bool nopartial, bool rewind, bool rellock);
+extern void process_and_execute_undo_actions_page(UndoRecPtr from_urecptr,
+ Relation rel, Buffer buffer, uint32 epoch,
+ TransactionId xid, int slot_no);
+
+/*
+ * This function will be responsible to truncate the undo logs
+ * for transactions that become all-visible after RecentGlobalXmin has
+ * advanced (value is different than latestRecentGlobalXmin). The easiest
+ * way could be to traverse the undo log array that contains least transaction
+ * id for that undo log and see if it precedes RecentGlobalXmin, then start
+ * discarding the undo log for that transaction (moving the tail pointer of
+ * undo log) till it finds the transaction which is not all-visible. This also
+ * discards the buffers by calling ForgetBuffer for which undo log is
+ * removed. This function can be invoked by undoworker or after commit in
+ * single user mode.
+ */
+extern void recover_undo_pages();
+
+/*
+ * To increase the efficiency of the zheap system, we create a hash table for
+ * the rollbacks. All the rollback requests exceeding certain threshold, are
+ * pushed to this table. Undo worker starts reading the entries from this hash
+ * table one at a time, performs undo actions related to the respective xid and
+ * removes them from the hash table. This way backend is free from performing the
+ * undo actions in case of heavy rollbacks. The data structures and the routines
+ * required for this infrastructure are as follows.
+ */
+
+/* This is the data structure for each hash table entry for rollbacks. */
+typedef struct RollbackHashEntry
+{
+ UndoRecPtr start_urec_ptr;
+ UndoRecPtr end_urec_ptr;
+ Oid dbid;
+} RollbackHashEntry;
+
+extern bool RollbackHTIsFull(void);
+
+/* To push the rollback requests from backend to the respective hash table */
+extern bool PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr,
+ Oid dbid);
+
+/* To perform the undo actions reading from the hash table */
+extern void RollbackFromHT(Oid dbid);
+/* To calculate the size of the hash table size for rollabcks. */
+extern int RollbackHTSize(void);
+
+/* To initialize the hash table in shared memory for rollbacks. */
+extern void InitRollbackHashTable(void);
+extern List *RollbackHTGetDBList(void);
+extern bool ConditionTransactionUndoActionLock(TransactionId xid);
+extern void TransactionUndoActionLockRelease(TransactionId xid);
+#endif /* _UNDOLOOP_H */
diff --git a/src/include/postmaster/undoworker.h b/src/include/postmaster/undoworker.h
new file mode 100644
index 0000000..c277e01
--- /dev/null
+++ b/src/include/postmaster/undoworker.h
@@ -0,0 +1,39 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.h
+ * Exports from postmaster/undoworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/undoworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOWORKER_H
+#define _UNDOWORKER_H
+
+/* GUC options */
+/* undo worker sleep time between rounds */
+extern int UndoWorkerDelay;
+
+/*
+ * This function will perform multiple actions based on need. (a) retreive
+ * transaction and its corresponding undopoiter from shared memory queue and
+ * call undoloop to perform undo actions. After applying all the undo records
+ * for a particular transaction, it will increment the tail pointer in undo log.
+ * (b) it needs to retrieve transactions which have become all-visible and truncate
+ * the associated undo logs or will increment the tail pointer. (c) udjust the
+ * number of undo workers based on the work required to perform undo actions
+ * (it could be size of shared memory queue containing transactions that needs
+ * aborts). (d) drop the buffers corresponding to truncated pages (e) Sleep for
+ * UndoWorkerDelay, if there is no more work.
+ */
+extern void UndoWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void UndoLauncherRegister(void);
+extern void UndoLauncherShmemInit(void);
+extern Size UndoLauncherShmemSize(void);
+extern void UndoLauncherMain(Datum main_arg);
+extern void UndoWorkerMain(Datum main_arg);
+
+#endif /* _UNDOWORKER_H */
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index a37fda7..675761c 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -150,6 +150,8 @@ typedef enum LockTagType
LOCKTAG_VIRTUALTRANSACTION, /* virtual transaction (ditto) */
/* ID info for a virtual transaction is its VirtualTransactionId */
LOCKTAG_SPECULATIVE_TOKEN, /* speculative insertion Xid and token */
+ /* ID info for an transaction undoaction is transaction id */
+ LOCKTAG_TRANSACTION_UNDOACTION, /* transaction (waiting for undoaction) */
/* ID info for a transaction is its TransactionId */
LOCKTAG_OBJECT, /* non-relation database object */
/* ID info for an object is DB OID + CLASS OID + OBJECT OID + SUBID */
@@ -246,6 +248,14 @@ typedef struct LOCKTAG
(locktag).locktag_type = LOCKTAG_SPECULATIVE_TOKEN, \
(locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+#define SET_LOCKTAG_TRANSACTION_UNDOACTION(locktag,xid) \
+ ((locktag).locktag_field1 = (xid), \
+ (locktag).locktag_field2 = 0, \
+ (locktag).locktag_field3 = 0, \
+ (locktag).locktag_field4 = 0, \
+ (locktag).locktag_type = LOCKTAG_TRANSACTION_UNDOACTION, \
+ (locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+
#define SET_LOCKTAG_OBJECT(locktag,dboid,classoid,objoid,objsubid) \
((locktag).locktag_field1 = (dboid), \
(locktag).locktag_field2 = (classoid), \
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index cb613c8..f6b9d98 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -270,6 +270,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Oldest transaction id which is having undo. */
+ pg_atomic_uint64 oldestXidWithEpochHavingUndo;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
1.8.3.1
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: Undo worker and transaction rollback
2018-11-05 11:40 Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-05 12:32 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-30 15:13 ` Re: Undo worker and transaction rollback Dmitry Dolgov <[email protected]>
2018-12-03 12:44 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-12-04 09:43 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-12-31 05:34 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
@ 2019-01-11 04:09 ` Dilip Kumar <[email protected]>
2019-01-23 04:09 ` Re: Undo worker and transaction rollback Amit Kapila <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Dilip Kumar @ 2019-01-11 04:09 UTC (permalink / raw)
To: Dmitry Dolgov <[email protected]>; +Cc: pgsql-hackers
On Mon, Dec 31, 2018 at 11:04 AM Dilip Kumar <[email protected]> wrote:
> On Tue, Dec 4, 2018 at 3:13 PM Dilip Kumar <[email protected]> wrote:
>
> After the latest change in undo interface patch[1], this patch need to
> be rebased. Attaching the rebased version of the patch.
>
> [1]
> https://www.postgresql.org/message-id/CAFiTN-tfquvm6tnWFaJNYYz-vSY%3D%2BSQTMv%2BFv1jPozTrW4mtKw%40ma...
>
>
I have rebased this patch on top of latest version of undolog and
undo-interface patch set [1]
These all patch set including uno-log patch applies on commit
(be2e329f2e700032df087e08ac2103ba3d1fa811).
[1]
https://www.postgresql.org/message-id/CAFiTN-tqpHbip3621ABQ_TNW6K7JO1-N8qvpiOaBE%2BJS2fnAvA%40mail.g...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] 0001-Undo-worker-and-transaction-rollback_v6.patch (134.6K, ../../CAFiTN-vjvdw+Z8dw5xg7B+WAz9qA36iqgbUZjUCx6sPrWYOL3g@mail.gmail.com/3-0001-Undo-worker-and-transaction-rollback_v6.patch)
download | inline diff:
From 3cad4c467d130ab30a9abadd24b2683074d998ce Mon Sep 17 00:00:00 2001
From: Dilip Kumar <[email protected]>
Date: Thu, 10 Jan 2019 12:06:08 +0530
Subject: [PATCH] Undo worker and transaction rollback
Patch provides mechanism for discarding the older undo when they
are no longer relavent. It also providea a mechanism for rolling
back the effect of the aborted transactiona. It handles the rollbacks
directly from backends as well as from undo worker for the larger
transaction.
Dilip Kumar, with help from Rafia Sabih, Amit Kapila, Mithun.CY, Thomas Munro and Kuntal Ghosh
---
src/backend/access/rmgrdesc/Makefile | 3 +-
src/backend/access/rmgrdesc/undoactiondesc.c | 64 +++
src/backend/access/rmgrdesc/xlogdesc.c | 4 +-
src/backend/access/transam/rmgr.c | 5 +-
src/backend/access/transam/twophase.c | 45 +-
src/backend/access/transam/varsup.c | 11 +
src/backend/access/transam/xact.c | 441 ++++++++++++++++-
src/backend/access/transam/xlog.c | 20 +
src/backend/access/undo/Makefile | 2 +-
src/backend/access/undo/undoaction.c | 688 ++++++++++++++++++++++++++
src/backend/access/undo/undoactionxlog.c | 49 ++
src/backend/access/undo/undodiscard.c | 447 +++++++++++++++++
src/backend/access/undo/undoinsert.c | 52 +-
src/backend/access/undo/undorecord.c | 3 +
src/backend/commands/vacuum.c | 10 +
src/backend/postmaster/Makefile | 4 +-
src/backend/postmaster/bgworker.c | 11 +
src/backend/postmaster/discardworker.c | 170 +++++++
src/backend/postmaster/pgstat.c | 7 +-
src/backend/postmaster/postmaster.c | 7 +
src/backend/postmaster/undoworker.c | 664 +++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 4 +
src/backend/storage/ipc/ipci.c | 6 +
src/backend/storage/lmgr/lwlocknames.txt | 2 +
src/backend/storage/lmgr/proc.c | 2 +
src/backend/utils/adt/lockfuncs.c | 1 +
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc.c | 11 +
src/backend/utils/misc/pg_controldata.c | 9 +-
src/backend/utils/misc/postgresql.conf.sample | 7 +
src/bin/pg_controldata/pg_controldata.c | 2 +
src/bin/pg_resetwal/pg_resetwal.c | 7 +
src/bin/pg_rewind/parsexlog.c | 2 +-
src/bin/pg_waldump/rmgrdesc.c | 3 +-
src/include/access/rmgr.h | 2 +-
src/include/access/rmgrlist.h | 47 +-
src/include/access/transam.h | 4 +
src/include/access/twophase.h | 2 +-
src/include/access/undoaction.h | 28 ++
src/include/access/undoaction_xlog.h | 74 +++
src/include/access/undodiscard.h | 31 ++
src/include/access/undoinsert.h | 2 +
src/include/access/undorecord.h | 15 +-
src/include/access/xact.h | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlog_internal.h | 9 +-
src/include/catalog/pg_control.h | 7 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +
src/include/postmaster/discardworker.h | 25 +
src/include/postmaster/undoloop.h | 89 ++++
src/include/postmaster/undoworker.h | 39 ++
src/include/storage/lock.h | 10 +
src/include/storage/proc.h | 2 +
54 files changed, 3112 insertions(+), 46 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/undoactiondesc.c
create mode 100644 src/backend/access/undo/undoaction.c
create mode 100644 src/backend/access/undo/undoactionxlog.c
create mode 100644 src/backend/access/undo/undodiscard.c
create mode 100644 src/backend/postmaster/discardworker.c
create mode 100644 src/backend/postmaster/undoworker.c
create mode 100644 src/include/access/undoaction.h
create mode 100644 src/include/access/undoaction_xlog.h
create mode 100644 src/include/access/undodiscard.h
create mode 100644 src/include/postmaster/discardworker.h
create mode 100644 src/include/postmaster/undoloop.h
create mode 100644 src/include/postmaster/undoworker.h
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index 91ad1ef..640d37f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = brindesc.o clogdesc.o committsdesc.o dbasedesc.o genericdesc.o \
gindesc.o gistdesc.o hashdesc.o heapdesc.o logicalmsgdesc.o \
mxactdesc.o nbtdesc.o relmapdesc.o replorigindesc.o seqdesc.o \
- smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undologdesc.o xactdesc.o xlogdesc.o
+ smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undoactiondesc.o \
+ undologdesc.o xactdesc.o xlogdesc.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/rmgrdesc/undoactiondesc.c b/src/backend/access/rmgrdesc/undoactiondesc.c
new file mode 100644
index 0000000..89343b8
--- /dev/null
+++ b/src/backend/access/rmgrdesc/undoactiondesc.c
@@ -0,0 +1,64 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactiondesc.c
+ * rmgr descriptor routines for access/undo/undoactionxlog.c
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/access/rmgrdesc/undoactiondesc.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+
+void
+undoaction_desc(StringInfo buf, XLogReaderState *record)
+{
+ char *rec = XLogRecGetData(record);
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ if (info == XLOG_UNDO_PAGE)
+ {
+ uint8 *flags = (uint8 *) rec;
+
+ appendStringInfo(buf, "page_contains_tpd_slot: %c ",
+ (*flags & XLU_PAGE_CONTAINS_TPD_SLOT) ? 'T' : 'F');
+ appendStringInfo(buf, "is_page_initialized: %c ",
+ (*flags & XLU_INIT_PAGE) ? 'T' : 'F');
+ if (*flags & XLU_PAGE_CONTAINS_TPD_SLOT)
+ {
+ xl_undoaction_page *xlrec =
+ (xl_undoaction_page *) ((char *) flags + sizeof(uint8));
+
+ appendStringInfo(buf, "urec_ptr %lu xid %u trans_slot_id %u",
+ xlrec->urec_ptr, xlrec->xid, xlrec->trans_slot_id);
+ }
+ }
+ else if (info == XLOG_UNDO_APPLY_PROGRESS)
+ {
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) rec;
+
+ appendStringInfo(buf, "urec_ptr %lu progress %u",
+ xlrec->urec_ptr, xlrec->progress);
+ }
+}
+
+const char *
+undoaction_identify(uint8 info)
+{
+ const char *id = NULL;
+
+ switch (info & ~XLR_INFO_MASK)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ id = "UNDO APPLY PROGRESS";
+ break;
+ }
+
+ return id;
+}
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 7f73251..2d4c736 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -47,7 +47,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
"tli %u; prev tli %u; fpw %s; xid %u:%u; oid %u; multi %u; offset %u; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
- "oldest running xid %u; %s",
+ "oldest running xid %u; "
+ "oldest xid having undo %u ; %s",
(uint32) (checkpoint->redo >> 32), (uint32) checkpoint->redo,
checkpoint->ThisTimeLineID,
checkpoint->PrevTimeLineID,
@@ -63,6 +64,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
checkpoint->oldestCommitTsXid,
checkpoint->newestCommitTsXid,
checkpoint->oldestActiveXid,
+ checkpoint->oldestXidHavingUndo,
(info == XLOG_CHECKPOINT_SHUTDOWN) ? "shutdown" : "online");
}
else if (info == XLOG_NEXTOID)
diff --git a/src/backend/access/transam/rmgr.c b/src/backend/access/transam/rmgr.c
index 8b05374..6238240 100644
--- a/src/backend/access/transam/rmgr.c
+++ b/src/backend/access/transam/rmgr.c
@@ -18,6 +18,7 @@
#include "access/multixact.h"
#include "access/nbtxlog.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -31,8 +32,8 @@
#include "utils/relmapper.h"
/* must be kept in sync with RmgrData definition in xlog_internal.h */
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
- { name, redo, desc, identify, startup, cleanup, mask },
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
+ { name, redo, desc, identify, startup, cleanup, mask, undo, undo_desc },
const RmgrData RmgrTable[RM_MAX_ID + 1] = {
#include "access/rmgrlist.h"
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 9a8a6bb..1e8d7c5 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -93,6 +93,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/origin.h"
#include "replication/syncrep.h"
#include "replication/walsender.h"
@@ -915,6 +916,12 @@ typedef struct TwoPhaseFileHeader
uint16 gidlen; /* length of the GID - GID follows the header */
XLogRecPtr origin_lsn; /* lsn of this record at origin node */
TimestampTz origin_timestamp; /* time of prepare at origin node */
+ /*
+ * We need the locations of start and end undo record pointers when rollbacks
+ * are to be performed for prepared transactions using zheap relations.
+ */
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
} TwoPhaseFileHeader;
/*
@@ -989,7 +996,8 @@ save_state_data(const void *data, uint32 len)
* Initializes data structure and inserts the 2PC file header record.
*/
void
-StartPrepare(GlobalTransaction gxact)
+StartPrepare(GlobalTransaction gxact, UndoRecPtr *start_urec_ptr,
+ UndoRecPtr *end_urec_ptr)
{
PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
PGXACT *pgxact = &ProcGlobal->allPgXact[gxact->pgprocno];
@@ -1027,6 +1035,10 @@ StartPrepare(GlobalTransaction gxact)
&hdr.initfileinval);
hdr.gidlen = strlen(gxact->gid) + 1; /* Include '\0' */
+ /* save the start and end undo record pointers */
+ memcpy(hdr.start_urec_ptr, start_urec_ptr, sizeof(hdr.start_urec_ptr));
+ memcpy(hdr.end_urec_ptr, end_urec_ptr, sizeof(hdr.end_urec_ptr));
+
save_state_data(&hdr, sizeof(TwoPhaseFileHeader));
save_state_data(gxact->gid, hdr.gidlen);
@@ -1452,6 +1464,9 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
RelFileNode *delrels;
int ndelrels;
SharedInvalidationMessage *invalmsgs;
+ int i;
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
/*
* Validate the GID, and lock the GXACT to ensure that two backends do not
@@ -1489,6 +1504,34 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
invalmsgs = (SharedInvalidationMessage *) bufptr;
bufptr += MAXALIGN(hdr->ninvalmsgs * sizeof(SharedInvalidationMessage));
+ /* save the start and end undo record pointers */
+ memcpy(start_urec_ptr, hdr->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(end_urec_ptr, hdr->end_urec_ptr, sizeof(end_urec_ptr));
+
+ /*
+ * Perform undo actions, if there are undologs for this transaction.
+ * We need to perform undo actions while we are still in transaction.
+ * Never push rollbacks of temp tables to undo worker.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (end_urec_ptr[i] != InvalidUndoRecPtr && !isCommit)
+ {
+ bool result = false;
+ uint64 rollback_size = 0;
+
+ if (i != UNDO_TEMP)
+ rollback_size = end_urec_ptr[i] - start_urec_ptr[i];
+
+ if (rollback_size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(end_urec_ptr[i], start_urec_ptr[i], InvalidOid);
+
+ if (!result)
+ execute_undo_actions(end_urec_ptr[i], start_urec_ptr[i], true,
+ true, false);
+ }
+ }
+
/* compute latestXid among all children */
latestXid = TransactionIdLatest(xid, hdr->nsubxacts, children);
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index fe94fda..95db516 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -284,10 +284,21 @@ SetTransactionIdLimit(TransactionId oldest_datfrozenxid, Oid oldest_datoid)
TransactionId xidStopLimit;
TransactionId xidWrapLimit;
TransactionId curXid;
+ TransactionId oldestXidHavingUndo;
Assert(TransactionIdIsNormal(oldest_datfrozenxid));
/*
+ * To determine the last safe xid that can be allocated, we need to
+ * consider oldestXidHavingUndo. The oldestXidHavingUndo will be only
+ * valid for zheap storage engine, so it won't impact any other storage
+ * engine.
+ */
+ oldestXidHavingUndo = pg_atomic_read_u32(&ProcGlobal->oldestXidHavingUndo);
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ oldest_datfrozenxid = Min(oldest_datfrozenxid, oldestXidHavingUndo);
+
+ /*
* The place where we actually get into deep trouble is halfway around
* from the oldest potentially-existing XID. (This calculation is
* probably off by one or two counts, because the special XIDs reduce the
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 8af75a9..805b7a1 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -42,6 +42,7 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/origin.h"
@@ -280,6 +281,20 @@ typedef struct SubXactCallbackItem
static SubXactCallbackItem *SubXact_callbacks = NULL;
+/* Location in undo log from where to start applying the undo actions. */
+static UndoRecPtr UndoActionStartPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+/* Location in undo log up to which undo actions need to be applied. */
+static UndoRecPtr UndoActionEndPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+/* Do we need to perform any undo actions? */
+static bool PerformUndoActions = false;
/* local function prototypes */
static void AssignTransactionId(TransactionState s);
@@ -1835,6 +1850,7 @@ StartTransaction(void)
{
TransactionState s;
VirtualTransactionId vxid;
+ int i;
/*
* Let's just make sure the state stack is empty
@@ -1913,6 +1929,13 @@ StartTransaction(void)
nUnreportedXids = 0;
s->didLogXid = false;
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
/*
* must initialize resource-management stuff first
*/
@@ -2222,7 +2245,7 @@ CommitTransaction(void)
* NB: if you change this routine, better look at CommitTransaction too!
*/
static void
-PrepareTransaction(void)
+PrepareTransaction(UndoRecPtr *start_urec_ptr, UndoRecPtr *end_urec_ptr)
{
TransactionState s = CurrentTransactionState;
TransactionId xid = GetCurrentTransactionId();
@@ -2370,7 +2393,7 @@ PrepareTransaction(void)
* PREPARED; in particular, pay attention to whether things should happen
* before or after releasing the transaction's locks.
*/
- StartPrepare(gxact);
+ StartPrepare(gxact, start_urec_ptr, end_urec_ptr);
AtPrepare_Notify();
AtPrepare_Locks();
@@ -2493,6 +2516,65 @@ PrepareTransaction(void)
RESUME_INTERRUPTS();
}
+static void
+AtAbort_Rollback(void)
+{
+ TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i;
+
+ /* XXX: TODO: check this logic, which was moved out of UserAbortTransactionBlock */
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (CurrentTransactionState->state == TRANS_INPROGRESS)
+ {
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (latest_urec_ptr[i])
+ {
+ if (i == UNDO_TEMP)
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ false, true, true);
+ else
+ {
+ uint64 size = latest_urec_ptr[i] - s->start_urec_ptr[i];
+ bool result = false;
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions
+ * later.
+ */
+ if (size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ true, true, true);
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
+}
/*
* AbortTransaction
@@ -2594,6 +2676,7 @@ AbortTransaction(void)
*/
AfterTriggerEndXact(false); /* 'false' means it's abort */
AtAbort_Portals();
+ AtAbort_Rollback();
AtEOXact_LargeObject(false);
AtAbort_Notify();
AtEOXact_RelationMap(false, is_parallel_worker);
@@ -2803,6 +2886,12 @@ void
CommitTransactionCommand(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i;
+
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(end_urec_ptr, s->latest_urec_ptr, sizeof(end_urec_ptr));
switch (s->blockState)
{
@@ -2892,7 +2981,7 @@ CommitTransactionCommand(void)
* return to the idle state.
*/
case TBLOCK_PREPARE:
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, end_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
break;
@@ -2938,6 +3027,23 @@ CommitTransactionCommand(void)
{
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
+
+ /*
+ * Update the end undo record pointer if it's not valid with
+ * the currently popped transaction's end undo record pointer.
+ * This is particularly required when the first command of
+ * the transaction is of type which does not require an undo,
+ * e.g. savepoint x.
+ * Accordingly, update the start undo record pointer.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(end_urec_ptr[i]))
+ end_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(s->start_urec_ptr[i]))
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
} while (s->blockState == TBLOCK_SUBCOMMIT);
/* If we had a COMMIT command, finish off the main xact too */
if (s->blockState == TBLOCK_END)
@@ -2949,7 +3055,7 @@ CommitTransactionCommand(void)
else if (s->blockState == TBLOCK_PREPARE)
{
Assert(s->parent == NULL);
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, end_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
}
else
@@ -3043,6 +3149,18 @@ void
AbortCurrentTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
+
+ /*
+ * The undo actions are allowed to be executed at the end of statement
+ * execution when we are not in transaction block, otherwise they are
+ * executed when user explicitly ends the transaction.
+ *
+ * So if we are in a transaction block don't set the PerformUndoActions
+ * because this flag will be set when user explicitly issue rollback or
+ * rollback to savepoint.
+ */
+ PerformUndoActions = false;
switch (s->blockState)
{
@@ -3077,6 +3195,16 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * We are outside the transaction block so remember the required
+ * information to perform undo actions and also set the
+ * PerformUndoActions so that we execute it before completing this
+ * command.
+ */
+ PerformUndoActions = true;
+ memcpy (UndoActionStartPtr, s->latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
break;
/*
@@ -3113,6 +3241,9 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3132,6 +3263,9 @@ AbortCurrentTransaction(void)
case TBLOCK_ABORT_END:
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3142,6 +3276,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Failed while executing the rollback command, need perform any
+ * pending undo actions.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3153,6 +3293,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Perform any pending actions if failed while preparing the
+ * transaction.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3175,6 +3321,17 @@ AbortCurrentTransaction(void)
case TBLOCK_SUBCOMMIT:
case TBLOCK_SUBABORT_PENDING:
case TBLOCK_SUBRESTART:
+ /*
+ * If we are here and still UndoActionStartPtr is valid that means
+ * the subtransaction failed while executing the undo action, so
+ * store its undo action start point in parent so that parent can
+ * start its undo action from this point.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ s->parent->latest_urec_ptr[i] = UndoActionStartPtr[i];
+ }
AbortSubTransaction();
CleanupSubTransaction();
AbortCurrentTransaction();
@@ -3192,6 +3349,109 @@ AbortCurrentTransaction(void)
}
/*
+ * XactPerformUndoActionsIfPending - Execute pending undo actions.
+ *
+ * If the parent transaction state is valid (when there is an error in the
+ * subtransaction and rollback to savepoint is executed), then allow to
+ * perform undo actions in it, otherwise perform them in a new transaction.
+ */
+void
+XactPerformUndoActionsIfPending()
+{
+ TransactionState s = CurrentTransactionState;
+ uint64 rollback_size = 0;
+ bool new_xact = true, result = false, no_pending_action = true;
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ if (!PerformUndoActions)
+ return;
+
+ /* If there is no undo log for any persistence level, then return. */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ {
+ no_pending_action = false;
+ break;
+ }
+ }
+
+ if (no_pending_action)
+ {
+ PerformUndoActions = false;
+ return;
+ }
+
+ /*
+ * Execute undo actions under parent transaction, if any. Otherwise start
+ * a new transaction.
+ */
+ if (GetTopTransactionIdIfAny() != InvalidTransactionId)
+ {
+ memcpy(parent_latest_urec_ptr, s->latest_urec_ptr,
+ sizeof (parent_latest_urec_ptr));
+ new_xact = false;
+ }
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions later.
+ * Never push the rollbacks for temp tables.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ continue;
+
+ if (i == UNDO_TEMP)
+ goto perform_rollback;
+ else
+ rollback_size = UndoActionStartPtr[i] - UndoActionEndPtr[i];
+
+ if (new_xact && rollback_size > rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+perform_rollback:
+ if (new_xact)
+ {
+ TransactionState xact;
+
+ /* Start a new transaction for performing the rollback */
+ StartTransactionCommand();
+ xact = CurrentTransactionState;
+
+ /*
+ * Store the previous transactions start and end undo record
+ * pointers into this transaction's state so that if there is
+ * some error while performing undo actions we can restart
+ * from begining.
+ */
+ memcpy(xact->start_urec_ptr, UndoActionEndPtr,
+ sizeof(UndoActionEndPtr));
+ memcpy(xact->latest_urec_ptr, UndoActionStartPtr,
+ sizeof(UndoActionStartPtr));
+ }
+
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ new_xact, true, true);
+
+ if (new_xact)
+ CommitTransactionCommand();
+ else
+ {
+ /* Restore parent's state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+ }
+
+ PerformUndoActions = false;
+}
+
+/*
* PreventInTransactionBlock
*
* This routine is to be called by statements that must not run inside
@@ -3592,6 +3852,10 @@ EndTransactionBlock(void)
{
TransactionState s = CurrentTransactionState;
bool result = false;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i ;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
switch (s->blockState)
{
@@ -3637,6 +3901,16 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+
+ /*
+ * We are calculating latest_urec_ptr, even though its a commit
+ * case. This is to handle any error during the commit path.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_END;
@@ -3662,6 +3936,11 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if(!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_ABORT_PENDING;
@@ -3714,6 +3993,18 @@ EndTransactionBlock(void)
break;
}
+ /*
+ * We need to perform undo actions if the transaction is failed. Remember
+ * the required information to perform undo actions at the end of
+ * statement execution.
+ */
+ if (!result)
+ PerformUndoActions = true;
+
+ memcpy(UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy(UndoActionEndPtr, TopTransactionStateData.start_urec_ptr,
+ sizeof(UndoActionEndPtr));
+
return result;
}
@@ -3971,6 +4262,12 @@ ReleaseSavepoint(const char *name)
TransactionState s = CurrentTransactionState;
TransactionState target,
xact;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4064,8 +4361,32 @@ ReleaseSavepoint(const char *name)
if (xact == target)
break;
xact = xact->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = xact->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(xact->start_urec_ptr[i]))
+ start_urec_ptr[i] = xact->start_urec_ptr[i];
+ }
+
Assert(PointerIsValid(xact));
}
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction. Also set the start_urec_ptr if
+ * parent start_urec_ptr is not valid.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ xact->parent->latest_urec_ptr[i] = latest_urec_ptr[i];
+ if (!UndoRecPtrIsValid(xact->parent->start_urec_ptr[i]))
+ xact->parent->start_urec_ptr[i] = start_urec_ptr[i];
+ }
}
/*
@@ -4276,6 +4597,7 @@ void
ReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4294,6 +4616,22 @@ ReleaseCurrentSubTransaction(void)
BlockStateAsString(s->blockState));
Assert(s->state == TRANS_INPROGRESS);
MemoryContextSwitchTo(CurTransactionContext);
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ s->parent->latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (!UndoRecPtrIsValid(s->parent->start_urec_ptr[i]))
+ s->parent->start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
Assert(s->state == TRANS_INPROGRESS);
@@ -4310,6 +4648,14 @@ void
RollbackAndReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i;
/*
* Unlike ReleaseCurrentSubTransaction(), this is nominally permitted
@@ -4356,6 +4702,19 @@ RollbackAndReleaseCurrentSubTransaction(void)
if (s->blockState == TBLOCK_SUBINPROGRESS)
AbortSubTransaction();
+ /*
+ * Remember the required information to perform undo actions before
+ * cleaning up the subtransaction state.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ {
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+ }
+
/* And clean it up, too */
CleanupSubTransaction();
@@ -4364,6 +4723,30 @@ RollbackAndReleaseCurrentSubTransaction(void)
s->blockState == TBLOCK_INPROGRESS ||
s->blockState == TBLOCK_IMPLICIT_INPROGRESS ||
s->blockState == TBLOCK_STARTED);
+
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ parent_latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ /*
+ * Store the undo action start point in the parent state so that
+ * we can apply undo actions these undos also during rollback of
+ * parent transaction in case of error while applying the undo
+ * actions.
+ */
+ s->latest_urec_ptr[i] = latest_urec_ptr[i];
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false,
+ true, true);
+
+ /* Restore parent state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+
+ /* Successfully performed undo actions so reset the flag. */
+ PerformUndoActions = false;
}
/*
@@ -4577,6 +4960,7 @@ static void
StartSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
if (s->state != TRANS_DEFAULT)
elog(WARNING, "StartSubTransaction while in %s state",
@@ -4594,6 +4978,13 @@ StartSubTransaction(void)
AtSubStart_Notify();
AfterTriggerBeginSubXact();
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
s->state = TRANS_INPROGRESS;
/*
@@ -4717,6 +5108,47 @@ CommitSubTransaction(void)
PopTransaction();
}
+static void
+AtSubAbort_Rollback(TransactionState s)
+{
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ /* XXX: TODO: Check this logic, which was moved out of RollbackToSavepoint() */
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (s->state == TRANS_INPROGRESS)
+ {
+ for ( i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false, true, false);
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
+}
+
/*
* AbortSubTransaction
*/
@@ -4811,6 +5243,7 @@ AbortSubTransaction(void)
s->parent->subTransactionId,
s->curTransactionOwner,
s->parent->curTransactionOwner);
+ AtSubAbort_Rollback(s);
AtEOSubXact_LargeObject(false, s->subTransactionId,
s->parent->subTransactionId);
AtSubAbort_Notify();
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index c4c5ab4..d124499 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -5169,6 +5169,7 @@ BootStrapXLOG(void)
checkPoint.newestCommitTsXid = InvalidTransactionId;
checkPoint.time = (pg_time_t) time(NULL);
checkPoint.oldestActiveXid = InvalidTransactionId;
+ checkPoint.oldestXidHavingUndo = InvalidTransactionId;
ShmemVariableCache->nextXid = checkPoint.nextXid;
ShmemVariableCache->nextOid = checkPoint.nextOid;
@@ -6610,6 +6611,10 @@ StartupXLOG(void)
(errmsg_internal("commit timestamp Xid oldest/newest: %u/%u",
checkPoint.oldestCommitTsXid,
checkPoint.newestCommitTsXid)));
+ ereport(DEBUG1,
+ (errmsg_internal("oldest xid having undo: %u",
+ checkPoint.oldestXidHavingUndo)));
+
if (!TransactionIdIsNormal(checkPoint.nextXid))
ereport(PANIC,
(errmsg("invalid next transaction ID")));
@@ -6627,6 +6632,10 @@ StartupXLOG(void)
XLogCtl->ckptXidEpoch = checkPoint.nextXidEpoch;
XLogCtl->ckptXid = checkPoint.nextXid;
+ /* Read oldest xid having undo from checkpoint and set in proc global. */
+ pg_atomic_write_u32(&ProcGlobal->oldestXidHavingUndo,
+ checkPoint.oldestXidHavingUndo);
+
/*
* Initialize replication slots, before there's a chance to remove
* required resources.
@@ -8768,6 +8777,9 @@ CreateCheckPoint(int flags)
checkPoint.nextOid += ShmemVariableCache->oidCount;
LWLockRelease(OidGenLock);
+ checkPoint.oldestXidHavingUndo =
+ pg_atomic_read_u32(&ProcGlobal->oldestXidHavingUndo);
+
MultiXactGetCheckptMulti(shutdown,
&checkPoint.nextMulti,
&checkPoint.nextMultiOffset,
@@ -9678,6 +9690,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceOldest(checkPoint.oldestMulti,
checkPoint.oldestMultiDB);
+ pg_atomic_write_u32(&ProcGlobal->oldestXidHavingUndo,
+ checkPoint.oldestXidHavingUndo);
+
/*
* No need to set oldestClogXid here as well; it'll be set when we
* redo an xl_clog_truncate if it changed since initialization.
@@ -9736,6 +9751,8 @@ xlog_redo(XLogReaderState *record)
/* ControlFile->checkPointCopy always tracks the latest ckpt XID */
ControlFile->checkPointCopy.nextXidEpoch = checkPoint.nextXidEpoch;
ControlFile->checkPointCopy.nextXid = checkPoint.nextXid;
+ ControlFile->checkPointCopy.oldestXidHavingUndo =
+ checkPoint.oldestXidHavingUndo;
/* Update shared-memory copy of checkpoint XID/epoch */
SpinLockAcquire(&XLogCtl->info_lck);
@@ -9785,6 +9802,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceNextMXact(checkPoint.nextMulti,
checkPoint.nextMultiOffset);
+ pg_atomic_write_u32(&ProcGlobal->oldestXidHavingUndo,
+ checkPoint.oldestXidHavingUndo);
+
/*
* NB: This may perform multixact truncation when replaying WAL
* generated by an older primary.
diff --git a/src/backend/access/undo/Makefile b/src/backend/access/undo/Makefile
index f41e8f7..fdf7f7d 100644
--- a/src/backend/access/undo/Makefile
+++ b/src/backend/access/undo/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/access/undo
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = undoinsert.o undolog.o undorecord.o
+OBJS = undoaction.o undoactionxlog.o undodiscard.o undoinsert.o undolog.o undorecord.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/undo/undoaction.c b/src/backend/access/undo/undoaction.c
new file mode 100644
index 0000000..230c12a
--- /dev/null
+++ b/src/backend/access/undo/undoaction.c
@@ -0,0 +1,688 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.c
+ * execute undo actions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undoaction.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/undoaction.h"
+#include "access/undoaction_xlog.h"
+#include "access/undolog.h"
+#include "access/undorecord.h"
+#include "access/visibilitymap.h"
+#include "access/xact.h"
+#include "access/xloginsert.h"
+#include "access/xlog_internal.h"
+#include "nodes/pg_list.h"
+#include "pgstat.h"
+#include "postmaster/undoloop.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "utils/rel.h"
+#include "utils/relfilenodemap.h"
+#include "miscadmin.h"
+#include "storage/shmem.h"
+#include "access/undodiscard.h"
+
+#define ROLLBACK_HT_SIZE 1024
+
+static bool execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr,
+ Oid reloid, TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool norellock, int options);
+static void RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr);
+
+/* This is the hash table to store all the rollabck requests. */
+static HTAB *RollbackHT;
+
+/*
+ * execute_undo_actions - Execute the undo actions
+ *
+ * from_urecptr - undo record pointer from where to start applying undo action.
+ * to_urecptr - undo record pointer upto which point apply undo action.
+ * nopartial - true if rollback is for complete transaction.
+ * rewind - whether to rewind the insert location of the undo log or not.
+ * Only the backend executed the transaction can rewind, but
+ * any other process e.g. undo worker should not rewind it.
+ * Because, if the backend have already inserted new undo records
+ * for the next transaction and if we rewind then we will loose
+ * the undo record inserted for the new transaction.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ */
+void
+execute_undo_actions(UndoRecPtr from_urecptr, UndoRecPtr to_urecptr,
+ bool nopartial, bool rewind, bool rellock)
+{
+ UnpackedUndoRecord *uur = NULL;
+ UndoRecPtr urec_ptr, prev_urec_ptr;
+ UndoRecPtr save_urec_ptr;
+ Oid prev_reloid = InvalidOid;
+ ForkNumber prev_fork = InvalidForkNumber;
+ BlockNumber prev_block = InvalidBlockNumber;
+ List *luinfo = NIL;
+ bool more_undo;
+ int options = 0;
+ TransactionId xid = InvalidTransactionId;
+ UndoRecInfo *urec_info;
+
+ Assert(from_urecptr != InvalidUndoRecPtr);
+ Assert(UndoRecPtrGetLogNo(from_urecptr) != UndoRecPtrGetLogNo(to_urecptr) ||
+ from_urecptr >= to_urecptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ * FIXME: this won't work if undolog crossed the limit of 1TB, because
+ * then from_urecptr and to_urecptr will be from different lognos.
+ */
+ if (to_urecptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(from_urecptr);
+ to_urecptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ save_urec_ptr = urec_ptr = from_urecptr;
+
+ if (nopartial)
+ {
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber, InvalidOffsetNumber,
+ InvalidTransactionId, NULL, NULL);
+ if (uur == NULL)
+ return;
+
+ xid = uur->uur_xid;
+ UndoRecordRelease(uur);
+ uur = NULL;
+
+ /*
+ * Grab the undo action apply lock before start applying the undo action
+ * this will prevent applying undo actions concurrently. If we do not
+ * get the lock that mean its already being applied concurrently or the
+ * discard worker might be pushing its request to the rollback hash
+ * table
+ */
+ if (!ConditionTransactionUndoActionLock(xid))
+ return;
+ }
+
+ prev_urec_ptr = InvalidUndoRecPtr;
+ while (prev_urec_ptr != to_urecptr)
+ {
+ Oid reloid = InvalidOid;
+ uint16 urec_prevlen;
+ UndoRecPtr urec_prevurp;
+ bool non_page;
+
+ more_undo = true;
+
+ prev_urec_ptr = urec_ptr;
+
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId, NULL, NULL);
+
+ /* If there is no info block, this is not a page-based undo record. */
+ non_page = uur && !(uur->uur_info & UREC_INFO_BLOCK);
+
+ if (uur != NULL && !non_page)
+ reloid = uur->uur_reloid;
+
+ xid = uur->uur_xid;
+
+ if (non_page)
+ {
+ prev_reloid = InvalidOid;
+ urec_prevlen = uur->uur_prevlen;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /*
+ * Execute individual undo actions not associated with a page
+ * immediately.
+ */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->uur = uur;
+ urec_info->urp = urec_ptr;
+ luinfo = lappend(luinfo, urec_info);
+ execute_undo_actions_page(luinfo, urec_ptr, reloid, xid,
+ InvalidBlockNumber, false, rellock,
+ 0);
+ pfree(urec_info);
+ urec_info = NULL;
+ list_free(luinfo);
+ luinfo = NIL;
+ UndoRecordRelease(uur);
+
+ /* Follow undo chain until to_urecptr. */
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ /*
+ * If the record is already discarded by undo worker or if the relation
+ * is dropped or truncated, then we cannot fetch record successfully.
+ * Hence, skip quietly.
+ *
+ * Note: reloid remains InvalidOid for a discarded record.
+ */
+ else if (!OidIsValid(reloid))
+ {
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ urec_prevlen = uur->uur_prevlen;
+
+ /* Release the just-fetched record */
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else if (!OidIsValid(prev_reloid) ||
+ (prev_reloid == reloid &&
+ prev_fork == uur->uur_fork &&
+ prev_block == uur->uur_block))
+ {
+ /* Collect the undo records that belong to the same page. */
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+
+ luinfo = lappend(luinfo, urec_info);
+ urec_prevlen = uur->uur_prevlen;
+ urec_prevurp = uur->uur_prevurp;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else
+ {
+ more_undo = true;
+ }
+
+ /*
+ * If no more undo is left to be processed and we are rolling back the
+ * complete transaction, then we can consider that the undo chain for a
+ * block is complete.
+ * If the previous undo pointer in the page is invalid, then also the
+ * undo chain for the current block is completed.
+ */
+ if (luinfo &&
+ ((!more_undo && nopartial) || !UndoRecPtrIsValid(save_urec_ptr)))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, true, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+ else if (luinfo)
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, false, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ /*
+ * There are still more records to process, so keep moving backwards
+ * in the chain.
+ */
+ if (more_undo)
+ {
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+ luinfo = lappend(luinfo, urec_info);
+
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /*
+ * Continue to process the records if this is not the last undo
+ * record in chain.
+ */
+ urec_prevlen = uur->uur_prevlen;
+ urec_prevurp = uur->uur_prevurp;
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ else
+ break;
+ }
+ else
+ break;
+ }
+
+ /* Apply the undo actions for the remaining records. */
+ if (list_length(luinfo))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, nopartial ? true : false,
+ rellock, options);
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+ }
+
+ if (rewind)
+ {
+ /* Read the prevlen from the first record of this transaction. */
+ uur = UndoFetchRecord(to_urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If undo is already discarded before we rewind, then do nothing.
+ */
+ if (uur == NULL)
+ return;
+
+
+ /*
+ * Rewind the insert location to start of this transaction. This is
+ * to avoid reapplying some intermediate undo. We do not need to wal
+ * log this information here, because if the system crash before we
+ * rewind the insert pointer then after recovery we can identify
+ * whether the undo is already applied or not from the slot undo record
+ * pointer. Also set the correct prevlen value (what we have fetched
+ * from the undo).
+ */
+ UndoLogRewind(to_urecptr, uur->uur_prevlen);
+
+ UndoRecordRelease(uur);
+ }
+
+ if (nopartial)
+ {
+ /*
+ * Set undo action apply completed in the transaction header if this is
+ * a main transaction and we have not rewound its undo.
+ */
+ if (!rewind)
+ {
+ /*
+ * Undo action is applied so delete the hash table entry and release
+ * the undo action lock.
+ */
+ RollbackHTRemoveEntry(from_urecptr);
+
+ /*
+ * Prepare and update the progress of the undo action apply in the
+ * transaction header.
+ */
+ PrepareUpdateUndoActionProgress(to_urecptr, 1);
+
+ START_CRIT_SECTION();
+
+ /* Update the progress in the transaction header. */
+ UndoRecordUpdateTransInfo(0);
+
+ /* WAL log the undo apply progress. */
+ {
+ xl_undoapply_progress xlrec;
+
+ xlrec.urec_ptr = to_urecptr;
+ xlrec.progress = 1;
+
+ /*
+ * FIXME : We need to register undo buffers and set LSN for them
+ * that will be required for FPW of the undo buffers.
+ */
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, sizeof(xlrec));
+
+ (void) XLogInsert(RM_UNDOACTION_ID, XLOG_UNDO_APPLY_PROGRESS);
+ }
+
+ END_CRIT_SECTION();
+ UnlockReleaseUndoBuffers();
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+}
+
+/*
+ * execute_undo_actions_page - Execute the undo actions for a page
+ *
+ * After applying all the undo actions for a page, we clear the transaction
+ * slot on a page if the undo chain for block is complete, otherwise just
+ * rewind the undo pointer to the last record for that block that precedes
+ * the last undo record for which action is replayed.
+ *
+ * luinfo - list of undo records (along with their location) for which undo
+ * action needs to be replayed.
+ * urec_ptr - undo record pointer to which we need to rewind.
+ * reloid - OID of relation on which undo actions needs to be applied.
+ * blkno - block number on which undo actions needs to be applied.
+ * blk_chain_complete - indicates whether the undo chain for block is
+ * complete.
+ * nopartial - true if rollback is for complete transaction. If we are not
+ * rolling back the complete transaction then we need to apply the
+ * undo action for UNDO_INVALID_XACT_SLOT also because in such
+ * case we will rewind the insert undo location.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ * options - options for executing undo actions.
+ *
+ * returns true, if successfully applied the undo actions, otherwise, false.
+ */
+static bool
+execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock, int options)
+{
+ UndoRecInfo *first;
+
+ /*
+ * All records passed to us are for the same RMGR, so we just use the
+ * first record to dispatch.
+ */
+ Assert(luinfo != NIL);
+ first = (UndoRecInfo *) linitial(luinfo);
+
+ return RmgrTable[first->uur->uur_rmid].rm_undo(luinfo, urec_ptr, reloid,
+ xid, blkno,
+ blk_chain_complete, rellock,
+ options);
+}
+
+/*
+ * To return the size of the hash-table for rollbacks.
+ */
+int
+RollbackHTSize(void)
+{
+ return hash_estimate_size(ROLLBACK_HT_SIZE, sizeof(RollbackHashEntry));
+}
+
+/*
+ * To initialize the hash-table for rollbacks in shared memory
+ * for the given size.
+ */
+void
+InitRollbackHashTable(void)
+{
+ HASHCTL info;
+ MemSet(&info, 0, sizeof(info));
+
+ info.keysize = sizeof(UndoRecPtr);
+ info.entrysize = sizeof(RollbackHashEntry);
+ info.hash = tag_hash;
+
+ RollbackHT = ShmemInitHash("Undo actions Lookup Table",
+ ROLLBACK_HT_SIZE, ROLLBACK_HT_SIZE, &info,
+ HASH_ELEM | HASH_FUNCTION | HASH_FIXED_SIZE);
+}
+
+/*
+ * To push the rollback requests from backend to the hash-table.
+ * Return true if the request is successfully added, else false
+ * and the caller may execute undo actions itself.
+ */
+bool
+PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr, Oid dbid)
+{
+ bool found = false;
+ RollbackHashEntry *rh;
+
+ Assert(UndoRecPtrGetLogNo(start_urec_ptr) != UndoRecPtrGetLogNo(end_urec_ptr) ||
+ start_urec_ptr >= end_urec_ptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ */
+ if (start_urec_ptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(end_urec_ptr);
+ start_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ Assert(UndoRecPtrIsValid(start_urec_ptr));
+
+ /* If there is no space to accomodate new request, then we can't proceed. */
+ if (RollbackHTIsFull())
+ return false;
+
+ if(!UndoRecPtrIsValid(end_urec_ptr))
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(start_urec_ptr);
+ end_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ rh = (RollbackHashEntry *) hash_search(RollbackHT, &start_urec_ptr,
+ HASH_ENTER_NULL, &found);
+ if (!rh)
+ {
+ LWLockRelease(RollbackHTLock);
+ return false;
+ }
+ /* We shouldn't try to push the same rollback request again. */
+ if (!found)
+ {
+ rh->start_urec_ptr = start_urec_ptr;
+ rh->end_urec_ptr = end_urec_ptr;
+ rh->dbid = (dbid == InvalidOid) ? MyDatabaseId : dbid;
+ }
+ LWLockRelease(RollbackHTLock);
+
+ return true;
+}
+
+/*
+ * To perform the undo actions for the transactions whose rollback
+ * requests are in hash table. Sequentially, scan the hash-table
+ * and perform the undo-actions for the respective transactions.
+ * Once, the undo-actions are applied, remove the entry from the
+ * hash table.
+ */
+void
+RollbackFromHT(Oid dbid)
+{
+ UndoRecPtr start[ROLLBACK_HT_SIZE];
+ UndoRecPtr end[ROLLBACK_HT_SIZE];
+ RollbackHashEntry *rh;
+ HASH_SEQ_STATUS status;
+ int i = 0;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ Assert(hash_get_num_entries(RollbackHT) <= ROLLBACK_HT_SIZE);
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ {
+ if (rh->dbid == dbid)
+ {
+ start[i] = rh->start_urec_ptr;
+ end[i] = rh->end_urec_ptr;
+ i++;
+ }
+ }
+
+ LWLockRelease(RollbackHTLock);
+
+ /* Execute the rollback requests */
+ while(--i >= 0)
+ {
+ Assert(UndoRecPtrIsValid(start[i]));
+ Assert(UndoRecPtrIsValid(end[i]));
+
+ StartTransactionCommand();
+ execute_undo_actions(start[i], end[i], true, false, true);
+ CommitTransactionCommand();
+ }
+}
+
+/*
+ * Remove the rollback request entry from the rollback hash table.
+ */
+static void
+RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr)
+{
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ hash_search(RollbackHT, &start_urec_ptr, HASH_REMOVE, NULL);
+
+ LWLockRelease(RollbackHTLock);
+}
+
+/*
+ * To check if the rollback requests in the hash table are all
+ * completed or not. This is required because we don't not want to
+ * expose RollbackHT in xact.c, where it is required to ensure
+ * that we push the resuests only when there is some space in
+ * the hash-table.
+ */
+bool
+RollbackHTIsFull(void)
+{
+ bool result = false;
+
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ if (hash_get_num_entries(RollbackHT) >= ROLLBACK_HT_SIZE)
+ result = true;
+
+ LWLockRelease(RollbackHTLock);
+
+ return result;
+}
+
+/*
+ * Get database list from the rollback hash table.
+ */
+List *
+RollbackHTGetDBList()
+{
+ HASH_SEQ_STATUS status;
+ RollbackHashEntry *rh;
+ List *dblist = NIL;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ dblist = list_append_unique_oid(dblist, rh->dbid);
+
+ LWLockRelease(RollbackHTLock);
+
+ return dblist;
+}
+
+/*
+ * ConditionTransactionUndoActionLock
+ *
+ * Insert a lock showing that the undo action for given transaction is in
+ * progress. This is only done for the main transaction not for the
+ * sub-transaction.
+ */
+bool
+ConditionTransactionUndoActionLock(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ if (LOCKACQUIRE_NOT_AVAIL == LockAcquire(&tag, ExclusiveLock, false, true))
+ return false;
+ else
+ return true;
+}
+
+/*
+ * TransactionUndoActionLockRelease
+ *
+ * Delete the lock showing that the undo action given transaction ID is in
+ * progress.
+ */
+void
+TransactionUndoActionLockRelease(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ LockRelease(&tag, ExclusiveLock, false);
+}
diff --git a/src/backend/access/undo/undoactionxlog.c b/src/backend/access/undo/undoactionxlog.c
new file mode 100644
index 0000000..253d8f1
--- /dev/null
+++ b/src/backend/access/undo/undoactionxlog.c
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactionxlog.c
+ * WAL replay logic for undo actions.
+ *
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/undo/undoactionxlog.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+#include "access/undoinsert.h"
+#include "access/visibilitymap.h"
+#include "access/xlog.h"
+#include "access/xlogutils.h"
+
+/*
+ * Replay of undo apply progress.
+ */
+static void
+undo_xlog_apply_progress(XLogReaderState *record)
+{
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) XLogRecGetData(record);
+
+ /* Update the progress in the transaction header. */
+ PrepareUpdateUndoActionProgress(xlrec->urec_ptr, xlrec->progress);
+ UndoRecordUpdateTransInfo(0);
+}
+
+void
+undoaction_redo(XLogReaderState *record)
+{
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ switch (info)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ undo_xlog_apply_progress(record);
+ break;
+ default:
+ elog(PANIC, "undoaction_redo: unknown op code %u", info);
+ }
+}
diff --git a/src/backend/access/undo/undodiscard.c b/src/backend/access/undo/undodiscard.c
new file mode 100644
index 0000000..afe5834
--- /dev/null
+++ b/src/backend/access/undo/undodiscard.c
@@ -0,0 +1,447 @@
+/*-------------------------------------------------------------------------
+ *
+ * undodiscard.c
+ * discard undo records
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undodiscard.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/xlog.h"
+#include "access/xact.h"
+#include "access/undolog.h"
+#include "access/undodiscard.h"
+#include "catalog/pg_tablespace.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "storage/shmem.h"
+#include "storage/proc.h"
+#include "utils/resowner.h"
+#include "postmaster/undoloop.h"
+
+static UndoRecPtr FetchLatestUndoPtrForXid(UndoRecPtr urecptr,
+ UnpackedUndoRecord *uur_start,
+ UndoLogControl *log);
+
+/*
+ * Discard the undo for the log
+ *
+ * Search the undo log, get the start record for each transaction until we get
+ * the transaction with xid >= xmin or an invalid xid. Then call undolog
+ * routine to discard upto that point and update the memory structure for the
+ * log slot. We set the hibernate flag if we do not have any undo logs, this
+ * flag is passed to the undo worker wherein it determines if system is idle
+ * and it should sleep for sometime.
+ *
+ * Return the oldest xid remaining in this undo log (which should be >= xmin,
+ * since we'll discard everything older). Return InvalidTransactionId if the
+ * undo log is empty.
+ */
+static TransactionId
+UndoDiscardOneLog(UndoLogControl *log, TransactionId xmin, bool *hibernate)
+{
+ UndoRecPtr undo_recptr, next_insert, from_urecptr;
+ UndoRecPtr next_urecptr = InvalidUndoRecPtr;
+ UnpackedUndoRecord *uur = NULL;
+ bool need_discard = false;
+ bool log_complete = false;
+ TransactionId undoxid = InvalidTransactionId;
+ TransactionId xid = log->oldest_xid;
+ TransactionId latest_discardxid = InvalidTransactionId;
+
+ undo_recptr = log->oldest_data;
+
+ /* There might not be any undo log and hibernation might be needed. */
+ *hibernate = true;
+
+ /* Loop until we run out of discardable transactions. */
+ do
+ {
+ bool pending_abort = false;
+
+ next_insert = UndoLogGetNextInsertPtr(log->logno, xid);
+
+ /*
+ * If the next insert location in the undo log is same as the oldest
+ * data for the log then there is nothing more to discard in this log
+ * so discard upto this point.
+ */
+ if (next_insert == undo_recptr)
+ {
+ /*
+ * If the discard location and the insert location is same then
+ * there is nothing to discard.
+ */
+ if (undo_recptr == log->oldest_data)
+ break;
+ else
+ log_complete = true;
+ }
+ else
+ {
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+
+ Assert(uur != NULL);
+
+ if (!TransactionIdDidCommit(uur->uur_xid) &&
+ TransactionIdPrecedes(uur->uur_xid, xmin) &&
+ uur->uur_progress == 0)
+ {
+ /*
+ * At the time of recovery, we might not have a valid next undo
+ * record pointer and in that case we'll calculate the location
+ * of from pointer using the last record of next insert
+ * location.
+ */
+ if (ConditionTransactionUndoActionLock(uur->uur_xid))
+ {
+ TransactionId xid = uur->uur_xid;
+ UndoLogControl *log = NULL;
+ UndoLogNumber logno;
+
+ logno = UndoRecPtrGetLogNo(undo_recptr);
+ log = UndoLogGet(logno, false);
+
+ /*
+ * If the corresponding log got rewinded to a location
+ * prior to undo_recptr, the undo actions are already
+ * applied.
+ */
+ if (log->meta.insert > undo_recptr)
+ {
+ UndoRecordRelease(uur);
+
+ /* Fetch the undo record under undo action lock. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the undo actions for the aborted transaction is
+ * already applied then continue discarding the undo log
+ * otherwise discard till current point and stop processing
+ * this undo log.
+ * Also, check this is indeed the transaction id we're
+ * looking for. It is possible that after rewinding
+ * some other transaction has inserted an undo record.
+ */
+ if (uur->uur_xid == xid && uur->uur_progress == 0)
+ {
+ from_urecptr = FetchLatestUndoPtrForXid(undo_recptr, uur, log);
+ (void)PushRollbackReq(from_urecptr, undo_recptr, uur->uur_dbid);
+ pending_abort = true;
+ }
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+ else
+ pending_abort = true;
+ }
+
+ next_urecptr = uur->uur_next;
+ undoxid = uur->uur_xid;
+ xid = undoxid;
+ }
+
+ /* we can discard upto this point. */
+ if (TransactionIdFollowsOrEquals(undoxid, xmin) ||
+ next_urecptr == SpecialUndoRecPtr ||
+ UndoRecPtrGetLogNo(next_urecptr) != log->logno ||
+ log_complete || pending_abort)
+ {
+ /* Hey, I got some undo log to discard, can not hibernate now. */
+ *hibernate = false;
+
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /*
+ * If Transaction id is smaller than the xmin that means this must
+ * be the last transaction in this undo log, so we need to get the
+ * last insert point in this undo log and discard till that point.
+ * Also, if the transaction has pending abort, we stop discarding
+ * undo from the same location.
+ */
+ if (TransactionIdPrecedes(undoxid, xmin) && !pending_abort)
+ {
+ UndoRecPtr next_insert = InvalidUndoRecPtr;
+
+ /*
+ * Get the last insert location for this transaction Id, if it
+ * returns invalid pointer that means there is new transaction
+ * has started for this undolog. So we need to refetch the undo
+ * and continue the process.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, undoxid);
+ if (!UndoRecPtrIsValid(next_insert))
+ continue;
+
+ undo_recptr = next_insert;
+ need_discard = true;
+ latest_discardxid = undoxid;
+ undoxid = InvalidTransactionId;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_EXCLUSIVE);
+
+ /*
+ * If no more pending undo logs then set the oldest transaction to
+ * InvalidTransactionId.
+ */
+ if (log_complete)
+ log->oldest_xid = InvalidTransactionId;
+ else
+ log->oldest_xid = undoxid;
+
+ log->oldest_data = undo_recptr;
+ LWLockRelease(&log->discard_lock);
+
+ if (need_discard)
+ UndoLogDiscard(undo_recptr, latest_discardxid);
+
+ break;
+ }
+
+ /*
+ * This transaction is smaller than the xmin so lets jump to the next
+ * transaction.
+ */
+ undo_recptr = next_urecptr;
+ latest_discardxid = undoxid;
+
+ if(uur != NULL)
+ {
+ UndoRecordRelease(uur);
+ uur = NULL;
+ }
+
+ need_discard = true;
+ } while (true);
+
+ return undoxid;
+}
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+void
+UndoDiscard(TransactionId oldestXmin, bool *hibernate)
+{
+ TransactionId oldestXidHavingUndo = oldestXmin;
+ UndoLogControl *log = NULL;
+
+ /*
+ * TODO: Ideally we'd arrange undo logs so that we can efficiently find
+ * those with oldest_xid < oldestXmin, but for now we'll just scan all of
+ * them.
+ */
+ while ((log = UndoLogNext(log)))
+ {
+ TransactionId oldest_xid = InvalidTransactionId;
+
+ /* We can't process temporary undo logs. */
+ if (log->meta.persistence == UNDO_TEMP)
+ continue;
+
+ /*
+ * If the first xid of the undo log is smaller than the xmin the try
+ * to discard the undo log.
+ */
+ if (TransactionIdPrecedes(log->oldest_xid, oldestXmin))
+ {
+ /*
+ * If the XID in the discard entry is invalid then start scanning
+ * from the first valid undorecord in the log.
+ */
+ if (!TransactionIdIsValid(log->oldest_xid))
+ {
+ bool full = false;
+ UndoRecPtr urp = UndoLogGetFirstValidRecord(log, &full);
+
+ if (!UndoRecPtrIsValid(urp))
+ {
+ /*
+ * There is nothing to be discarded. If there is also no
+ * more free space, then a call to UndoLogDiscard() will
+ * discard it the undo log completely and free up the
+ * UndoLogControl slot.
+ */
+ if (full)
+ UndoLogDiscard(MakeUndoRecPtr(log->meta.logno,
+ log->meta.discard),
+ InvalidTransactionId);
+ continue;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_SHARED);
+ log->oldest_data = urp;
+ LWLockRelease(&log->discard_lock);
+ }
+
+ /* Process the undo log. */
+ oldest_xid = UndoDiscardOneLog(log, oldestXmin, hibernate);
+ }
+
+ if (TransactionIdIsValid(oldest_xid) &&
+ TransactionIdPrecedes(oldest_xid, oldestXidHavingUndo))
+ oldestXidHavingUndo = oldest_xid;
+ }
+
+ /*
+ * Update the oldestXidHavingUndo in the shared memory.
+ *
+ * XXX In future if multiple worker can perform discard then we may need
+ * to use compare and swap for updating the shared memory value.
+ */
+ pg_atomic_write_u32(&ProcGlobal->oldestXidHavingUndo, oldestXidHavingUndo);
+}
+
+/*
+ * Fetch the latest urec pointer for the transaction.
+ */
+UndoRecPtr
+FetchLatestUndoPtrForXid(UndoRecPtr urecptr, UnpackedUndoRecord *uur_start,
+ UndoLogControl *log)
+{
+ UndoRecPtr next_urecptr, from_urecptr;
+ uint16 prevlen;
+ UndoLogOffset next_insert;
+ UnpackedUndoRecord *uur;
+ bool refetch = false;
+
+ uur = uur_start;
+
+ while (true)
+ {
+ /* fetch the undo record again if required. */
+ if (refetch)
+ {
+ uur = UndoFetchRecord(urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ refetch = false;
+ }
+
+ next_urecptr = uur->uur_next;
+ prevlen = UndoLogGetPrevLen(log->logno);
+
+ /*
+ * If this is the last transaction in the log then calculate the latest
+ * urec pointer using next insert location of the undo log. Otherwise,
+ * calculate using next transaction's start pointer.
+ */
+ if (uur->uur_next == SpecialUndoRecPtr)
+ {
+ /*
+ * While fetching the next insert location if the new transaction
+ * has already started in this log then lets re-fetch the undo
+ * record.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+ if (!UndoRecPtrIsValid(next_insert))
+ {
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ refetch = true;
+ continue;
+ }
+
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen,
+ InvalidUndoRecPtr);
+ break;
+ }
+ else if ((UndoRecPtrGetLogNo(next_urecptr) != log->logno) &&
+ UndoLogIsDiscarded(next_urecptr))
+ {
+ /*
+ * If next_urecptr is in different undolog and its already discarded
+ * that means the undo actions for this transaction which are in the
+ * next log has already been executed and we only need to execute
+ * which are remaining in this log.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+
+ Assert(UndoRecPtrIsValid(next_insert));
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen,
+ InvalidUndoRecPtr);
+ break;
+ }
+ else
+ {
+ UnpackedUndoRecord *next_uur;
+
+ next_uur = UndoFetchRecord(next_urecptr,
+ InvalidBlockNumber,
+ InvalidOffsetNumber,
+ InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the next_urecptr is in the same log then calculate the
+ * from pointer using prevlen.
+ */
+ if (UndoRecPtrGetLogNo(next_urecptr) == log->logno)
+ {
+ from_urecptr =
+ UndoGetPrevUndoRecptr(next_urecptr, next_uur->uur_prevlen,
+ InvalidUndoRecPtr);
+ UndoRecordRelease(next_uur);
+ break;
+ }
+ else
+ {
+ /*
+ * The transaction is overflowed to the next log, so restart
+ * the processing from then next log.
+ */
+ log = UndoLogGet(UndoRecPtrGetLogNo(next_urecptr), false);
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ uur = next_uur;
+ continue;
+ }
+
+ UndoRecordRelease(next_uur);
+ }
+ }
+
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+
+ return from_urecptr;
+}
+
+/*
+ * Discard the undo logs for temp tables.
+ */
+void
+TempUndoDiscard(UndoLogNumber logno)
+{
+ UndoLogControl *log = UndoLogGet(logno, false);
+
+ /*
+ * Discard the undo log for temp table only. Ensure that there is
+ * something to be discarded there.
+ */
+ Assert (log->meta.persistence == UNDO_TEMP);
+
+ /* Process the undo log. */
+ UndoLogDiscard(MakeUndoRecPtr(log->logno, log->meta.insert),
+ InvalidTransactionId);
+}
diff --git a/src/backend/access/undo/undoinsert.c b/src/backend/access/undo/undoinsert.c
index 9ffa46c..4e0fe13 100644
--- a/src/backend/access/undo/undoinsert.c
+++ b/src/backend/access/undo/undoinsert.c
@@ -171,7 +171,6 @@ static UnpackedUndoRecord *UndoGetOneRecord(UnpackedUndoRecord *urec,
UndoPersistence persistence);
static void UndoRecordPrepareTransInfo(UndoRecPtr urecptr,
UndoRecPtr xact_urp);
-static void UndoRecordUpdateTransInfo(int idx);
static int UndoGetBufferSlot(RelFileNode rnode, BlockNumber blk,
ReadBufferMode rbm,
UndoPersistence persistence);
@@ -302,12 +301,61 @@ UndoRecordPrepareTransInfo(UndoRecPtr urecptr, UndoRecPtr xact_urp)
}
/*
+ * Update the progress of the undo record in the transaction header.
+ */
+void
+PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress)
+{
+ Buffer buffer = InvalidBuffer;
+ BlockNumber cur_blk;
+ RelFileNode rnode;
+ UndoLogNumber logno = UndoRecPtrGetLogNo(urecptr);
+ UndoLogControl *log;
+ Page page;
+ int already_decoded = 0;
+ int starting_byte;
+ int bufidx;
+ int index = 0;
+
+ log = UndoLogGet(logno, false);
+
+ if (log->meta.persistence == UNDO_TEMP)
+ return;
+
+ UndoRecPtrAssignRelFileNode(rnode, urecptr);
+ cur_blk = UndoRecPtrGetBlockNum(urecptr);
+ starting_byte = UndoRecPtrGetPageOffset(urecptr);
+
+ while (true)
+ {
+ bufidx = UndoGetBufferSlot(rnode, cur_blk,
+ RBM_NORMAL,
+ log->meta.persistence);
+
+ xact_urec_info[xact_urec_info_idx].idx_undo_buffers[index++] = bufidx;
+ buffer = undo_buffer[bufidx].buf;
+ page = BufferGetPage(buffer);
+
+ if (UnpackUndoRecord(&xact_urec_info[xact_urec_info_idx].uur, page, starting_byte,
+ &already_decoded, true))
+ break;
+
+ starting_byte = UndoLogBlockHeaderSize;
+ cur_blk++;
+ }
+
+ xact_urec_info[xact_urec_info_idx].urecptr = urecptr;
+ xact_urec_info[xact_urec_info_idx].uur.uur_progress = progress;
+ xact_urec_info_idx++;
+}
+
+/*
* Overwrite the first undo record of the previous transaction to update its
* next pointer. This will just insert the already prepared record by
* UndoRecordPrepareTransInfo. This must be called under the critical section.
* This will just overwrite the undo header not the data.
*/
-static void
+void
UndoRecordUpdateTransInfo(int idx)
{
UndoLogNumber logno = UndoRecPtrGetLogNo(xact_urec_info[idx].urecptr);
diff --git a/src/backend/access/undo/undorecord.c b/src/backend/access/undo/undorecord.c
index 23ed374..9af8e45 100644
--- a/src/backend/access/undo/undorecord.c
+++ b/src/backend/access/undo/undorecord.c
@@ -98,6 +98,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
*/
if (*already_written == 0)
{
+ work_hdr.urec_rmid = uur->uur_rmid;
work_hdr.urec_type = uur->uur_type;
work_hdr.urec_info = uur->uur_info;
work_hdr.urec_prevlen = uur->uur_prevlen;
@@ -122,6 +123,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
* We should have been passed the same record descriptor as before, or
* caller has messed up.
*/
+ Assert(work_hdr.urec_rmid == uur->uur_rmid);
Assert(work_hdr.urec_type == uur->uur_type);
Assert(work_hdr.urec_info == uur->uur_info);
Assert(work_hdr.urec_prevlen == uur->uur_prevlen);
@@ -285,6 +287,7 @@ UnpackUndoRecord(UnpackedUndoRecord *uur, Page page, int starting_byte,
&my_bytes_decoded, already_decoded, false))
return false;
+ uur->uur_rmid = work_hdr.urec_rmid;
uur->uur_type = work_hdr.urec_type;
uur->uur_info = work_hdr.urec_info;
uur->uur_prevlen = work_hdr.urec_prevlen;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index ff1e178..e7a0bbe 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1381,6 +1381,7 @@ vac_truncate_clog(TransactionId frozenXID,
MultiXactId lastSaneMinMulti)
{
TransactionId nextXID = ReadNewTransactionId();
+ TransactionId oldestXidHavingUndo;
Relation relation;
HeapScanDesc scan;
HeapTuple tuple;
@@ -1475,6 +1476,15 @@ vac_truncate_clog(TransactionId frozenXID,
return;
/*
+ * We can't truncate the clog for transactions that still have undo. The
+ * oldestXidHavingUndo will be only valid for zheap storage engine, so it
+ * won't impact any other storage engine.
+ */
+ oldestXidHavingUndo = pg_atomic_read_u32(&ProcGlobal->oldestXidHavingUndo);
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ frozenXID = Min(frozenXID, oldestXidHavingUndo);
+
+ /*
* Advance the oldest value for commit timestamps before truncating, so
* that if a user requests a timestamp for a transaction we're truncating
* away right after this point, they get NULL instead of an ugly "file not
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 71c2321..9ce6ff0 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -12,7 +12,7 @@ subdir = src/backend/postmaster
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
- pgarch.o pgstat.o postmaster.o startup.o syslogger.o walwriter.o
+OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o discardworker.o fork_process.o \
+ pgarch.o pgstat.o postmaster.o startup.o syslogger.o undoworker.o walwriter.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index f5db5a8..772b865 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -20,7 +20,9 @@
#include "pgstat.h"
#include "port/atomics.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "storage/dsm.h"
@@ -129,6 +131,15 @@ static const struct
},
{
"ApplyWorkerMain", ApplyWorkerMain
+ },
+ {
+ "UndoLauncherMain", UndoLauncherMain
+ },
+ {
+ "UndoWorkerMain", UndoWorkerMain
+ },
+ {
+ "DiscardWorkerMain", DiscardWorkerMain
}
};
diff --git a/src/backend/postmaster/discardworker.c b/src/backend/postmaster/discardworker.c
new file mode 100644
index 0000000..7cbe6c4
--- /dev/null
+++ b/src/backend/postmaster/discardworker.c
@@ -0,0 +1,170 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.c
+ * The undo discard worker for asynchronous undo management.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/discardworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include <unistd.h>
+
+/* These are always necessary for a bgworker. */
+#include "access/transam.h"
+#include "miscadmin.h"
+#include "postmaster/bgworker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/lwlock.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+#include "access/undodiscard.h"
+#include "pgstat.h"
+#include "postmaster/discardworker.h"
+#include "storage/procarray.h"
+#include "tcop/tcopprot.h"
+#include "utils/guc.h"
+#include "utils/resowner.h"
+
+static void undoworker_sigterm_handler(SIGNAL_ARGS);
+
+/* max sleep time between cycles (100 milliseconds) */
+#define MIN_NAPTIME_PER_CYCLE 100L
+#define DELAYED_NAPTIME 10 * MIN_NAPTIME_PER_CYCLE
+#define MAX_NAPTIME_PER_CYCLE 100 * MIN_NAPTIME_PER_CYCLE
+
+static bool got_SIGTERM = false;
+static bool hibernate = false;
+static long wait_time = MIN_NAPTIME_PER_CYCLE;
+
+/* SIGTERM: set flag to exit at next convenient time */
+static void
+undoworker_sigterm_handler(SIGNAL_ARGS)
+{
+ got_SIGTERM = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+}
+
+/*
+ * DiscardWorkerRegister -- Register a undo discard worker.
+ */
+void
+DiscardWorkerRegister(void)
+{
+ BackgroundWorker bgw;
+
+ /* TODO: This should be configurable. */
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "discard worker");
+ sprintf(bgw.bgw_library_name, "postgres");
+ sprintf(bgw.bgw_function_name, "DiscardWorkerMain");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * DiscardWorkerMain -- Main loop for the undo discard worker.
+ */
+void
+DiscardWorkerMain(Datum main_arg)
+{
+ ereport(LOG,
+ (errmsg("discard worker started")));
+
+ /* Establish signal handlers. */
+ pqsignal(SIGTERM, undoworker_sigterm_handler);
+ BackgroundWorkerUnblockSignals();
+
+ /* Make it easy to identify our processes. */
+ SetConfigOption("application_name", MyBgworkerEntry->bgw_name,
+ PGC_USERSET, PGC_S_SESSION);
+
+ /*
+ * Create resource owner for discard worker as it need to read the undo
+ * records outside the transaction blocks which intern access buffer read
+ * routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ /* Enter main loop */
+ while (!got_SIGTERM)
+ {
+ int rc;
+ TransactionId OldestXmin, oldestXidHavingUndo;
+
+ OldestXmin = GetOldestXmin(NULL, PROCARRAY_FLAGS_DEFAULT);
+
+ oldestXidHavingUndo =
+ pg_atomic_read_u32(&ProcGlobal->oldestXidHavingUndo);
+
+ /*
+ * Call the discard routine if there oldestXidHavingUndo is lagging
+ * behind OldestXmin.
+ */
+ if (OldestXmin != InvalidTransactionId &&
+ TransactionIdPrecedes(oldestXidHavingUndo, OldestXmin))
+ {
+ UndoDiscard(OldestXmin, &hibernate);
+
+ /*
+ * If we got some undo logs to discard or discarded something,
+ * then reset the wait_time as we have got work to do.
+ * Note that if there are some undologs that cannot be discarded,
+ * then above condition will remain unsatisified till oldestXmin
+ * remains unchanged and the wait_time will not reset in that case.
+ */
+ if (!hibernate)
+ wait_time = MIN_NAPTIME_PER_CYCLE;
+ }
+
+ /* Wait for more work. */
+ rc = WaitLatch(&MyProc->procLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wait_time,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN);
+
+ ResetLatch(&MyProc->procLatch);
+
+ /*
+ * Increase the wait_time based on the length of inactivity. If wait_time
+ * is within one second, then increment it by 100 ms at a time. Henceforth,
+ * increment it one second at a time, till it reaches ten seconds. Never
+ * increase the wait_time more than ten seconds, it will be too much of
+ * waiting otherwise.
+ */
+ if (rc & WL_TIMEOUT && hibernate)
+ {
+ wait_time += (wait_time < DELAYED_NAPTIME ?
+ MIN_NAPTIME_PER_CYCLE : DELAYED_NAPTIME);
+ if (wait_time > MAX_NAPTIME_PER_CYCLE)
+ wait_time = MAX_NAPTIME_PER_CYCLE;
+ }
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+
+ ReleaseAuxProcessResources(true);
+
+ /* we're done */
+ ereport(LOG,
+ (errmsg("discard worker shutting down")));
+
+ proc_exit(0);
+}
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 03591bf..6cae438 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -3506,6 +3506,12 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_SYSLOGGER_MAIN:
event_name = "SysLoggerMain";
break;
+ case WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN:
+ event_name = "UndoDiscardWorkerMain";
+ break;
+ case WAIT_EVENT_UNDO_LAUNCHER_MAIN:
+ event_name = "UndoLauncherMain";
+ break;
case WAIT_EVENT_WAL_RECEIVER_MAIN:
event_name = "WalReceiverMain";
break;
@@ -3918,7 +3924,6 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_UNDO_FILE_SYNC:
event_name = "UndoFileSync";
break;
-
case WAIT_EVENT_WALSENDER_TIMELINE_HISTORY_READ:
event_name = "WALSenderTimelineHistoryRead";
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a707d4d..c4acd72 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -111,10 +111,12 @@
#include "port/pg_bswap.h"
#include "postmaster/autovacuum.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/walsender.h"
#include "storage/fd.h"
@@ -981,6 +983,11 @@ PostmasterMain(int argc, char *argv[])
*/
ApplyLauncherRegister();
+ UndoLauncherRegister();
+
+ /* Register the Undo Discard worker. */
+ DiscardWorkerRegister();
+
/*
* process any libraries that should be preloaded at postmaster start
*/
diff --git a/src/backend/postmaster/undoworker.c b/src/backend/postmaster/undoworker.c
new file mode 100644
index 0000000..bf72203
--- /dev/null
+++ b/src/backend/postmaster/undoworker.c
@@ -0,0 +1,664 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.c
+ * undo launcher and undo worker process.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/undoworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "funcapi.h"
+#include "miscadmin.h"
+#include "pgstat.h"
+
+#include "access/heapam.h"
+#include "access/htup.h"
+#include "access/htup_details.h"
+#include "access/sysattr.h"
+#include "access/xact.h"
+
+#include "catalog/indexing.h"
+#include "catalog/pg_database.h"
+
+#include "libpq/pqsignal.h"
+
+#include "postmaster/bgworker.h"
+#include "postmaster/fork_process.h"
+#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
+
+#include "replication/slot.h"
+#include "replication/worker_internal.h"
+
+#include "storage/ipc.h"
+#include "storage/lmgr.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+
+#include "tcop/tcopprot.h"
+
+#include "utils/fmgroids.h"
+#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/resowner.h"
+
+/* max sleep time between cycles (100 milliseconds) */
+#define DEFAULT_NAPTIME_PER_CYCLE 100L
+#define DEFAULT_RETRY_NAPTIME 50L
+
+int max_undo_workers = 5;
+
+typedef struct UndoApplyWorker
+{
+ /* Indicates if this slot is used or free. */
+ bool in_use;
+
+ /* Increased everytime the slot is taken by new worker. */
+ uint16 generation;
+
+ /* Pointer to proc array. NULL if not running. */
+ PGPROC *proc;
+
+ /* Database id to connect to. */
+ Oid dbid;
+} UndoApplyWorker;
+
+UndoApplyWorker *MyUndoWorker = NULL;
+
+typedef struct UndoApplyCtxStruct
+{
+ /* Supervisor process. */
+ pid_t launcher_pid;
+
+ /* Background workers. */
+ UndoApplyWorker workers[FLEXIBLE_ARRAY_MEMBER];
+} UndoApplyCtxStruct;
+
+UndoApplyCtxStruct *UndoApplyCtx;
+
+static void undo_worker_onexit(int code, Datum arg);
+static void undo_worker_cleanup(UndoApplyWorker *worker);
+
+static volatile sig_atomic_t got_SIGHUP = false;
+
+/*
+ * Wait for a background worker to start up and attach to the shmem context.
+ *
+ * This is only needed for cleaning up the shared memory in case the worker
+ * fails to attach.
+ */
+static void
+WaitForUndoWorkerAttach(UndoApplyWorker *worker,
+ uint16 generation,
+ BackgroundWorkerHandle *handle)
+{
+ BgwHandleStatus status;
+ int rc;
+
+ for (;;)
+ {
+ pid_t pid;
+
+ CHECK_FOR_INTERRUPTS();
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+
+ /* Worker either died or has started; no need to do anything. */
+ if (!worker->in_use || worker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ LWLockRelease(UndoWorkerLock);
+
+ /* Check if worker has died before attaching, and clean up after it. */
+ status = GetBackgroundWorkerPid(handle, &pid);
+
+ if (status == BGWH_STOPPED)
+ {
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ /* Ensure that this was indeed the worker we waited for. */
+ if (generation == worker->generation)
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ /*
+ * We need timeout because we generally don't get notified via latch
+ * about the worker attach. But we don't expect to have to wait long.
+ */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 10L, WAIT_EVENT_BGWORKER_STARTUP);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+ }
+
+ return;
+}
+
+/*
+ * Get dbid from the worker slot.
+ */
+static Oid
+slot_get_dbid(int slot)
+{
+ Oid dbid;
+
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty,",
+ slot)));
+ }
+
+ dbid = MyUndoWorker->dbid;
+
+ LWLockRelease(UndoWorkerLock);
+
+ return dbid;
+}
+
+/*
+ * Attach to a slot.
+ */
+static void
+undo_worker_attach(int slot)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty, cannot attach",
+ slot)));
+ }
+
+ if (MyUndoWorker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is already used by "
+ "another worker, cannot attach", slot)));
+ }
+
+ MyUndoWorker->proc = MyProc;
+ before_shmem_exit(undo_worker_onexit, (Datum) 0);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Walks the workers array and searches for one that matches given
+ * dbid.
+ */
+static UndoApplyWorker *
+undo_worker_find(Oid dbid)
+{
+ int i;
+ UndoApplyWorker *res = NULL;
+
+ Assert(LWLockHeldByMe(UndoWorkerLock));
+
+ /* Search for attached worker for a given db id. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (w->in_use && w->dbid == dbid)
+ {
+ res = w;
+ break;
+ }
+ }
+
+ return res;
+}
+
+/*
+ * Check whether the dbid exist or not.
+ *
+ * Refer comments from GetDatabaseTupleByOid.
+ * FIXME: Should we expose GetDatabaseTupleByOid and directly use it.
+ */
+static bool
+dbid_exist(Oid dboid)
+{
+ HeapTuple tuple;
+ Relation relation;
+ SysScanDesc scan;
+ ScanKeyData key[1];
+ bool result = false;
+ /*
+ * form a scan key
+ */
+ ScanKeyInit(&key[0],
+ Anum_pg_database_oid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dboid));
+
+ relation = heap_open(DatabaseRelationId, AccessShareLock);
+ scan = systable_beginscan(relation, DatabaseOidIndexId,
+ criticalSharedRelcachesBuilt,
+ NULL,
+ 1, key);
+
+ tuple = systable_getnext(scan);
+
+ if (HeapTupleIsValid(tuple))
+ result = true;
+
+ /* all done */
+ systable_endscan(scan);
+ heap_close(relation, AccessShareLock);
+
+ return result;
+}
+
+/*
+ * Start new undo apply background worker, if possible otherwise return false.
+ */
+static bool
+undo_worker_launch(Oid dbid)
+{
+ BackgroundWorker bgw;
+ BackgroundWorkerHandle *bgw_handle;
+ uint16 generation;
+ int i;
+ int slot = 0;
+ UndoApplyWorker *worker = NULL;
+
+ /*
+ * We need to do the modification of the shared memory under lock so that
+ * we have consistent view.
+ */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ /* Find unused worker slot. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (!w->in_use)
+ {
+ worker = w;
+ slot = i;
+ break;
+ }
+ }
+
+ /* There are no more free worker slots */
+ if (worker == NULL)
+ return false;
+
+ /* Prepare the worker slot. */
+ worker->in_use = true;
+ worker->proc = NULL;
+ worker->dbid = dbid;
+ worker->generation++;
+
+ generation = worker->generation;
+ LWLockRelease(UndoWorkerLock);
+
+ /* Register the new dynamic worker. */
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoWorkerMain");
+ snprintf(bgw.bgw_type, BGW_MAXLEN, "undo apply worker");
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "undo apply worker");
+
+ bgw.bgw_restart_time = BGW_NEVER_RESTART;
+ bgw.bgw_notify_pid = MyProcPid;
+ bgw.bgw_main_arg = Int32GetDatum(slot);
+
+ StartTransactionCommand();
+ /* Check the database exists or not. */
+ if (!dbid_exist(dbid))
+ {
+ CommitTransactionCommand();
+
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return true;
+ }
+
+ /*
+ * Acquire database object lock before launching the worker so that it
+ * doesn't get dropped while worker is connecting to the database.
+ */
+ LockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+
+ /* Recheck whether database still exists or not. */
+ if (!dbid_exist(dbid))
+ {
+ CommitTransactionCommand();
+
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return true;
+ }
+
+ if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle))
+ {
+ /* Failed to start worker, so clean up the worker slot. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+
+ UnlockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+ CommitTransactionCommand();
+
+ return false;
+ }
+
+ /* Now wait until it attaches. */
+ WaitForUndoWorkerAttach(worker, generation, bgw_handle);
+
+ /*
+ * By this point the undo-worker has already connected to the database so we
+ * can release the database lock.
+ */
+ UnlockSharedObject(DatabaseRelationId, dbid, 0, RowExclusiveLock);
+ CommitTransactionCommand();
+
+ return true;
+}
+
+/*
+ * Detach the worker (cleans up the worker info).
+ */
+static void
+undo_worker_detach(void)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ undo_worker_cleanup(MyUndoWorker);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Clean up worker info.
+ */
+static void
+undo_worker_cleanup(UndoApplyWorker *worker)
+{
+ Assert(LWLockHeldByMeInMode(UndoWorkerLock, LW_EXCLUSIVE));
+
+ worker->in_use = false;
+ worker->proc = NULL;
+ worker->dbid = InvalidOid;
+}
+
+/*
+ * Cleanup function for undo worker launcher.
+ *
+ * Called on undo worker launcher exit.
+ */
+static void
+undo_launcher_onexit(int code, Datum arg)
+{
+ UndoApplyCtx->launcher_pid = 0;
+}
+
+/* SIGHUP: set flag to reload configuration at next convenient time */
+static void
+undo_launcher_sighup(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGHUP = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+undo_worker_onexit(int code, Datum arg)
+{
+ undo_worker_detach();
+}
+
+/*
+ * UndoLauncherShmemSize
+ * Compute space needed for undo launcher shared memory
+ */
+Size
+UndoLauncherShmemSize(void)
+{
+ Size size;
+
+ /*
+ * Need the fixed struct and the array of LogicalRepWorker.
+ */
+ size = sizeof(UndoApplyCtxStruct);
+ size = MAXALIGN(size);
+ size = add_size(size, mul_size(max_undo_workers,
+ sizeof(UndoApplyWorker)));
+ return size;
+}
+
+/*
+ * UndoLauncherRegister
+ * Register a background worker running the undo worker launcher.
+ */
+void
+UndoLauncherRegister(void)
+{
+ BackgroundWorker bgw;
+
+ if (max_undo_workers == 0)
+ return;
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoLauncherMain");
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "undo worker launcher");
+ snprintf(bgw.bgw_type, BGW_MAXLEN,
+ "undo worker launcher");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * UndoLauncherShmemInit
+ * Allocate and initialize undo worker launcher shared memory
+ */
+void
+UndoLauncherShmemInit(void)
+{
+ bool found;
+
+ UndoApplyCtx = (UndoApplyCtxStruct *)
+ ShmemInitStruct("Undo Worker Launcher Data",
+ UndoLauncherShmemSize(),
+ &found);
+
+ if (!found)
+ memset(UndoApplyCtx, 0, UndoLauncherShmemSize());
+}
+
+/*
+ * Main loop for the undo worker launcher process.
+ */
+void
+UndoLauncherMain(Datum main_arg)
+{
+ MemoryContext tmpctx;
+ MemoryContext oldctx;
+
+ ereport(DEBUG1,
+ (errmsg("undo launcher started")));
+
+ before_shmem_exit(undo_launcher_onexit, (Datum) 0);
+
+ Assert(UndoApplyCtx->launcher_pid == 0);
+ UndoApplyCtx->launcher_pid = MyProcPid;
+
+ /* Establish signal handlers. */
+ pqsignal(SIGHUP, undo_launcher_sighup);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * Establish connection to nailed catalogs (we only ever access
+ * pg_subscription).
+ */
+ BackgroundWorkerInitializeConnection(NULL, NULL, 0);
+
+ /* Use temporary context for the database list and worker info. */
+ tmpctx = AllocSetContextCreate(TopMemoryContext,
+ "Undo worker Launcher context",
+ ALLOCSET_DEFAULT_SIZES);
+ /* Enter main loop */
+ for (;;)
+ {
+ int rc;
+ List *dblist;
+ ListCell *l;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* switch to the temp context. */
+ oldctx = MemoryContextSwitchTo(tmpctx);
+ dblist = RollbackHTGetDBList();
+
+ foreach(l, dblist)
+ {
+ UndoApplyWorker *w;
+ Oid dbid = lfirst_oid(l);
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+ w = undo_worker_find(dbid);
+ LWLockRelease(UndoWorkerLock);
+
+ if (w == NULL)
+ {
+retry:
+ if (!undo_worker_launch(dbid))
+ {
+ /* Could not launch the worker, retry after sometime, */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_RETRY_NAPTIME,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+ goto retry;
+ }
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+
+ /* Clean the temporary memory. */
+ MemoryContextReset(tmpctx);
+
+ /* Wait for more work. */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_NAPTIME_PER_CYCLE,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ if (got_SIGHUP)
+ {
+ got_SIGHUP = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+}
+
+/*
+ * UndoWorkerMain -- Main loop for the undo apply worker.
+ */
+void
+UndoWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ Oid dbid;
+
+ dbid = slot_get_dbid(worker_slot);
+
+ /* Setup signal handling */
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /* Connect to the database. */
+ BackgroundWorkerInitializeConnectionByOid(dbid, 0, 0);
+
+ /* Attach to slot */
+ undo_worker_attach(worker_slot);
+
+ /*
+ * Create resource owner for undo worker. Undo worker need this as it
+ * need to read the undo records outside the transaction blocks which
+ * intern access buffer read routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ RollbackFromHT(dbid);
+
+ ReleaseAuxProcessResources(true);
+
+ proc_exit(0);
+}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index a8aa11a..52e094c 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -159,6 +159,10 @@ LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *recor
ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(record),
buf.origptr);
break;
+ case RM_UNDOACTION_ID:
+ /* Logical decoding is not yet implemented for undoactions. */
+ Assert(0);
+ break;
case RM_NEXT_ID:
elog(ERROR, "unexpected RM_NEXT_ID rmgr_id: %u", (RmgrIds) XLogRecGetRmid(buf.record));
}
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index faedafb..ab043c1 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -29,6 +29,8 @@
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
#include "replication/walreceiver.h"
@@ -150,6 +152,8 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, RollbackHTSize());
+ size = add_size(size, UndoLauncherShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -224,6 +228,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
SUBTRANSShmemInit();
MultiXactShmemInit();
InitBufferPool();
+ InitRollbackHashTable();
/*
* Set up lock manager
@@ -262,6 +267,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
WalSndShmemInit();
WalRcvShmemInit();
ApplyLauncherShmemInit();
+ UndoLauncherShmemInit();
/*
* Set up other modules that need some shared memory space
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index bd07ed6..a47ecd4 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -50,3 +50,5 @@ OldSnapshotTimeMapLock 42
LogicalRepWorkerLock 43
CLogTruncationLock 44
UndoLogLock 45
+RollbackHTLock 46
+UndoWorkerLock 47
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 89c80fb..bd2bc98 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -286,6 +286,8 @@ InitProcGlobal(void)
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemAlloc(sizeof(slock_t));
SpinLockInit(ProcStructLock);
+
+ pg_atomic_init_u32(&ProcGlobal->oldestXidHavingUndo, 0);
}
/*
diff --git a/src/backend/utils/adt/lockfuncs.c b/src/backend/utils/adt/lockfuncs.c
index ffd1970..bdb6606 100644
--- a/src/backend/utils/adt/lockfuncs.c
+++ b/src/backend/utils/adt/lockfuncs.c
@@ -32,6 +32,7 @@ const char *const LockTagTypeNames[] = {
"virtualxid",
"speculative token",
"object",
+ "undoaction",
"userlock",
"advisory"
};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index fd51934..08489a1 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -121,6 +121,7 @@ bool allowSystemTableMods = false;
int work_mem = 1024;
int maintenance_work_mem = 16384;
int max_parallel_maintenance_workers = 2;
+int rollback_overflow_size = 64;
/*
* Primary determinants of sizes of shared-memory structures.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index c5698b5..7a8878d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2862,6 +2862,17 @@ static struct config_int ConfigureNamesInt[] =
},
{
+ {"rollback_overflow_size", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Rollbacks greater than this size are done lazily"),
+ NULL,
+ GUC_UNIT_MB
+ },
+ &rollback_overflow_size,
+ 64, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
{"wal_segment_size", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the size of write ahead log segments."),
NULL,
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index e6742dc..96908ea 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -78,8 +78,8 @@ pg_control_system(PG_FUNCTION_ARGS)
Datum
pg_control_checkpoint(PG_FUNCTION_ARGS)
{
- Datum values[19];
- bool nulls[19];
+ Datum values[20];
+ bool nulls[20];
TupleDesc tupdesc;
HeapTuple htup;
ControlFileData *ControlFile;
@@ -128,6 +128,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
XIDOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 18, "checkpoint_time",
TIMESTAMPTZOID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 19, "oldest_xid_with_epoch_having_undo",
+ TIMESTAMPTZOID, -1, 0);
tupdesc = BlessTupleDesc(tupdesc);
/* Read the control file. */
@@ -202,6 +204,9 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
time_t_to_timestamptz(ControlFile->checkPointCopy.time));
nulls[17] = false;
+ values[18] = Int32GetDatum(ControlFile->checkPointCopy.oldestXidHavingUndo);
+ nulls[18] = false;
+
htup = heap_form_tuple(tupdesc, values, nulls);
PG_RETURN_DATUM(HeapTupleGetDatum(htup));
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1fa02d2..82b9cf6 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -737,4 +737,11 @@
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
+# If often there are large transactions requiring rollbacks, then we can push
+# them to undo-workers for better performance. The size specifeid by the
+# parameter below, determines the minimum size of the rollback requests to be
+# sent to the undo-worker.
+#
+#rollback_overflow_size = 64
+
# Add settings for extensions here
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 895a51f..dd4f12a 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -278,6 +278,8 @@ main(int argc, char *argv[])
ControlFile->checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile->checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidHavingUndo:%u\n"),
+ ControlFile->checkPointCopy.oldestXidHavingUndo);
printf(_("Time of latest checkpoint: %s\n"),
ckpttime_str);
printf(_("Fake LSN counter for unlogged rels: %X/%X\n"),
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index e3213d4..e30375a 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -448,6 +448,7 @@ main(int argc, char *argv[])
if (ControlFile.checkPointCopy.oldestXid < FirstNormalTransactionId)
ControlFile.checkPointCopy.oldestXid += FirstNormalTransactionId;
ControlFile.checkPointCopy.oldestXidDB = InvalidOid;
+ ControlFile.checkPointCopy.oldestXidHavingUndo = 0;
}
if (set_oldest_commit_ts_xid != 0)
@@ -716,6 +717,8 @@ GuessControlValues(void)
ControlFile.checkPointCopy.oldestMultiDB = InvalidOid;
ControlFile.checkPointCopy.time = (pg_time_t) time(NULL);
ControlFile.checkPointCopy.oldestActiveXid = InvalidTransactionId;
+ ControlFile.checkPointCopy.nextXid = 0;
+ ControlFile.checkPointCopy.oldestXidHavingUndo = 0;
ControlFile.state = DB_SHUTDOWNED;
ControlFile.time = (pg_time_t) time(NULL);
@@ -808,6 +811,8 @@ PrintControlValues(bool guessed)
ControlFile.checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile.checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidHavingUndo:%u\n"),
+ ControlFile.checkPointCopy.oldestXidHavingUndo);
printf(_("Maximum data alignment: %u\n"),
ControlFile.maxAlign);
/* we don't print floatFormat since can't say much useful about it */
@@ -884,6 +889,8 @@ PrintNewControlValues(void)
ControlFile.checkPointCopy.oldestXid);
printf(_("OldestXID's DB: %u\n"),
ControlFile.checkPointCopy.oldestXidDB);
+ printf(_("OldestXidHavingUndo: %u\n"),
+ ControlFile.checkPointCopy.oldestXidHavingUndo);
}
if (set_xid_epoch != -1)
diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c
index e19c265..ef0b1a0 100644
--- a/src/bin/pg_rewind/parsexlog.c
+++ b/src/bin/pg_rewind/parsexlog.c
@@ -29,7 +29,7 @@
* RmgrNames is an array of resource manager names, to make error messages
* a bit nicer.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
name,
static const char *RmgrNames[RM_MAX_ID + 1] = {
diff --git a/src/bin/pg_waldump/rmgrdesc.c b/src/bin/pg_waldump/rmgrdesc.c
index 938150d..3ef0a60 100644
--- a/src/bin/pg_waldump/rmgrdesc.c
+++ b/src/bin/pg_waldump/rmgrdesc.c
@@ -20,6 +20,7 @@
#include "access/nbtxlog.h"
#include "access/rmgr.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -33,7 +34,7 @@
#include "storage/standbydefs.h"
#include "utils/relmapper.h"
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
{ name, desc, identify},
const RmgrDescData RmgrDescTable[RM_MAX_ID + 1] = {
diff --git a/src/include/access/rmgr.h b/src/include/access/rmgr.h
index c9b5c56..e1fb42a 100644
--- a/src/include/access/rmgr.h
+++ b/src/include/access/rmgr.h
@@ -19,7 +19,7 @@ typedef uint8 RmgrId;
* Note: RM_MAX_ID must fit in RmgrId; widening that type will affect the XLOG
* file format.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
symname,
typedef enum RmgrIds
diff --git a/src/include/access/rmgrlist.h b/src/include/access/rmgrlist.h
index 6945e3e..ef0f6ac 100644
--- a/src/include/access/rmgrlist.h
+++ b/src/include/access/rmgrlist.h
@@ -25,26 +25,27 @@
*/
/* symbol name, textual name, redo, desc, identify, startup, cleanup */
-PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL)
-PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL)
-PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL)
-PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL)
-PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL)
-PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask)
-PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask)
-PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask)
-PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask)
-PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask)
-PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask)
-PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask)
-PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL)
-PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL)
-PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask)
-PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL)
-PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL)
+PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask, NULL, NULL)
+PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask, NULL, NULL)
+PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask, NULL, NULL)
+PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask, NULL, NULL)
+PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask, NULL, NULL)
+PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask, NULL, NULL)
+PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask, NULL, NULL)
+PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask, NULL, NULL)
+PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOACTION_ID, "UndoAction", undoaction_redo, undoaction_desc, undoaction_identify, NULL, NULL, NULL, NULL, NULL)
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 4a7eab0..92d5d51 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -68,6 +68,10 @@
(AssertMacro(TransactionIdIsNormal(id1) && TransactionIdIsNormal(id2)), \
(int32) ((id1) - (id2)) > 0)
+/* Extract xid from a value comprised of epoch and xid */
+#define GetXidFromEpochXid(epochxid) \
+ ((uint32) (epochxid) & 0XFFFFFFFF)
+
/* ----------
* Object ID (OID) zero is InvalidOid.
*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 6228b09..0b1bcf4 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -41,7 +41,7 @@ extern GlobalTransaction MarkAsPreparing(TransactionId xid, const char *gid,
TimestampTz prepared_at,
Oid owner, Oid databaseid);
-extern void StartPrepare(GlobalTransaction gxact);
+extern void StartPrepare(GlobalTransaction gxact, UndoRecPtr *, UndoRecPtr *);
extern void EndPrepare(GlobalTransaction gxact);
extern bool StandbyTransactionIdIsPrepared(TransactionId xid);
diff --git a/src/include/access/undoaction.h b/src/include/access/undoaction.h
new file mode 100644
index 0000000..5455259
--- /dev/null
+++ b/src/include/access/undoaction.h
@@ -0,0 +1,28 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.h
+ * undo action prototypes
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_H
+#define UNDOACTION_H
+
+#include "postgres.h"
+
+#include "access/undolog.h"
+#include "access/undorecord.h"
+
+/* undo record information */
+typedef struct UndoRecInfo
+{
+ UndoRecPtr urp; /* undo recptr (undo record location). */
+ UnpackedUndoRecord *uur; /* actual undo record. */
+} UndoRecInfo;
+
+#endif
diff --git a/src/include/access/undoaction_xlog.h b/src/include/access/undoaction_xlog.h
new file mode 100644
index 0000000..bfc6418
--- /dev/null
+++ b/src/include/access/undoaction_xlog.h
@@ -0,0 +1,74 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction_xlog.h
+ * undo action XLOG definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction_xlog.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_XLOG_H
+#define UNDOACTION_XLOG_H
+
+#include "access/undolog.h"
+#include "access/xlogreader.h"
+#include "lib/stringinfo.h"
+#include "storage/off.h"
+
+/*
+ * WAL record definitions for undoactions.c's WAL operations
+ */
+#define XLOG_UNDO_PAGE 0x00
+#define XLOG_UNDO_RESET_SLOT 0x10
+#define XLOG_UNDO_APPLY_PROGRESS 0x20
+
+/*
+ * xl_undoaction_page flag values, 8 bits are available.
+ */
+#define XLU_PAGE_CONTAINS_TPD_SLOT (1<<0)
+#define XLU_PAGE_CLEAR_VISIBILITY_MAP (1<<1)
+#define XLU_CONTAINS_TPD_OFFSET_MAP (1<<2)
+#define XLU_INIT_PAGE (1<<3)
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_page
+{
+ UndoRecPtr urec_ptr;
+ TransactionId xid;
+ int trans_slot_id; /* transaction slot id */
+} xl_undoaction_page;
+
+#define SizeOfUndoActionPage (offsetof(xl_undoaction_page, trans_slot_id) + sizeof(int))
+
+/* This is what we need to know about undo apply progress */
+typedef struct xl_undoapply_progress
+{
+ UndoRecPtr urec_ptr;
+ uint32 progress;
+} xl_undoapply_progress;
+
+#define SizeOfUndoActionProgress (offsetof(xl_undoapply_progress, progress) + sizeof(uint32))
+
+/*
+ * xl_undoaction_reset_slot flag values, 8 bits are available.
+ */
+#define XLU_RESET_CONTAINS_TPD_SLOT (1<<0)
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_reset_slot
+{
+ UndoRecPtr urec_ptr;
+ int trans_slot_id; /* transaction slot id */
+ uint8 flags;
+} xl_undoaction_reset_slot;
+
+#define SizeOfUndoActionResetSlot (offsetof(xl_undoaction_reset_slot, flags) + sizeof(uint8))
+
+extern void undoaction_redo(XLogReaderState *record);
+extern void undoaction_desc(StringInfo buf, XLogReaderState *record);
+extern const char *undoaction_identify(uint8 info);
+
+#endif /* UNDOACTION_XLOG_H */
diff --git a/src/include/access/undodiscard.h b/src/include/access/undodiscard.h
new file mode 100644
index 0000000..70d6408
--- /dev/null
+++ b/src/include/access/undodiscard.h
@@ -0,0 +1,31 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoinsert.h
+ * undo discard definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undodiscard.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDODISCARD_H
+#define UNDODISCARD_H
+
+#include "access/undolog.h"
+#include "access/xlogdefs.h"
+#include "catalog/pg_class.h"
+#include "storage/lwlock.h"
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+extern void UndoDiscard(TransactionId xmin, bool *hibernate);
+
+#endif /* UNDODISCARD_H */
diff --git a/src/include/access/undoinsert.h b/src/include/access/undoinsert.h
index c333f00..5e208ad 100644
--- a/src/include/access/undoinsert.h
+++ b/src/include/access/undoinsert.h
@@ -46,5 +46,7 @@ extern void UndoSetPrepareSize(UnpackedUndoRecord *undorecords, int nrecords,
TransactionId xid, UndoPersistence upersistence);
extern UndoRecPtr UndoGetPrevUndoRecptr(UndoRecPtr urp, uint16 prevlen, UndoRecPtr prevurp);
extern void ResetUndoBuffers(void);
+extern void PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress);
+extern void UndoRecordUpdateTransInfo(int idx);
#endif /* UNDOINSERT_H */
diff --git a/src/include/access/undorecord.h b/src/include/access/undorecord.h
index 0dcf1b1..79eb173 100644
--- a/src/include/access/undorecord.h
+++ b/src/include/access/undorecord.h
@@ -20,6 +20,17 @@
#include "storage/buf.h"
#include "storage/off.h"
+typedef enum undorectype
+{
+ UNDO_INSERT,
+ UNDO_MULTI_INSERT,
+ UNDO_DELETE,
+ UNDO_INPLACE_UPDATE,
+ UNDO_UPDATE,
+ UNDO_XID_LOCK_ONLY,
+ UNDO_XID_MULTI_LOCK_ONLY,
+ UNDO_ITEMID_UNUSED
+} undorectype;
/*
* Every undo record begins with an UndoRecordHeader structure, which is
@@ -30,6 +41,7 @@
*/
typedef struct UndoRecordHeader
{
+ RmgrId urec_rmid; /* RMGR [XXX:TODO: this creates an alignment hole?] */
uint8 urec_type; /* record type code */
uint8 urec_info; /* flag bits */
uint16 urec_prevlen; /* length of previous record in bytes */
@@ -37,7 +49,7 @@ typedef struct UndoRecordHeader
/*
* Transaction id that has modified the tuple present in this undo record.
- * If this is older than oldestXidWithEpochHavingUndo, then we can
+ * If this is older than oldestXidHavingUndo, then we can
* consider the tuple in this undo record as visible.
*/
TransactionId urec_prevxid;
@@ -161,6 +173,7 @@ typedef struct UndoRecordPayload
*/
typedef struct UnpackedUndoRecord
{
+ RmgrId uur_rmid; /* rmgr ID */
uint8 uur_type; /* record type code */
uint8 uur_info; /* flag bits */
uint16 uur_prevlen; /* length of previous record */
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index ddaa633..f20a47c 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -419,6 +419,7 @@ extern XLogRecPtr XactLogAbortRecord(TimestampTz abort_time,
int xactflags, TransactionId twophase_xid,
const char *twophase_gid);
extern void xact_redo(XLogReaderState *record);
+extern void XactPerformUndoActionsIfPending(void);
/* xactdesc.c */
extern void xact_desc(StringInfo buf, XLogReaderState *record);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d18a1cd..b630d8c 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -225,6 +225,9 @@ extern bool XLOG_DEBUG;
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+/* Generate a 64-bit xid by using epoch and 32-bit xid. */
+#define MakeEpochXid(epoch, xid) \
+ ((epoch << 32) | (xid))
/* Checkpoint statistics */
typedef struct CheckpointStatsData
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 3c86037..18c64db 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -21,8 +21,10 @@
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
+#include "access/undorecord.h"
#include "datatype/timestamp.h"
#include "lib/stringinfo.h"
+#include "nodes/pg_list.h"
#include "pgtime.h"
#include "storage/block.h"
#include "storage/relfilenode.h"
@@ -294,9 +296,14 @@ typedef struct RmgrData
void (*rm_startup) (void);
void (*rm_cleanup) (void);
void (*rm_mask) (char *pagedata, BlockNumber blkno);
+ bool (*rm_undo) (List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock,
+ int options);
+ void (*rm_undo_desc) (StringInfo buf, UnpackedUndoRecord *record);
} RmgrData;
-extern const RmgrData RmgrTable[];
+extern PGDLLIMPORT const RmgrData RmgrTable[];
/*
* Exported to support xlog switching from checkpointer
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index a4aa83b..8a52124 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -61,6 +61,13 @@ typedef struct CheckPoint
* set to InvalidTransactionId.
*/
TransactionId oldestActiveXid;
+
+ /*
+ * Oldest transaction id which is having undo. Include this value
+ * in the checkpoint record so that whenever server starts we get proper
+ * value.
+ */
+ uint32 oldestXidHavingUndo;
} CheckPoint;
/* XLOG info values for XLOG rmgr */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index c9e3500..8edbbda 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -245,6 +245,7 @@ extern PGDLLIMPORT bool allowSystemTableMods;
extern PGDLLIMPORT int work_mem;
extern PGDLLIMPORT int maintenance_work_mem;
extern PGDLLIMPORT int max_parallel_maintenance_workers;
+extern PGDLLIMPORT int rollback_overflow_size;
extern int VacuumCostPageHit;
extern int VacuumCostPageMiss;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e6b22f4..319b36f 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -749,6 +749,7 @@ typedef enum BackendState
#define PG_WAIT_TIMEOUT 0x09000000U
#define PG_WAIT_IO 0x0A000000U
#define PG_WAIT_PAGE_TRANS_SLOT 0x0B000000U
+#define PG_WAIT_ROLLBACK_HT 0x0C000000U
/* ----------
* Wait Events - Activity
@@ -774,6 +775,8 @@ typedef enum
WAIT_EVENT_WAL_RECEIVER_MAIN,
WAIT_EVENT_WAL_SENDER_MAIN,
WAIT_EVENT_WAL_WRITER_MAIN,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN
} WaitEventActivity;
/* ----------
diff --git a/src/include/postmaster/discardworker.h b/src/include/postmaster/discardworker.h
new file mode 100644
index 0000000..f00c6c4
--- /dev/null
+++ b/src/include/postmaster/discardworker.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.h
+ * Exports from postmaster/discardworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/discardworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _DISCARDWORKER_H
+#define _DISCARDWORKER_H
+
+/*
+ * This function will perform multiple actions based on need. (a) retrieve
+ * transactions which have become all-visible and truncate the associated undo
+ * logs or will increment the tail pointer. (b) drop the buffers corresponding
+ * to truncated pages.
+ */
+extern void DiscardWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void DiscardWorkerRegister(void);
+
+#endif /* _DISCARDWORKER_H */
diff --git a/src/include/postmaster/undoloop.h b/src/include/postmaster/undoloop.h
new file mode 100644
index 0000000..21d91c3
--- /dev/null
+++ b/src/include/postmaster/undoloop.h
@@ -0,0 +1,89 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoloop.h
+ * Exports from postmaster/undoloop.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ *
+ * src/include/postmaster/undoloop.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOLOOP_H
+#define _UNDOLOOP_H
+
+#include "access/undoinsert.h"
+#include "utils/hsearch.h"
+#include "utils/relcache.h"
+
+
+/* Various options while executing the undo actions for the page. */
+#define UNDO_ACTION_UPDATE_TPD 0x0001
+
+/* Remembers the last seen RecentGlobalXmin */
+TransactionId latestRecentGlobalXmin;
+
+/*
+ * This function will read the undo records starting from the undo
+ * from_urecptr till to_urecptr and if to_urecptr is invalid then till the
+ * first undo location of transaction. This also discards the buffers by
+ * calling DropUndoBuffers for which undo log is removed. This function
+ * can be used by RollbackToSavePoint, by Rollback, by undoworker to complete
+ * the work of errored out transactions or when there is an error in single
+ * user mode.
+ */
+extern void execute_undo_actions(UndoRecPtr from_urecptr,
+ UndoRecPtr to_urecptr, bool nopartial, bool rewind, bool rellock);
+extern void process_and_execute_undo_actions_page(UndoRecPtr from_urecptr,
+ Relation rel, Buffer buffer, uint32 epoch,
+ TransactionId xid, int slot_no);
+
+/*
+ * This function will be responsible to truncate the undo logs
+ * for transactions that become all-visible after RecentGlobalXmin has
+ * advanced (value is different than latestRecentGlobalXmin). The easiest
+ * way could be to traverse the undo log array that contains least transaction
+ * id for that undo log and see if it precedes RecentGlobalXmin, then start
+ * discarding the undo log for that transaction (moving the tail pointer of
+ * undo log) till it finds the transaction which is not all-visible. This also
+ * discards the buffers by calling ForgetBuffer for which undo log is
+ * removed. This function can be invoked by undoworker or after commit in
+ * single user mode.
+ */
+extern void recover_undo_pages();
+
+/*
+ * To increase the efficiency of the zheap system, we create a hash table for
+ * the rollbacks. All the rollback requests exceeding certain threshold, are
+ * pushed to this table. Undo worker starts reading the entries from this hash
+ * table one at a time, performs undo actions related to the respective xid and
+ * removes them from the hash table. This way backend is free from performing the
+ * undo actions in case of heavy rollbacks. The data structures and the routines
+ * required for this infrastructure are as follows.
+ */
+
+/* This is the data structure for each hash table entry for rollbacks. */
+typedef struct RollbackHashEntry
+{
+ UndoRecPtr start_urec_ptr;
+ UndoRecPtr end_urec_ptr;
+ Oid dbid;
+} RollbackHashEntry;
+
+extern bool RollbackHTIsFull(void);
+
+/* To push the rollback requests from backend to the respective hash table */
+extern bool PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr,
+ Oid dbid);
+
+/* To perform the undo actions reading from the hash table */
+extern void RollbackFromHT(Oid dbid);
+/* To calculate the size of the hash table size for rollabcks. */
+extern int RollbackHTSize(void);
+
+/* To initialize the hash table in shared memory for rollbacks. */
+extern void InitRollbackHashTable(void);
+extern List *RollbackHTGetDBList(void);
+extern bool ConditionTransactionUndoActionLock(TransactionId xid);
+extern void TransactionUndoActionLockRelease(TransactionId xid);
+#endif /* _UNDOLOOP_H */
diff --git a/src/include/postmaster/undoworker.h b/src/include/postmaster/undoworker.h
new file mode 100644
index 0000000..c277e01
--- /dev/null
+++ b/src/include/postmaster/undoworker.h
@@ -0,0 +1,39 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.h
+ * Exports from postmaster/undoworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/undoworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOWORKER_H
+#define _UNDOWORKER_H
+
+/* GUC options */
+/* undo worker sleep time between rounds */
+extern int UndoWorkerDelay;
+
+/*
+ * This function will perform multiple actions based on need. (a) retreive
+ * transaction and its corresponding undopoiter from shared memory queue and
+ * call undoloop to perform undo actions. After applying all the undo records
+ * for a particular transaction, it will increment the tail pointer in undo log.
+ * (b) it needs to retrieve transactions which have become all-visible and truncate
+ * the associated undo logs or will increment the tail pointer. (c) udjust the
+ * number of undo workers based on the work required to perform undo actions
+ * (it could be size of shared memory queue containing transactions that needs
+ * aborts). (d) drop the buffers corresponding to truncated pages (e) Sleep for
+ * UndoWorkerDelay, if there is no more work.
+ */
+extern void UndoWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void UndoLauncherRegister(void);
+extern void UndoLauncherShmemInit(void);
+extern Size UndoLauncherShmemSize(void);
+extern void UndoLauncherMain(Datum main_arg);
+extern void UndoWorkerMain(Datum main_arg);
+
+#endif /* _UNDOWORKER_H */
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 16b927c..1d3ed2e 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -150,6 +150,8 @@ typedef enum LockTagType
LOCKTAG_VIRTUALTRANSACTION, /* virtual transaction (ditto) */
/* ID info for a virtual transaction is its VirtualTransactionId */
LOCKTAG_SPECULATIVE_TOKEN, /* speculative insertion Xid and token */
+ /* ID info for an transaction undoaction is transaction id */
+ LOCKTAG_TRANSACTION_UNDOACTION, /* transaction (waiting for undoaction) */
/* ID info for a transaction is its TransactionId */
LOCKTAG_OBJECT, /* non-relation database object */
/* ID info for an object is DB OID + CLASS OID + OBJECT OID + SUBID */
@@ -246,6 +248,14 @@ typedef struct LOCKTAG
(locktag).locktag_type = LOCKTAG_SPECULATIVE_TOKEN, \
(locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+#define SET_LOCKTAG_TRANSACTION_UNDOACTION(locktag,xid) \
+ ((locktag).locktag_field1 = (xid), \
+ (locktag).locktag_field2 = 0, \
+ (locktag).locktag_field3 = 0, \
+ (locktag).locktag_field4 = 0, \
+ (locktag).locktag_type = LOCKTAG_TRANSACTION_UNDOACTION, \
+ (locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+
#define SET_LOCKTAG_OBJECT(locktag,dboid,classoid,objoid,objsubid) \
((locktag).locktag_field1 = (dboid), \
(locktag).locktag_field2 = (classoid), \
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index d203acb..efde86d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -270,6 +270,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Oldest transaction id which is having undo. */
+ pg_atomic_uint32 oldestXidHavingUndo;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
1.8.3.1
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: Undo worker and transaction rollback
2018-11-05 11:40 Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-05 12:32 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-30 15:13 ` Re: Undo worker and transaction rollback Dmitry Dolgov <[email protected]>
2018-12-03 12:44 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-12-04 09:43 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-12-31 05:34 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2019-01-11 04:09 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
@ 2019-01-23 04:09 ` Amit Kapila <[email protected]>
2019-01-24 09:01 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Amit Kapila @ 2019-01-23 04:09 UTC (permalink / raw)
To: Dilip Kumar <[email protected]>; +Cc: Dmitry Dolgov <[email protected]>; pgsql-hackers
On Fri, Jan 11, 2019 at 9:39 AM Dilip Kumar <[email protected]> wrote:
>
> On Mon, Dec 31, 2018 at 11:04 AM Dilip Kumar <[email protected]> wrote:
>>
>> On Tue, Dec 4, 2018 at 3:13 PM Dilip Kumar <[email protected]> wrote:
>>
>> After the latest change in undo interface patch[1], this patch need to
>> be rebased. Attaching the rebased version of the patch.
>>
>> [1]https://www.postgresql.org/message-id/CAFiTN-tfquvm6tnWFaJNYYz-vSY%3D%2BSQTMv%2BFv1jPozTrW4mtKw%40ma...
>>
>
> I have rebased this patch on top of latest version of undolog and undo-interface patch set [1]
>
I have started reviewing your patch and below are some initial
comments. To be clear to everyone, there is some part of this patch
for which I have also written code and I am also involved in the
high-level design of this work, but I have never done the detailed
review of this patch which I am planning to do now. I think it will
be really good if some other hackers also review this patch especially
the interaction with transaction machinery and how the error rollbacks
work. I have few comments below in that regard as well and I have
requested to add more comments to explain that part of the patch so
that it will be easier to understand how this works.
Few comments:
------------------------
1.
+void
+undoaction_desc(StringInfo buf, XLogReaderState *record)
+{
+ char *rec = XLogRecGetData(record);
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ if (info == XLOG_UNDO_PAGE)
+ {
+ uint8 *flags = (uint8 *) rec;
+
+ appendStringInfo(buf, "page_contains_tpd_slot: %c ",
+ (*flags & XLU_PAGE_CONTAINS_TPD_SLOT) ? 'T' : 'F');
+ appendStringInfo(buf, "is_page_initialized: %c ",
+ (*flags & XLU_INIT_PAGE) ? 'T' : 'F');
+ if (*flags & XLU_PAGE_CONTAINS_TPD_SLOT)
Do we need handling of TPD in this patch? This is mainly tied to
zheap, so, I think we can exclude it from this patch.
2.
const char *
+undoaction_identify(uint8 info)
+{
+ const char *id = NULL;
+
+ switch (info & ~XLR_INFO_MASK)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ id = "UNDO APPLY PROGRESS";
+ break;
+ }
+
+ return id;
+}
Don't we need to handle the case of XLOG_UNDO_PAGE in this function?
3.
@@ -1489,6 +1504,34 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
{
..
+ /*
+ * Perform undo actions, if there are undologs for this transaction.
+ * We need to perform undo actions while we are still in transaction.
+ * Never push rollbacks of temp tables to undo worker.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (end_urec_ptr[i] != InvalidUndoRecPtr && !isCommit)
+ {
+ bool result = false;
+ uint64 rollback_size = 0;
+
+ if (i != UNDO_TEMP)
+ rollback_size = end_urec_ptr[i] - start_urec_ptr[i];
+
+ if (rollback_size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(end_urec_ptr[i], start_urec_ptr[i], InvalidOid);
The comment and code don't seem to be completely in sync. It seems
for rollback_overflow_size as 0, we will push the undo for temp tables
as well. I think you should have a stronger check here.
4.
+ /*
+ * We need the locations of start and end undo record pointers when rollbacks
+ * are to be performed for prepared transactions using zheap relations.
+ */
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
} TwoPhaseFileHeader;
I think you can expand this comment a bit by telling the reason why
you need to store these in the file. It seems it is because you want
to rollback prepared transactions after recovery.
5.
@@ -284,10 +284,21 @@ SetTransactionIdLimit(TransactionId
oldest_datfrozenxid, Oid oldest_datoid)
TransactionId xidStopLimit;
TransactionId xidWrapLimit;
TransactionId curXid;
+ TransactionId oldestXidHavingUndo;
Assert(TransactionIdIsNormal(oldest_datfrozenxid));
/*
+ * To determine the last safe xid that can be allocated, we need to
+ * consider oldestXidHavingUndo. The oldestXidHavingUndo will be only
+ * valid for zheap storage engine, so it won't impact any other storage
+ * engine.
+ */
+ oldestXidHavingUndo = pg_atomic_read_u32(&ProcGlobal->oldestXidHavingUndo);
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ oldest_datfrozenxid = Min(oldest_datfrozenxid, oldestXidHavingUndo);
+
Here, I think we need to mention the reason as well in the comments
why we need to care about oldestXidHavingUndo.
6.
+/* Location in undo log from where to start applying the undo actions. */
+static UndoRecPtr UndoActionStartPtr[UndoPersistenceLevels] =
+
{InvalidUndoRecPtr,
+
InvalidUndoRecPtr,
+
InvalidUndoRecPtr};
+
+/* Location in undo log up to which undo actions need to be applied. */
+static UndoRecPtr UndoActionEndPtr[UndoPersistenceLevels] =
+
{InvalidUndoRecPtr,
+
InvalidUndoRecPtr,
+
InvalidUndoRecPtr};
+
+/* Do we need to perform any undo actions? */
+static bool PerformUndoActions = false;
Normally, we track to and from undo locations for each transaction
state. I think these are used for error rollbacks which have
different handling. If so, can we write a few comments here to
explain how that works and then it will be easier to understand the
purpose of these variables?
7.
+static void
+AtAbort_Rollback(void)
{
..
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (CurrentTransactionState->state == TRANS_INPROGRESS)
..
}
As such this code is okay, but it is better to use IsTransactionState
as that is what we commonly used throughout the code for the same
purpose. The same change is required in AtSubAbort_Rollback.
8.
+static void
+AtAbort_Rollback(void)
{
..
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (CurrentTransactionState->state == TRANS_INPROGRESS)
..
}
It is not clear from above commentary and code how the rollbacks will
work if we get the error while executing undo actions. Basically, how
will PerformUndoActions will be set in such a case? I think you are
going to set it during AbortCurrentTransaction, but if that is the
case, can't we set the values of UndoActionStartPtr and
UndoActionEndPtr at that time? IIUC, these two variables are used
mainly when we want to postpone executing undo actions when we are not
in a good transaction state (like during error rollbacks) and
PerformUndoActions will indicate whether there are any pending
actions, so I feel these three variables should be set together. If
that is not the case, I think we need to add more comments to explain
the same.
9.
@@ -2594,6 +2676,7 @@ AbortTransaction(void)
..
/*
* set the current transaction state information appropriately during the
* abort processing
*/
s->state = TRANS_ABORT;
*/
AfterTriggerEndXact(false); /* 'false' means it's abort */
AtAbort_Portals();
+ AtAbort_Rollback();
..
+AtAbort_Rollback(void)
{
..
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (CurrentTransactionState->state == TRANS_INPROGRESS)
The function AtAbort_Rollback is called only from one place
AbortTransaction in the patch and by that time we have already set the
transaction state as TRANS_ABORT, so it is never going to get the
chance to execute the undo actions. How will it work when the user has
specifically given Rollback and we want to execute the undo actions?
Why can't we do this in UserAbortTransactionBlock? There is a comment
indicating the correctness of this method "/* XXX: TODO: check this
logic, which was moved out of UserAbortTransactionBlock */", but it is
not very clear why we need to do this in abort path only.
10.
@@ -4811,6 +5243,7 @@ AbortSubTransaction(void)
s->parent->subTransactionId,
s->curTransactionOwner,
s->parent->curTransactionOwner);
+ AtSubAbort_Rollback(s);
I see a similar problem as mentioned in point-9 in
AtSubAbort_Rollback. I think there is one problem here which is that
if executing the undo actions are postponed during Rollback To
Savepoint, then we would have released the locks and some other
backend which was supposed to wait on subtransaction will not wait and
might update the tuple or in some cases need to perform actions. I
understand this is more of a zheap specific stuff, but I feel in
general also we should execute undo actions of Rollback To Savepoint
immediately; if not, you might also need to invent some mechanism to
transfer (to and from) undo record pointers from the sub-transaction
state.
11.
@@ -2803,6 +2886,12 @@ void
CommitTransactionCommand(void)
{
..
+
+ /*
+ * Update the end undo record pointer if it's not valid with
+ * the currently popped transaction's end undo record pointer.
+ * This is particularly required when the first command of
+ * the transaction is of type which does not require an undo,
+ * e.g. savepoint x.
+ * Accordingly, update the start undo record pointer.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(end_urec_ptr[i]))
+ end_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(s->start_urec_ptr[i]))
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
..
}
I am not able to understand the above piece of code. Basically, you
have written in comments that this is required when the first command
in a transaction is of the type which doesn't generate undo, but there
is no explanation why it is required for that case. Also, the comment
and code don't seem to match because of below points:
(a) The comment says "Update the end undo record pointer if it's not
valid with the currently popped transaction's end undo record
pointer." and the code is not checking current transactions end
pointer validity, so it is not clear to me what you want to say here.
(b) The comment says "Accordingly, update the start undo record
pointer.". However from start undo record pointer, you are checking
the current state's start pointer which is different from what you do
for end pointer.
(c) How will the saved start_urec_ptr and end_urec_ptr be used after this?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: Undo worker and transaction rollback
2018-11-05 11:40 Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-05 12:32 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-30 15:13 ` Re: Undo worker and transaction rollback Dmitry Dolgov <[email protected]>
2018-12-03 12:44 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-12-04 09:43 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-12-31 05:34 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2019-01-11 04:09 ` Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2019-01-23 04:09 ` Re: Undo worker and transaction rollback Amit Kapila <[email protected]>
@ 2019-01-24 09:01 ` Dilip Kumar <[email protected]>
0 siblings, 0 replies; 37+ messages in thread
From: Dilip Kumar @ 2019-01-24 09:01 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Dmitry Dolgov <[email protected]>; pgsql-hackers
On Wed, Jan 23, 2019 at 9:39 AM Amit Kapila <[email protected]> wrote:
>
> On Fri, Jan 11, 2019 at 9:39 AM Dilip Kumar <[email protected]> wrote:
> >
> > On Mon, Dec 31, 2018 at 11:04 AM Dilip Kumar <[email protected]> wrote:
> >>
> >> On Tue, Dec 4, 2018 at 3:13 PM Dilip Kumar <[email protected]> wrote:
> >>
> >> After the latest change in undo interface patch[1], this patch need to
> >> be rebased. Attaching the rebased version of the patch.
> >>
> >> [1]https://www.postgresql.org/message-id/CAFiTN-tfquvm6tnWFaJNYYz-vSY%3D%2BSQTMv%2BFv1jPozTrW4mtKw%40ma...
> >>
> >
> > I have rebased this patch on top of latest version of undolog and undo-interface patch set [1]
> >
>
> I have started reviewing your patch and below are some initial
> comments. To be clear to everyone, there is some part of this patch
> for which I have also written code and I am also involved in the
> high-level design of this work, but I have never done the detailed
> review of this patch which I am planning to do now. I think it will
> be really good if some other hackers also review this patch especially
> the interaction with transaction machinery and how the error rollbacks
> work. I have few comments below in that regard as well and I have
> requested to add more comments to explain that part of the patch so
> that it will be easier to understand how this works.
Thanks for review please find my reply inline.
>
> Few comments:
> ------------------------
> 1.
> +void
> +undoaction_desc(StringInfo buf, XLogReaderState *record)
> +{
> + char *rec = XLogRecGetData(record);
> + uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
> +
> + if (info == XLOG_UNDO_PAGE)
> + {
> + uint8 *flags = (uint8 *) rec;
> +
> + appendStringInfo(buf, "page_contains_tpd_slot: %c ",
> + (*flags & XLU_PAGE_CONTAINS_TPD_SLOT) ? 'T' : 'F');
> + appendStringInfo(buf, "is_page_initialized: %c ",
> + (*flags & XLU_INIT_PAGE) ? 'T' : 'F');
> + if (*flags & XLU_PAGE_CONTAINS_TPD_SLOT)
>
> Do we need handling of TPD in this patch? This is mainly tied to
> zheap, so, I think we can exclude it from this patch.
Fixed
>
> 2.
> const char *
> +undoaction_identify(uint8 info)
> +{
> + const char *id = NULL;
> +
> + switch (info & ~XLR_INFO_MASK)
> + {
> + case XLOG_UNDO_APPLY_PROGRESS:
> + id = "UNDO APPLY PROGRESS";
> + break;
> + }
> +
> + return id;
> +}
>
> Don't we need to handle the case of XLOG_UNDO_PAGE in this function?
Fixed
>
> 3.
> @@ -1489,6 +1504,34 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
> {
> ..
> + /*
> + * Perform undo actions, if there are undologs for this transaction.
> + * We need to perform undo actions while we are still in transaction.
> + * Never push rollbacks of temp tables to undo worker.
> + */
> + for (i = 0; i < UndoPersistenceLevels; i++)
> + {
> + if (end_urec_ptr[i] != InvalidUndoRecPtr && !isCommit)
> + {
> + bool result = false;
> + uint64 rollback_size = 0;
> +
> + if (i != UNDO_TEMP)
> + rollback_size = end_urec_ptr[i] - start_urec_ptr[i];
> +
> + if (rollback_size >= rollback_overflow_size * 1024 * 1024)
> + result = PushRollbackReq(end_urec_ptr[i], start_urec_ptr[i], InvalidOid);
>
> The comment and code don't seem to be completely in sync. It seems
> for rollback_overflow_size as 0, we will push the undo for temp tables
> as well. I think you should have a stronger check here.
Yeah, it's a problem. I have fixed it.
>
> 4.
> + /*
> + * We need the locations of start and end undo record pointers when rollbacks
> + * are to be performed for prepared transactions using zheap relations.
> + */
> + UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
> + UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
> } TwoPhaseFileHeader;
>
>
> I think you can expand this comment a bit by telling the reason why
> you need to store these in the file. It seems it is because you want
> to rollback prepared transactions after recovery.
Done
>
> 5.
> @@ -284,10 +284,21 @@ SetTransactionIdLimit(TransactionId
> oldest_datfrozenxid, Oid oldest_datoid)
> TransactionId xidStopLimit;
> TransactionId xidWrapLimit;
> TransactionId curXid;
> + TransactionId oldestXidHavingUndo;
>
> Assert(TransactionIdIsNormal(oldest_datfrozenxid));
>
> /*
> + * To determine the last safe xid that can be allocated, we need to
> + * consider oldestXidHavingUndo. The oldestXidHavingUndo will be only
> + * valid for zheap storage engine, so it won't impact any other storage
> + * engine.
> + */
> + oldestXidHavingUndo = pg_atomic_read_u32(&ProcGlobal->oldestXidHavingUndo);
> + if (TransactionIdIsValid(oldestXidHavingUndo))
> + oldest_datfrozenxid = Min(oldest_datfrozenxid, oldestXidHavingUndo);
> +
>
> Here, I think we need to mention the reason as well in the comments
> why we need to care about oldestXidHavingUndo.
Done
>
> 6.
> +/* Location in undo log from where to start applying the undo actions. */
> +static UndoRecPtr UndoActionStartPtr[UndoPersistenceLevels] =
> +
> {InvalidUndoRecPtr,
> +
> InvalidUndoRecPtr,
> +
> InvalidUndoRecPtr};
> +
> +/* Location in undo log up to which undo actions need to be applied. */
> +static UndoRecPtr UndoActionEndPtr[UndoPersistenceLevels] =
> +
> {InvalidUndoRecPtr,
> +
> InvalidUndoRecPtr,
> +
> InvalidUndoRecPtr};
> +
> +/* Do we need to perform any undo actions? */
> +static bool PerformUndoActions = false;
>
> Normally, we track to and from undo locations for each transaction
> state. I think these are used for error rollbacks which have
> different handling. If so, can we write a few comments here to
> explain how that works and then it will be easier to understand the
> purpose of these variables?
Done
>
> 7.
> +static void
> +AtAbort_Rollback(void)
> {
> ..
> + /*
> + * If we are in a valid transaction state then execute the undo action here
> + * itself, otherwise we have already stored the required information for
> + * executing the undo action later.
> + */
> + if (CurrentTransactionState->state == TRANS_INPROGRESS)
> ..
> }
>
> As such this code is okay, but it is better to use IsTransactionState
> as that is what we commonly used throughout the code for the same
> purpose. The same change is required in AtSubAbort_Rollback.
>
AtAbort_Rollback is removed now as per comment 9.
> 8.
> +static void
> +AtAbort_Rollback(void)
> {
> ..
> + /*
> + * Remember the required information for performing undo actions. So that
> + * if there is any failure in executing the undo action we can execute
> + * it later.
> + */
> + memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
> + memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
> +
> + /*
> + * If we are in a valid transaction state then execute the undo action here
> + * itself, otherwise we have already stored the required information for
> + * executing the undo action later.
> + */
> + if (CurrentTransactionState->state == TRANS_INPROGRESS)
> ..
> }
>
> It is not clear from above commentary and code how the rollbacks will
> work if we get the error while executing undo actions. Basically, how
> will PerformUndoActions will be set in such a case? I think you are
> going to set it during AbortCurrentTransaction, but if that is the
> case, can't we set the values of UndoActionStartPtr and
> UndoActionEndPtr at that time? IIUC, these two variables are used
> mainly when we want to postpone executing undo actions when we are not
> in a good transaction state (like during error rollbacks) and
> PerformUndoActions will indicate whether there are any pending
> actions, so I feel these three variables should be set together. If
> that is not the case, I think we need to add more comments to explain
> the same.
AtAbort_Rollback is removed now as per comment 9.
>
> 9.
> @@ -2594,6 +2676,7 @@ AbortTransaction(void)
> ..
> /*
> * set the current transaction state information appropriately during the
> * abort processing
> */
> s->state = TRANS_ABORT;
> */
> AfterTriggerEndXact(false); /* 'false' means it's abort */
> AtAbort_Portals();
> + AtAbort_Rollback();
>
> ..
>
> +AtAbort_Rollback(void)
> {
> ..
> + /*
> + * If we are in a valid transaction state then execute the undo action here
> + * itself, otherwise we have already stored the required information for
> + * executing the undo action later.
> + */
> + if (CurrentTransactionState->state == TRANS_INPROGRESS)
>
> The function AtAbort_Rollback is called only from one place
> AbortTransaction in the patch and by that time we have already set the
> transaction state as TRANS_ABORT, so it is never going to get the
> chance to execute the undo actions. How will it work when the user has
> specifically given Rollback and we want to execute the undo actions?
> Why can't we do this in UserAbortTransactionBlock? There is a comment
> indicating the correctness of this method "/* XXX: TODO: check this
> logic, which was moved out of UserAbortTransactionBlock */", but it is
> not very clear why we need to do this in abort path only.
I agree with your point. Now we are handling in UserAbortTransactionBlock.
>
> 10.
> @@ -4811,6 +5243,7 @@ AbortSubTransaction(void)
> s->parent->subTransactionId,
> s->curTransactionOwner,
> s->parent->curTransactionOwner);
> + AtSubAbort_Rollback(s);
>
> I see a similar problem as mentioned in point-9 in
> AtSubAbort_Rollback. I think there is one problem here which is that
> if executing the undo actions are postponed during Rollback To
> Savepoint, then we would have released the locks and some other
> backend which was supposed to wait on subtransaction will not wait and
> might update the tuple or in some cases need to perform actions. I
> understand this is more of a zheap specific stuff, but I feel in
> general also we should execute undo actions of Rollback To Savepoint
> immediately; if not, you might also need to invent some mechanism to
> transfer (to and from) undo record pointers from the sub-transaction
> state.
Moved execution to RollbackToSavepoint function
>
> 11.
> @@ -2803,6 +2886,12 @@ void
> CommitTransactionCommand(void)
> {
> ..
> +
> + /*
> + * Update the end undo record pointer if it's not valid with
> + * the currently popped transaction's end undo record pointer.
> + * This is particularly required when the first command of
> + * the transaction is of type which does not require an undo,
> + * e.g. savepoint x.
> + * Accordingly, update the start undo record pointer.
> + */
> + for (i = 0; i < UndoPersistenceLevels; i++)
> + {
> + if (!UndoRecPtrIsValid(end_urec_ptr[i]))
> + end_urec_ptr[i] = s->latest_urec_ptr[i];
> +
> + if (UndoRecPtrIsValid(s->start_urec_ptr[i]))
> + start_urec_ptr[i] = s->start_urec_ptr[i];
> + }
> ..
> }
>
> I am not able to understand the above piece of code. Basically, you
> have written in comments that this is required when the first command
> in a transaction is of the type which doesn't generate undo, but there
> is no explanation why it is required for that case. Also, the comment
> and code don't seem to match because of below points:
> (a) The comment says "Update the end undo record pointer if it's not
> valid with the currently popped transaction's end undo record
> pointer." and the code is not checking current transactions end
> pointer validity, so it is not clear to me what you want to say here.
end undo record pointer (recently renamed to latest_urec_ptr) is the first valid
latest_urec_ptr while popping out the transaction from transaction
stack. So once
we got a valid latest we don't need to update, but I agree there are
current extra
statement getting executed i.e. even if the current transaction's
latest_urec_ptr is
not valid it's getting assiged. So added extra check to avoid extra
assignment cycles.
> (b) The comment says "Accordingly, update the start undo record
> pointer.". However from start undo record pointer, you are checking
> the current state's start pointer which is different from what you do
> for end pointer.
Start pointer we have to overwrite every time so that we can reach to
the first start pointer of the transaction
> (c) How will the saved start_urec_ptr and end_urec_ptr be used after this?
I have tried to explain it in the comment.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] 0001-Undo-worker-and-transaction-rollback_v8.patch (136.1K, ../../CAFiTN-vZan-=ZtbbBaxEMDt6aj9Z79kjRe_GKovRgBEHXM+6ow@mail.gmail.com/2-0001-Undo-worker-and-transaction-rollback_v8.patch)
download | inline diff:
From 68d5093977f3c82b94a1e9b8746cd3a30e1f731d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <[email protected]>
Date: Thu, 10 Jan 2019 12:06:08 +0530
Subject: [PATCH] Undo worker and transaction rollback
Patch provides mechanism for discarding the older undo when they
are no longer relavent. It also provide a mechanism for rolling
back the effect of the aborted transactiona. It handles the rollbacks
directly from backends as well as from undo worker for the larger
transaction.
Dilip Kumar, with help from Rafia Sabih, Amit Kapila, Mithun.CY, Thomas Munro and Kuntal Ghosh
---
src/backend/access/rmgrdesc/Makefile | 3 +-
src/backend/access/rmgrdesc/undoactiondesc.c | 67 +++
src/backend/access/rmgrdesc/xlogdesc.c | 4 +-
src/backend/access/transam/rmgr.c | 5 +-
src/backend/access/transam/twophase.c | 53 +-
src/backend/access/transam/varsup.c | 12 +
src/backend/access/transam/xact.c | 454 +++++++++++++++-
src/backend/access/transam/xlog.c | 20 +
src/backend/access/undo/Makefile | 2 +-
src/backend/access/undo/undoaction.c | 719 ++++++++++++++++++++++++++
src/backend/access/undo/undoactionxlog.c | 49 ++
src/backend/access/undo/undodiscard.c | 447 ++++++++++++++++
src/backend/access/undo/undoinsert.c | 52 +-
src/backend/access/undo/undorecord.c | 3 +
src/backend/commands/vacuum.c | 10 +
src/backend/postmaster/Makefile | 4 +-
src/backend/postmaster/bgworker.c | 11 +
src/backend/postmaster/discardworker.c | 170 ++++++
src/backend/postmaster/pgstat.c | 7 +-
src/backend/postmaster/postmaster.c | 7 +
src/backend/postmaster/undoworker.c | 644 +++++++++++++++++++++++
src/backend/replication/logical/decode.c | 4 +
src/backend/storage/ipc/ipci.c | 6 +
src/backend/storage/lmgr/lwlocknames.txt | 2 +
src/backend/storage/lmgr/proc.c | 2 +
src/backend/utils/adt/lockfuncs.c | 1 +
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc.c | 11 +
src/backend/utils/misc/pg_controldata.c | 9 +-
src/backend/utils/misc/postgresql.conf.sample | 7 +
src/bin/pg_controldata/pg_controldata.c | 2 +
src/bin/pg_resetwal/pg_resetwal.c | 7 +
src/bin/pg_rewind/parsexlog.c | 2 +-
src/bin/pg_waldump/rmgrdesc.c | 3 +-
src/include/access/rmgr.h | 2 +-
src/include/access/rmgrlist.h | 47 +-
src/include/access/transam.h | 4 +
src/include/access/twophase.h | 2 +-
src/include/access/undoaction.h | 28 +
src/include/access/undoaction_xlog.h | 67 +++
src/include/access/undodiscard.h | 31 ++
src/include/access/undoinsert.h | 2 +
src/include/access/undorecord.h | 15 +-
src/include/access/xact.h | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlog_internal.h | 9 +-
src/include/catalog/pg_control.h | 7 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +
src/include/postmaster/discardworker.h | 25 +
src/include/postmaster/undoloop.h | 90 ++++
src/include/postmaster/undoworker.h | 39 ++
src/include/storage/lock.h | 10 +
src/include/storage/proc.h | 2 +
54 files changed, 3141 insertions(+), 47 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/undoactiondesc.c
create mode 100644 src/backend/access/undo/undoaction.c
create mode 100644 src/backend/access/undo/undoactionxlog.c
create mode 100644 src/backend/access/undo/undodiscard.c
create mode 100644 src/backend/postmaster/discardworker.c
create mode 100644 src/backend/postmaster/undoworker.c
create mode 100644 src/include/access/undoaction.h
create mode 100644 src/include/access/undoaction_xlog.h
create mode 100644 src/include/access/undodiscard.h
create mode 100644 src/include/postmaster/discardworker.h
create mode 100644 src/include/postmaster/undoloop.h
create mode 100644 src/include/postmaster/undoworker.h
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index 91ad1ef..640d37f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = brindesc.o clogdesc.o committsdesc.o dbasedesc.o genericdesc.o \
gindesc.o gistdesc.o hashdesc.o heapdesc.o logicalmsgdesc.o \
mxactdesc.o nbtdesc.o relmapdesc.o replorigindesc.o seqdesc.o \
- smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undologdesc.o xactdesc.o xlogdesc.o
+ smgrdesc.o spgdesc.o standbydesc.o tblspcdesc.o undoactiondesc.o \
+ undologdesc.o xactdesc.o xlogdesc.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/rmgrdesc/undoactiondesc.c b/src/backend/access/rmgrdesc/undoactiondesc.c
new file mode 100644
index 0000000..a48e837
--- /dev/null
+++ b/src/backend/access/rmgrdesc/undoactiondesc.c
@@ -0,0 +1,67 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactiondesc.c
+ * rmgr descriptor routines for access/undo/undoactionxlog.c
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/access/rmgrdesc/undoactiondesc.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+
+void
+undoaction_desc(StringInfo buf, XLogReaderState *record)
+{
+ char *rec = XLogRecGetData(record);
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ if (info == XLOG_UNDO_PAGE)
+ {
+ uint8 *flags = (uint8 *) rec;
+
+ appendStringInfo(buf, "is_page_initialized: %c ",
+ (*flags & XLU_INIT_PAGE) ? 'T' : 'F');
+ }
+ if (info == XLOG_UNDO_RESET_SLOT)
+ {
+ xl_undoaction_reset_slot *xlrec = (xl_undoaction_reset_slot *) rec;
+
+ appendStringInfo(buf, "urec_ptr %lu slot %u",
+ xlrec->urec_ptr, xlrec->trans_slot_id);
+ }
+ else if (info == XLOG_UNDO_APPLY_PROGRESS)
+ {
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) rec;
+
+ appendStringInfo(buf, "urec_ptr %lu progress %u",
+ xlrec->urec_ptr, xlrec->progress);
+ }
+}
+
+const char *
+undoaction_identify(uint8 info)
+{
+ const char *id = NULL;
+
+ switch (info & ~XLR_INFO_MASK)
+ {
+ case XLOG_UNDO_PAGE:
+ id = "UNDO PAGE";
+ break;
+ case XLOG_UNDO_RESET_SLOT:
+ id = "UNDO RESET SLOT";
+ break;
+ case XLOG_UNDO_APPLY_PROGRESS:
+ id = "UNDO APPLY PROGRESS";
+ break;
+ }
+
+ return id;
+}
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 7f73251..2d4c736 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -47,7 +47,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
"tli %u; prev tli %u; fpw %s; xid %u:%u; oid %u; multi %u; offset %u; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
- "oldest running xid %u; %s",
+ "oldest running xid %u; "
+ "oldest xid having undo %u ; %s",
(uint32) (checkpoint->redo >> 32), (uint32) checkpoint->redo,
checkpoint->ThisTimeLineID,
checkpoint->PrevTimeLineID,
@@ -63,6 +64,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
checkpoint->oldestCommitTsXid,
checkpoint->newestCommitTsXid,
checkpoint->oldestActiveXid,
+ checkpoint->oldestXidHavingUndo,
(info == XLOG_CHECKPOINT_SHUTDOWN) ? "shutdown" : "online");
}
else if (info == XLOG_NEXTOID)
diff --git a/src/backend/access/transam/rmgr.c b/src/backend/access/transam/rmgr.c
index 8b05374..6238240 100644
--- a/src/backend/access/transam/rmgr.c
+++ b/src/backend/access/transam/rmgr.c
@@ -18,6 +18,7 @@
#include "access/multixact.h"
#include "access/nbtxlog.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -31,8 +32,8 @@
#include "utils/relmapper.h"
/* must be kept in sync with RmgrData definition in xlog_internal.h */
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
- { name, redo, desc, identify, startup, cleanup, mask },
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
+ { name, redo, desc, identify, startup, cleanup, mask, undo, undo_desc },
const RmgrData RmgrTable[RM_MAX_ID + 1] = {
#include "access/rmgrlist.h"
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 9a8a6bb..d91853fb 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -93,6 +93,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/origin.h"
#include "replication/syncrep.h"
#include "replication/walsender.h"
@@ -915,6 +916,15 @@ typedef struct TwoPhaseFileHeader
uint16 gidlen; /* length of the GID - GID follows the header */
XLogRecPtr origin_lsn; /* lsn of this record at origin node */
TimestampTz origin_timestamp; /* time of prepare at origin node */
+ /*
+ * We need the locations of start and end undo record pointers when rollbacks
+ * are to be performed for prepared transactions using zheap relations.
+ * We need to store these information in file as user might rollback the
+ * prepared transaction after recovery and for that we need it's start and
+ * end undo locations.
+ */
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
} TwoPhaseFileHeader;
/*
@@ -989,7 +999,8 @@ save_state_data(const void *data, uint32 len)
* Initializes data structure and inserts the 2PC file header record.
*/
void
-StartPrepare(GlobalTransaction gxact)
+StartPrepare(GlobalTransaction gxact, UndoRecPtr *start_urec_ptr,
+ UndoRecPtr *end_urec_ptr)
{
PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
PGXACT *pgxact = &ProcGlobal->allPgXact[gxact->pgprocno];
@@ -1027,6 +1038,10 @@ StartPrepare(GlobalTransaction gxact)
&hdr.initfileinval);
hdr.gidlen = strlen(gxact->gid) + 1; /* Include '\0' */
+ /* save the start and end undo record pointers */
+ memcpy(hdr.start_urec_ptr, start_urec_ptr, sizeof(hdr.start_urec_ptr));
+ memcpy(hdr.end_urec_ptr, end_urec_ptr, sizeof(hdr.end_urec_ptr));
+
save_state_data(&hdr, sizeof(TwoPhaseFileHeader));
save_state_data(gxact->gid, hdr.gidlen);
@@ -1452,6 +1467,9 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
RelFileNode *delrels;
int ndelrels;
SharedInvalidationMessage *invalmsgs;
+ int i;
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr end_urec_ptr[UndoPersistenceLevels];
/*
* Validate the GID, and lock the GXACT to ensure that two backends do not
@@ -1489,6 +1507,39 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
invalmsgs = (SharedInvalidationMessage *) bufptr;
bufptr += MAXALIGN(hdr->ninvalmsgs * sizeof(SharedInvalidationMessage));
+ /* save the start and end undo record pointers */
+ memcpy(start_urec_ptr, hdr->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(end_urec_ptr, hdr->end_urec_ptr, sizeof(end_urec_ptr));
+
+ /*
+ * Perform undo actions, if there are undologs for this transaction,
+ * we need to perform undo actions while we are still in transaction.
+ * Never push rollbacks of temp tables to undo worker.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (end_urec_ptr[i] != InvalidUndoRecPtr && !isCommit)
+ {
+ bool result = false;
+
+ if (i != UNDO_TEMP)
+ {
+ uint64 rollback_size = 0;
+
+ rollback_size = end_urec_ptr[i] - start_urec_ptr[i];
+
+ if (rollback_size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(end_urec_ptr[i],
+ start_urec_ptr[i],
+ InvalidOid);
+ }
+
+ if (!result)
+ execute_undo_actions(end_urec_ptr[i], start_urec_ptr[i], true,
+ true, false);
+ }
+ }
+
/* compute latestXid among all children */
latestXid = TransactionIdLatest(xid, hdr->nsubxacts, children);
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index fe94fda..522d1ae 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -284,10 +284,22 @@ SetTransactionIdLimit(TransactionId oldest_datfrozenxid, Oid oldest_datoid)
TransactionId xidStopLimit;
TransactionId xidWrapLimit;
TransactionId curXid;
+ TransactionId oldestXidHavingUndo;
Assert(TransactionIdIsNormal(oldest_datfrozenxid));
/*
+ * To determine the last safe xid that can be allocated, we need to
+ * consider oldestXidHavingUndo because this is a oldest xid whose undo is
+ * not yet discarded so this is still a valid xid in system.
+ * The oldestXidHavingUndo will be only valid for zheap storage engine, so
+ * it won't impact any other storage engine.
+ */
+ oldestXidHavingUndo = pg_atomic_read_u32(&ProcGlobal->oldestXidHavingUndo);
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ oldest_datfrozenxid = Min(oldest_datfrozenxid, oldestXidHavingUndo);
+
+ /*
* The place where we actually get into deep trouble is halfway around
* from the oldest potentially-existing XID. (This calculation is
* probably off by one or two counts, because the special XIDs reduce the
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 8af75a9..bb871b3 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -42,6 +42,7 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/undoloop.h"
#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/origin.h"
@@ -280,6 +281,28 @@ typedef struct SubXactCallbackItem
static SubXactCallbackItem *SubXact_callbacks = NULL;
+/*
+ * Global states of transaction undo. Generally, we maintains these states
+ * in TransactionState but in case of error we remove the transaction state
+ * so we calculate the UndoActionStartPtr and UndoActionEndPtr based on user
+ * rollback command and store them in global state variables. We also set
+ * PerformUndoActions to true and this is indication for
+ * XactPerformUndoActionsIfPending whether we have any undo action to apply
+ * after each statement executed. After undo action is applied
+ * PerformUndoActions is reset.
+ */
+static UndoRecPtr UndoActionStartPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+static UndoRecPtr UndoActionEndPtr[UndoPersistenceLevels] =
+ {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+
+/* Do we need to perform any undo actions? */
+static bool PerformUndoActions = false;
/* local function prototypes */
static void AssignTransactionId(TransactionState s);
@@ -1835,6 +1858,7 @@ StartTransaction(void)
{
TransactionState s;
VirtualTransactionId vxid;
+ int i;
/*
* Let's just make sure the state stack is empty
@@ -1913,6 +1937,13 @@ StartTransaction(void)
nUnreportedXids = 0;
s->didLogXid = false;
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
/*
* must initialize resource-management stuff first
*/
@@ -2222,7 +2253,7 @@ CommitTransaction(void)
* NB: if you change this routine, better look at CommitTransaction too!
*/
static void
-PrepareTransaction(void)
+PrepareTransaction(UndoRecPtr *start_urec_ptr, UndoRecPtr *end_urec_ptr)
{
TransactionState s = CurrentTransactionState;
TransactionId xid = GetCurrentTransactionId();
@@ -2370,7 +2401,7 @@ PrepareTransaction(void)
* PREPARED; in particular, pay attention to whether things should happen
* before or after releasing the transaction's locks.
*/
- StartPrepare(gxact);
+ StartPrepare(gxact, start_urec_ptr, end_urec_ptr);
AtPrepare_Notify();
AtPrepare_Locks();
@@ -2493,7 +2524,6 @@ PrepareTransaction(void)
RESUME_INTERRUPTS();
}
-
/*
* AbortTransaction
*/
@@ -2803,6 +2833,12 @@ void
CommitTransactionCommand(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i;
+
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
switch (s->blockState)
{
@@ -2892,7 +2928,7 @@ CommitTransactionCommand(void)
* return to the idle state.
*/
case TBLOCK_PREPARE:
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, latest_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
break;
@@ -2938,6 +2974,28 @@ CommitTransactionCommand(void)
{
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
+
+ /*
+ * Compute latest_urec_ptr and start_urec_ptr to store them
+ * in TwoPhaseFileHeader if transaction is prepared so that
+ * we can execute its undo action if the transaction is rolled
+ * back at later stage.
+ * There is possibility that the lastest transaction which has
+ * a valid latest_urec_ptr is not the top most subtransaction
+ * in thectransaction stack and same is true with the
+ * start_urec_ptr. So while we are traversing the transaction
+ * stack we need to update latest_urec_ptr and start_urec_ptr
+ * appropriately.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]) &&
+ UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(s->start_urec_ptr[i]))
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
} while (s->blockState == TBLOCK_SUBCOMMIT);
/* If we had a COMMIT command, finish off the main xact too */
if (s->blockState == TBLOCK_END)
@@ -2949,7 +3007,7 @@ CommitTransactionCommand(void)
else if (s->blockState == TBLOCK_PREPARE)
{
Assert(s->parent == NULL);
- PrepareTransaction();
+ PrepareTransaction(start_urec_ptr, latest_urec_ptr);
s->blockState = TBLOCK_DEFAULT;
}
else
@@ -3043,6 +3101,18 @@ void
AbortCurrentTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
+
+ /*
+ * The undo actions are allowed to be executed at the end of statement
+ * execution when we are not in transaction block, otherwise they are
+ * executed when user explicitly ends the transaction.
+ *
+ * So if we are in a transaction block don't set the PerformUndoActions
+ * because this flag will be set when user explicitly issue rollback or
+ * rollback to savepoint.
+ */
+ PerformUndoActions = false;
switch (s->blockState)
{
@@ -3077,6 +3147,16 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * We are outside the transaction block so remember the required
+ * information to perform undo actions and also set the
+ * PerformUndoActions so that we execute it before completing this
+ * command.
+ */
+ PerformUndoActions = true;
+ memcpy (UndoActionStartPtr, s->latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
break;
/*
@@ -3113,6 +3193,9 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3132,6 +3215,9 @@ AbortCurrentTransaction(void)
case TBLOCK_ABORT_END:
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /* Failed during commit, so we need to perform the undo actions. */
+ PerformUndoActions = true;
break;
/*
@@ -3142,6 +3228,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Failed while executing the rollback command, need perform any
+ * pending undo actions.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3153,6 +3245,12 @@ AbortCurrentTransaction(void)
AbortTransaction();
CleanupTransaction();
s->blockState = TBLOCK_DEFAULT;
+
+ /*
+ * Perform any pending actions if failed while preparing the
+ * transaction.
+ */
+ PerformUndoActions = true;
break;
/*
@@ -3175,6 +3273,17 @@ AbortCurrentTransaction(void)
case TBLOCK_SUBCOMMIT:
case TBLOCK_SUBABORT_PENDING:
case TBLOCK_SUBRESTART:
+ /*
+ * If we are here and still UndoActionStartPtr is valid that means
+ * the subtransaction failed while executing the undo action, so
+ * store its undo action start point in parent so that parent can
+ * start its undo action from this point.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ s->parent->latest_urec_ptr[i] = UndoActionStartPtr[i];
+ }
AbortSubTransaction();
CleanupSubTransaction();
AbortCurrentTransaction();
@@ -3192,6 +3301,109 @@ AbortCurrentTransaction(void)
}
/*
+ * XactPerformUndoActionsIfPending - Execute pending undo actions.
+ *
+ * If the parent transaction state is valid (when there is an error in the
+ * subtransaction and rollback to savepoint is executed), then allow to
+ * perform undo actions in it, otherwise perform them in a new transaction.
+ */
+void
+XactPerformUndoActionsIfPending()
+{
+ TransactionState s = CurrentTransactionState;
+ uint64 rollback_size = 0;
+ bool new_xact = true, result = false, no_pending_action = true;
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ if (!PerformUndoActions)
+ return;
+
+ /* If there is no undo log for any persistence level, then return. */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ {
+ no_pending_action = false;
+ break;
+ }
+ }
+
+ if (no_pending_action)
+ {
+ PerformUndoActions = false;
+ return;
+ }
+
+ /*
+ * Execute undo actions under parent transaction, if any. Otherwise start
+ * a new transaction.
+ */
+ if (GetTopTransactionIdIfAny() != InvalidTransactionId)
+ {
+ memcpy(parent_latest_urec_ptr, s->latest_urec_ptr,
+ sizeof (parent_latest_urec_ptr));
+ new_xact = false;
+ }
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions later.
+ * Never push the rollbacks for temp tables.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(UndoActionStartPtr[i]))
+ continue;
+
+ if (i == UNDO_TEMP)
+ goto perform_rollback;
+ else
+ rollback_size = UndoActionStartPtr[i] - UndoActionEndPtr[i];
+
+ if (new_xact && rollback_size > rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+perform_rollback:
+ if (new_xact)
+ {
+ TransactionState xact;
+
+ /* Start a new transaction for performing the rollback */
+ StartTransactionCommand();
+ xact = CurrentTransactionState;
+
+ /*
+ * Store the previous transactions start and end undo record
+ * pointers into this transaction's state so that if there is
+ * some error while performing undo actions we can restart
+ * from begining.
+ */
+ memcpy(xact->start_urec_ptr, UndoActionEndPtr,
+ sizeof(UndoActionEndPtr));
+ memcpy(xact->latest_urec_ptr, UndoActionStartPtr,
+ sizeof(UndoActionStartPtr));
+ }
+
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ new_xact, true, true);
+
+ if (new_xact)
+ CommitTransactionCommand();
+ else
+ {
+ /* Restore parent's state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+ }
+
+ PerformUndoActions = false;
+}
+
+/*
* PreventInTransactionBlock
*
* This routine is to be called by statements that must not run inside
@@ -3592,6 +3804,10 @@ EndTransactionBlock(void)
{
TransactionState s = CurrentTransactionState;
bool result = false;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i ;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
switch (s->blockState)
{
@@ -3637,6 +3853,16 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+
+ /*
+ * We are calculating latest_urec_ptr, even though its a commit
+ * case. This is to handle any error during the commit path.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_END;
@@ -3662,6 +3888,11 @@ EndTransactionBlock(void)
elog(FATAL, "EndTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if(!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_ABORT_PENDING;
@@ -3714,6 +3945,18 @@ EndTransactionBlock(void)
break;
}
+ /*
+ * We need to perform undo actions if the transaction is failed. Remember
+ * the required information to perform undo actions at the end of
+ * statement execution.
+ */
+ if (!result)
+ PerformUndoActions = true;
+
+ memcpy(UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy(UndoActionEndPtr, TopTransactionStateData.start_urec_ptr,
+ sizeof(UndoActionEndPtr));
+
return result;
}
@@ -3727,6 +3970,10 @@ void
UserAbortTransactionBlock(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ int i ;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
switch (s->blockState)
{
@@ -3765,6 +4012,12 @@ UserAbortTransactionBlock(void)
elog(FATAL, "UserAbortTransactionBlock: unexpected state %s",
BlockStateAsString(s->blockState));
s = s->parent;
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ }
+
}
if (s->blockState == TBLOCK_INPROGRESS)
s->blockState = TBLOCK_ABORT_PENDING;
@@ -3822,6 +4075,54 @@ UserAbortTransactionBlock(void)
BlockStateAsString(s->blockState));
break;
}
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, s->start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (CurrentTransactionState->state == TRANS_INPROGRESS)
+ {
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (latest_urec_ptr[i])
+ {
+ if (i == UNDO_TEMP)
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ false, true, true);
+ else
+ {
+ uint64 size = latest_urec_ptr[i] - s->start_urec_ptr[i];
+ bool result = false;
+
+ /*
+ * If this is a large rollback request then push it to undo-worker
+ * through RollbackHT, undo-worker will perform it's undo actions
+ * later.
+ */
+ if (size >= rollback_overflow_size * 1024 * 1024)
+ result = PushRollbackReq(UndoActionStartPtr[i], UndoActionEndPtr[i], InvalidOid);
+
+ if (!result)
+ {
+ execute_undo_actions(UndoActionStartPtr[i], UndoActionEndPtr[i],
+ true, true, true);
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
}
/*
@@ -3971,6 +4272,12 @@ ReleaseSavepoint(const char *name)
TransactionState s = CurrentTransactionState;
TransactionState target,
xact;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4064,8 +4371,32 @@ ReleaseSavepoint(const char *name)
if (xact == target)
break;
xact = xact->parent;
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = xact->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(xact->start_urec_ptr[i]))
+ start_urec_ptr[i] = xact->start_urec_ptr[i];
+ }
+
Assert(PointerIsValid(xact));
}
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction. Also set the start_urec_ptr if
+ * parent start_urec_ptr is not valid.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ xact->parent->latest_urec_ptr[i] = latest_urec_ptr[i];
+ if (!UndoRecPtrIsValid(xact->parent->start_urec_ptr[i]))
+ xact->parent->start_urec_ptr[i] = start_urec_ptr[i];
+ }
}
/*
@@ -4080,6 +4411,12 @@ RollbackToSavepoint(const char *name)
TransactionState s = CurrentTransactionState;
TransactionState target,
xact;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels];
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels];
+ int i = 0;
+
+ memcpy(latest_urec_ptr, s->latest_urec_ptr, sizeof(latest_urec_ptr));
+ memcpy(start_urec_ptr, s->start_urec_ptr, sizeof(start_urec_ptr));
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4179,6 +4516,15 @@ RollbackToSavepoint(const char *name)
BlockStateAsString(xact->blockState));
xact = xact->parent;
Assert(PointerIsValid(xact));
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (!UndoRecPtrIsValid(latest_urec_ptr[i]))
+ latest_urec_ptr[i] = xact->latest_urec_ptr[i];
+
+ if (UndoRecPtrIsValid(xact->start_urec_ptr[i]))
+ start_urec_ptr[i] = xact->start_urec_ptr[i];
+ }
+
}
/* And mark the target as "restart pending" */
@@ -4189,6 +4535,34 @@ RollbackToSavepoint(const char *name)
else
elog(FATAL, "RollbackToSavepoint: unexpected state %s",
BlockStateAsString(xact->blockState));
+
+ /*
+ * Remember the required information for performing undo actions. So that
+ * if there is any failure in executing the undo action we can execute
+ * it later.
+ */
+ memcpy (UndoActionStartPtr, latest_urec_ptr, sizeof(UndoActionStartPtr));
+ memcpy (UndoActionEndPtr, start_urec_ptr, sizeof(UndoActionEndPtr));
+
+ /*
+ * If we are in a valid transaction state then execute the undo action here
+ * itself, otherwise we have already stored the required information for
+ * executing the undo action later.
+ */
+ if (s->state == TRANS_INPROGRESS)
+ {
+ for ( i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false, true, false);
+ xact->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ UndoActionStartPtr[i] = InvalidUndoRecPtr;
+ }
+ }
+ }
+ else
+ PerformUndoActions = true;
}
/*
@@ -4276,6 +4650,7 @@ void
ReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
/*
* Workers synchronize transaction state at the beginning of each parallel
@@ -4294,6 +4669,22 @@ ReleaseCurrentSubTransaction(void)
BlockStateAsString(s->blockState));
Assert(s->state == TRANS_INPROGRESS);
MemoryContextSwitchTo(CurTransactionContext);
+
+ /*
+ * Before cleaning up the current sub transaction state, overwrite parent
+ * transaction's latest_urec_ptr with current transaction's latest_urec_ptr
+ * so that in case parent transaction get aborted we will not skip
+ * performing undo for this transaction.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ s->parent->latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ if (!UndoRecPtrIsValid(s->parent->start_urec_ptr[i]))
+ s->parent->start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
Assert(s->state == TRANS_INPROGRESS);
@@ -4310,6 +4701,14 @@ void
RollbackAndReleaseCurrentSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ UndoRecPtr latest_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr start_urec_ptr[UndoPersistenceLevels] = {InvalidUndoRecPtr,
+ InvalidUndoRecPtr,
+ InvalidUndoRecPtr};
+ UndoRecPtr parent_latest_urec_ptr[UndoPersistenceLevels];
+ int i;
/*
* Unlike ReleaseCurrentSubTransaction(), this is nominally permitted
@@ -4356,6 +4755,19 @@ RollbackAndReleaseCurrentSubTransaction(void)
if (s->blockState == TBLOCK_SUBINPROGRESS)
AbortSubTransaction();
+ /*
+ * Remember the required information to perform undo actions before
+ * cleaning up the subtransaction state.
+ */
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(s->latest_urec_ptr[i]))
+ {
+ latest_urec_ptr[i] = s->latest_urec_ptr[i];
+ start_urec_ptr[i] = s->start_urec_ptr[i];
+ }
+ }
+
/* And clean it up, too */
CleanupSubTransaction();
@@ -4364,6 +4776,30 @@ RollbackAndReleaseCurrentSubTransaction(void)
s->blockState == TBLOCK_INPROGRESS ||
s->blockState == TBLOCK_IMPLICIT_INPROGRESS ||
s->blockState == TBLOCK_STARTED);
+
+ for (i = 0; i < UndoPersistenceLevels; i++)
+ {
+ if (UndoRecPtrIsValid(latest_urec_ptr[i]))
+ {
+ parent_latest_urec_ptr[i] = s->latest_urec_ptr[i];
+
+ /*
+ * Store the undo action start point in the parent state so that
+ * we can apply undo actions these undos also during rollback of
+ * parent transaction in case of error while applying the undo
+ * actions.
+ */
+ s->latest_urec_ptr[i] = latest_urec_ptr[i];
+ execute_undo_actions(latest_urec_ptr[i], start_urec_ptr[i], false,
+ true, true);
+
+ /* Restore parent state. */
+ s->latest_urec_ptr[i] = parent_latest_urec_ptr[i];
+ }
+ }
+
+ /* Successfully performed undo actions so reset the flag. */
+ PerformUndoActions = false;
}
/*
@@ -4577,6 +5013,7 @@ static void
StartSubTransaction(void)
{
TransactionState s = CurrentTransactionState;
+ int i;
if (s->state != TRANS_DEFAULT)
elog(WARNING, "StartSubTransaction while in %s state",
@@ -4594,6 +5031,13 @@ StartSubTransaction(void)
AtSubStart_Notify();
AfterTriggerBeginSubXact();
+ /* initialize undo record locations for the transaction */
+ for(i = 0; i < UndoPersistenceLevels; i++)
+ {
+ s->start_urec_ptr[i] = InvalidUndoRecPtr;
+ s->latest_urec_ptr[i] = InvalidUndoRecPtr;
+ }
+
s->state = TRANS_INPROGRESS;
/*
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index c4c5ab4..d124499 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -5169,6 +5169,7 @@ BootStrapXLOG(void)
checkPoint.newestCommitTsXid = InvalidTransactionId;
checkPoint.time = (pg_time_t) time(NULL);
checkPoint.oldestActiveXid = InvalidTransactionId;
+ checkPoint.oldestXidHavingUndo = InvalidTransactionId;
ShmemVariableCache->nextXid = checkPoint.nextXid;
ShmemVariableCache->nextOid = checkPoint.nextOid;
@@ -6610,6 +6611,10 @@ StartupXLOG(void)
(errmsg_internal("commit timestamp Xid oldest/newest: %u/%u",
checkPoint.oldestCommitTsXid,
checkPoint.newestCommitTsXid)));
+ ereport(DEBUG1,
+ (errmsg_internal("oldest xid having undo: %u",
+ checkPoint.oldestXidHavingUndo)));
+
if (!TransactionIdIsNormal(checkPoint.nextXid))
ereport(PANIC,
(errmsg("invalid next transaction ID")));
@@ -6627,6 +6632,10 @@ StartupXLOG(void)
XLogCtl->ckptXidEpoch = checkPoint.nextXidEpoch;
XLogCtl->ckptXid = checkPoint.nextXid;
+ /* Read oldest xid having undo from checkpoint and set in proc global. */
+ pg_atomic_write_u32(&ProcGlobal->oldestXidHavingUndo,
+ checkPoint.oldestXidHavingUndo);
+
/*
* Initialize replication slots, before there's a chance to remove
* required resources.
@@ -8768,6 +8777,9 @@ CreateCheckPoint(int flags)
checkPoint.nextOid += ShmemVariableCache->oidCount;
LWLockRelease(OidGenLock);
+ checkPoint.oldestXidHavingUndo =
+ pg_atomic_read_u32(&ProcGlobal->oldestXidHavingUndo);
+
MultiXactGetCheckptMulti(shutdown,
&checkPoint.nextMulti,
&checkPoint.nextMultiOffset,
@@ -9678,6 +9690,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceOldest(checkPoint.oldestMulti,
checkPoint.oldestMultiDB);
+ pg_atomic_write_u32(&ProcGlobal->oldestXidHavingUndo,
+ checkPoint.oldestXidHavingUndo);
+
/*
* No need to set oldestClogXid here as well; it'll be set when we
* redo an xl_clog_truncate if it changed since initialization.
@@ -9736,6 +9751,8 @@ xlog_redo(XLogReaderState *record)
/* ControlFile->checkPointCopy always tracks the latest ckpt XID */
ControlFile->checkPointCopy.nextXidEpoch = checkPoint.nextXidEpoch;
ControlFile->checkPointCopy.nextXid = checkPoint.nextXid;
+ ControlFile->checkPointCopy.oldestXidHavingUndo =
+ checkPoint.oldestXidHavingUndo;
/* Update shared-memory copy of checkpoint XID/epoch */
SpinLockAcquire(&XLogCtl->info_lck);
@@ -9785,6 +9802,9 @@ xlog_redo(XLogReaderState *record)
MultiXactAdvanceNextMXact(checkPoint.nextMulti,
checkPoint.nextMultiOffset);
+ pg_atomic_write_u32(&ProcGlobal->oldestXidHavingUndo,
+ checkPoint.oldestXidHavingUndo);
+
/*
* NB: This may perform multixact truncation when replaying WAL
* generated by an older primary.
diff --git a/src/backend/access/undo/Makefile b/src/backend/access/undo/Makefile
index f41e8f7..fdf7f7d 100644
--- a/src/backend/access/undo/Makefile
+++ b/src/backend/access/undo/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/access/undo
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = undoinsert.o undolog.o undorecord.o
+OBJS = undoaction.o undoactionxlog.o undodiscard.o undoinsert.o undolog.o undorecord.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/undo/undoaction.c b/src/backend/access/undo/undoaction.c
new file mode 100644
index 0000000..14aaa25
--- /dev/null
+++ b/src/backend/access/undo/undoaction.c
@@ -0,0 +1,719 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.c
+ * execute undo actions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undoaction.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/undoaction.h"
+#include "access/undoaction_xlog.h"
+#include "access/undolog.h"
+#include "access/undorecord.h"
+#include "access/visibilitymap.h"
+#include "access/xact.h"
+#include "access/xloginsert.h"
+#include "access/xlog_internal.h"
+#include "nodes/pg_list.h"
+#include "pgstat.h"
+#include "postmaster/undoloop.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "utils/rel.h"
+#include "utils/relfilenodemap.h"
+#include "miscadmin.h"
+#include "storage/shmem.h"
+#include "access/undodiscard.h"
+
+#define ROLLBACK_HT_SIZE 1024
+
+static bool execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr,
+ Oid reloid, TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool norellock, int options);
+static void RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr);
+
+/* This is the hash table to store all the rollabck requests. */
+static HTAB *RollbackHT;
+
+/*
+ * execute_undo_actions - Execute the undo actions
+ *
+ * from_urecptr - undo record pointer from where to start applying undo action.
+ * to_urecptr - undo record pointer upto which point apply undo action.
+ * nopartial - true if rollback is for complete transaction.
+ * rewind - whether to rewind the insert location of the undo log or not.
+ * Only the backend executed the transaction can rewind, but
+ * any other process e.g. undo worker should not rewind it.
+ * Because, if the backend have already inserted new undo records
+ * for the next transaction and if we rewind then we will loose
+ * the undo record inserted for the new transaction.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ */
+void
+execute_undo_actions(UndoRecPtr from_urecptr, UndoRecPtr to_urecptr,
+ bool nopartial, bool rewind, bool rellock)
+{
+ UnpackedUndoRecord *uur = NULL;
+ UndoRecPtr urec_ptr, prev_urec_ptr;
+ UndoRecPtr save_urec_ptr;
+ Oid prev_reloid = InvalidOid;
+ ForkNumber prev_fork = InvalidForkNumber;
+ BlockNumber prev_block = InvalidBlockNumber;
+ List *luinfo = NIL;
+ bool more_undo;
+ int options = 0;
+ TransactionId xid = InvalidTransactionId;
+ UndoRecInfo *urec_info;
+
+ Assert(from_urecptr != InvalidUndoRecPtr);
+ Assert(UndoRecPtrGetLogNo(from_urecptr) != UndoRecPtrGetLogNo(to_urecptr) ||
+ from_urecptr >= to_urecptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ * FIXME: this won't work if undolog crossed the limit of 1TB, because
+ * then from_urecptr and to_urecptr will be from different lognos.
+ */
+ if (to_urecptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(from_urecptr);
+ to_urecptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ save_urec_ptr = urec_ptr = from_urecptr;
+
+ if (nopartial)
+ {
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber, InvalidOffsetNumber,
+ InvalidTransactionId, NULL, NULL);
+ if (uur == NULL)
+ return;
+
+ xid = uur->uur_xid;
+ UndoRecordRelease(uur);
+ uur = NULL;
+
+ /*
+ * Grab the undo action apply lock before start applying the undo action
+ * this will prevent applying undo actions concurrently. If we do not
+ * get the lock that mean its already being applied concurrently or the
+ * discard worker might be pushing its request to the rollback hash
+ * table
+ */
+ if (!ConditionTransactionUndoActionLock(xid))
+ return;
+ }
+
+ prev_urec_ptr = InvalidUndoRecPtr;
+ while (prev_urec_ptr != to_urecptr)
+ {
+ Oid reloid = InvalidOid;
+ uint16 urec_prevlen;
+ UndoRecPtr urec_prevurp;
+ bool non_page;
+
+ more_undo = true;
+
+ prev_urec_ptr = urec_ptr;
+
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(urec_ptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId, NULL, NULL);
+
+ /* If there is no info block, this is not a page-based undo record. */
+ non_page = uur && !(uur->uur_info & UREC_INFO_BLOCK);
+
+ if (uur != NULL && !non_page)
+ reloid = uur->uur_reloid;
+
+ xid = uur->uur_xid;
+
+ if (non_page)
+ {
+ prev_reloid = InvalidOid;
+ urec_prevlen = uur->uur_prevlen;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /*
+ * Execute individual undo actions not associated with a page
+ * immediately.
+ */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->uur = uur;
+ urec_info->urp = urec_ptr;
+ luinfo = lappend(luinfo, urec_info);
+ execute_undo_actions_page(luinfo, urec_ptr, reloid, xid,
+ InvalidBlockNumber, false, rellock,
+ 0);
+ pfree(urec_info);
+ urec_info = NULL;
+ list_free(luinfo);
+ luinfo = NIL;
+ UndoRecordRelease(uur);
+
+ /* Follow undo chain until to_urecptr. */
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ /*
+ * If the record is already discarded by undo worker or if the relation
+ * is dropped or truncated, then we cannot fetch record successfully.
+ * Hence, skip quietly.
+ *
+ * Note: reloid remains InvalidOid for a discarded record.
+ */
+ else if (!OidIsValid(reloid))
+ {
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ urec_prevlen = uur->uur_prevlen;
+
+ /* Release the just-fetched record */
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else if (!OidIsValid(prev_reloid) ||
+ (prev_reloid == reloid &&
+ prev_fork == uur->uur_fork &&
+ prev_block == uur->uur_block))
+ {
+ /* Collect the undo records that belong to the same page. */
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+
+ luinfo = lappend(luinfo, urec_info);
+ urec_prevlen = uur->uur_prevlen;
+ urec_prevurp = uur->uur_prevurp;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /* The undo chain must continue till we reach to_urecptr */
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ {
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ continue;
+ }
+ else
+ more_undo = false;
+ }
+ else
+ {
+ more_undo = true;
+ }
+
+ /*
+ * If no more undo is left to be processed and we are rolling back the
+ * complete transaction, then we can consider that the undo chain for a
+ * block is complete.
+ * If the previous undo pointer in the page is invalid, then also the
+ * undo chain for the current block is completed.
+ */
+ if (luinfo &&
+ ((!more_undo && nopartial) || !UndoRecPtrIsValid(save_urec_ptr)))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, true, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+ else if (luinfo)
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, false, rellock, options);
+ /* Done with the page so reset the options. */
+ options = 0;
+ }
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+
+ /*
+ * There are still more records to process, so keep moving backwards
+ * in the chain.
+ */
+ if (more_undo)
+ {
+ /* Prepare an undo record information element. */
+ urec_info = palloc(sizeof(UndoRecInfo));
+ urec_info->urp = urec_ptr;
+ urec_info->uur = uur;
+ luinfo = lappend(luinfo, urec_info);
+
+ prev_reloid = reloid;
+ prev_fork = uur->uur_fork;
+ prev_block = uur->uur_block;
+ save_urec_ptr = uur->uur_blkprev;
+
+ /*
+ * Continue to process the records if this is not the last undo
+ * record in chain.
+ */
+ urec_prevlen = uur->uur_prevlen;
+ urec_prevurp = uur->uur_prevurp;
+ if (urec_ptr != to_urecptr &&
+ (urec_prevlen > 0 || UndoRecPtrIsValid(urec_prevurp)))
+ urec_ptr = UndoGetPrevUndoRecptr(urec_ptr, urec_prevlen,
+ urec_prevurp);
+ else
+ break;
+ }
+ else
+ break;
+ }
+
+ /* Apply the undo actions for the remaining records. */
+ if (list_length(luinfo))
+ {
+ execute_undo_actions_page(luinfo, save_urec_ptr, prev_reloid,
+ xid, prev_block, nopartial ? true : false,
+ rellock, options);
+
+ /* release the undo records for which action has been replayed */
+ while (luinfo)
+ {
+ UndoRecInfo *urec_info = (UndoRecInfo *) linitial(luinfo);
+
+ UndoRecordRelease(urec_info->uur);
+ pfree(urec_info);
+ luinfo = list_delete_first(luinfo);
+ }
+ }
+
+ if (rewind)
+ {
+ /* Read the prevlen from the first record of this transaction. */
+ uur = UndoFetchRecord(to_urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If undo is already discarded before we rewind, then do nothing.
+ */
+ if (uur == NULL)
+ return;
+
+
+ /*
+ * Rewind the insert location to start of this transaction. This is
+ * to avoid reapplying some intermediate undo. We do not need to wal
+ * log this information here, because if the system crash before we
+ * rewind the insert pointer then after recovery we can identify
+ * whether the undo is already applied or not from the slot undo record
+ * pointer. Also set the correct prevlen value (what we have fetched
+ * from the undo).
+ */
+ UndoLogRewind(to_urecptr, uur->uur_prevlen);
+
+ UndoRecordRelease(uur);
+ }
+
+ if (nopartial)
+ {
+ /*
+ * Set undo action apply completed in the transaction header if this is
+ * a main transaction and we have not rewound its undo.
+ */
+ if (!rewind)
+ {
+ /*
+ * Undo action is applied so delete the hash table entry and release
+ * the undo action lock.
+ */
+ RollbackHTRemoveEntry(from_urecptr);
+
+ /*
+ * Prepare and update the progress of the undo action apply in the
+ * transaction header.
+ */
+ PrepareUpdateUndoActionProgress(to_urecptr, 1);
+
+ START_CRIT_SECTION();
+
+ /* Update the progress in the transaction header. */
+ UndoRecordUpdateTransInfo(0);
+
+ /* WAL log the undo apply progress. */
+ {
+ xl_undoapply_progress xlrec;
+
+ xlrec.urec_ptr = to_urecptr;
+ xlrec.progress = 1;
+
+ /*
+ * FIXME : We need to register undo buffers and set LSN for them
+ * that will be required for FPW of the undo buffers.
+ */
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, sizeof(xlrec));
+
+ (void) XLogInsert(RM_UNDOACTION_ID, XLOG_UNDO_APPLY_PROGRESS);
+ }
+
+ END_CRIT_SECTION();
+ UnlockReleaseUndoBuffers();
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+}
+
+/*
+ * execute_undo_actions_page - Execute the undo actions for a page
+ *
+ * After applying all the undo actions for a page, we clear the transaction
+ * slot on a page if the undo chain for block is complete, otherwise just
+ * rewind the undo pointer to the last record for that block that precedes
+ * the last undo record for which action is replayed.
+ *
+ * luinfo - list of undo records (along with their location) for which undo
+ * action needs to be replayed.
+ * urec_ptr - undo record pointer to which we need to rewind.
+ * reloid - OID of relation on which undo actions needs to be applied.
+ * blkno - block number on which undo actions needs to be applied.
+ * blk_chain_complete - indicates whether the undo chain for block is
+ * complete.
+ * nopartial - true if rollback is for complete transaction. If we are not
+ * rolling back the complete transaction then we need to apply the
+ * undo action for UNDO_INVALID_XACT_SLOT also because in such
+ * case we will rewind the insert undo location.
+ * rellock - if the caller already has the lock on the required relation,
+ * then this flag is false, i.e. we do not need to acquire any
+ * lock here. If the flag is true then we need to acquire lock
+ * here itself, because caller will not be having any lock.
+ * When we are performing undo actions for prepared transactions,
+ * or for rollback to savepoint, we need not to lock as we already
+ * have the lock on the table. In cases like error or when
+ * rollbacking from the undo worker we need to have proper locks.
+ * options - options for executing undo actions.
+ *
+ * returns true, if successfully applied the undo actions, otherwise, false.
+ */
+static bool
+execute_undo_actions_page(List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock, int options)
+{
+ UndoRecInfo *first;
+
+ /*
+ * All records passed to us are for the same RMGR, so we just use the
+ * first record to dispatch.
+ */
+ Assert(luinfo != NIL);
+ first = (UndoRecInfo *) linitial(luinfo);
+
+ return RmgrTable[first->uur->uur_rmid].rm_undo(luinfo, urec_ptr, reloid,
+ xid, blkno,
+ blk_chain_complete, rellock,
+ options);
+}
+
+/*
+ * To return the size of the hash-table for rollbacks.
+ */
+int
+RollbackHTSize(void)
+{
+ return hash_estimate_size(ROLLBACK_HT_SIZE, sizeof(RollbackHashEntry));
+}
+
+/*
+ * To initialize the hash-table for rollbacks in shared memory
+ * for the given size.
+ */
+void
+InitRollbackHashTable(void)
+{
+ HASHCTL info;
+ MemSet(&info, 0, sizeof(info));
+
+ info.keysize = sizeof(UndoRecPtr);
+ info.entrysize = sizeof(RollbackHashEntry);
+ info.hash = tag_hash;
+
+ RollbackHT = ShmemInitHash("Undo actions Lookup Table",
+ ROLLBACK_HT_SIZE, ROLLBACK_HT_SIZE, &info,
+ HASH_ELEM | HASH_FUNCTION | HASH_FIXED_SIZE);
+}
+
+/*
+ * To push the rollback requests from backend to the hash-table.
+ * Return true if the request is successfully added, else false
+ * and the caller may execute undo actions itself.
+ */
+bool
+PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr, Oid dbid)
+{
+ bool found = false;
+ RollbackHashEntry *rh;
+
+ Assert(UndoRecPtrGetLogNo(start_urec_ptr) != UndoRecPtrGetLogNo(end_urec_ptr) ||
+ start_urec_ptr >= end_urec_ptr);
+ /*
+ * If the location upto which rollback need to be done is not provided,
+ * then rollback the complete transaction.
+ */
+ if (start_urec_ptr == InvalidUndoRecPtr)
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(end_urec_ptr);
+ start_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ Assert(UndoRecPtrIsValid(start_urec_ptr));
+
+ /* If there is no space to accomodate new request, then we can't proceed. */
+ if (RollbackHTIsFull())
+ return false;
+
+ if(!UndoRecPtrIsValid(end_urec_ptr))
+ {
+ UndoLogNumber logno = UndoRecPtrGetLogNo(start_urec_ptr);
+ end_urec_ptr = UndoLogGetLastXactStartPoint(logno);
+ }
+
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ rh = (RollbackHashEntry *) hash_search(RollbackHT, &start_urec_ptr,
+ HASH_ENTER_NULL, &found);
+ if (!rh)
+ {
+ LWLockRelease(RollbackHTLock);
+ return false;
+ }
+ /* We shouldn't try to push the same rollback request again. */
+ if (!found)
+ {
+ rh->start_urec_ptr = start_urec_ptr;
+ rh->end_urec_ptr = end_urec_ptr;
+ rh->dbid = (dbid == InvalidOid) ? MyDatabaseId : dbid;
+ }
+ LWLockRelease(RollbackHTLock);
+
+ return true;
+}
+
+/*
+ * To perform the undo actions for the transactions whose rollback
+ * requests are in hash table. Sequentially, scan the hash-table
+ * and perform the undo-actions for the respective transactions.
+ * Once, the undo-actions are applied, remove the entry from the
+ * hash table.
+ */
+void
+RollbackFromHT(Oid dbid)
+{
+ UndoRecPtr start[ROLLBACK_HT_SIZE];
+ UndoRecPtr end[ROLLBACK_HT_SIZE];
+ RollbackHashEntry *rh;
+ HASH_SEQ_STATUS status;
+ int i = 0;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ Assert(hash_get_num_entries(RollbackHT) <= ROLLBACK_HT_SIZE);
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ {
+ if (rh->dbid == dbid)
+ {
+ start[i] = rh->start_urec_ptr;
+ end[i] = rh->end_urec_ptr;
+ i++;
+ }
+ }
+
+ LWLockRelease(RollbackHTLock);
+
+ /* Execute the rollback requests */
+ while(--i >= 0)
+ {
+ Assert(UndoRecPtrIsValid(start[i]));
+ Assert(UndoRecPtrIsValid(end[i]));
+
+ StartTransactionCommand();
+ execute_undo_actions(start[i], end[i], true, false, true);
+ CommitTransactionCommand();
+ }
+}
+
+/*
+ * Remove the rollback request entry from the rollback hash table.
+ */
+static void
+RollbackHTRemoveEntry(UndoRecPtr start_urec_ptr)
+{
+ LWLockAcquire(RollbackHTLock, LW_EXCLUSIVE);
+
+ hash_search(RollbackHT, &start_urec_ptr, HASH_REMOVE, NULL);
+
+ LWLockRelease(RollbackHTLock);
+}
+
+/*
+ * To check if the rollback requests in the hash table are all
+ * completed or not. This is required because we don't not want to
+ * expose RollbackHT in xact.c, where it is required to ensure
+ * that we push the resuests only when there is some space in
+ * the hash-table.
+ */
+bool
+RollbackHTIsFull(void)
+{
+ bool result = false;
+
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ if (hash_get_num_entries(RollbackHT) >= ROLLBACK_HT_SIZE)
+ result = true;
+
+ LWLockRelease(RollbackHTLock);
+
+ return result;
+}
+
+/*
+ * Get database list from the rollback hash table.
+ */
+List *
+RollbackHTGetDBList()
+{
+ HASH_SEQ_STATUS status;
+ RollbackHashEntry *rh;
+ List *dblist = NIL;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ dblist = list_append_unique_oid(dblist, rh->dbid);
+
+ LWLockRelease(RollbackHTLock);
+
+ return dblist;
+}
+
+/*
+ * Remove all the entries for the given dbid. This is required in cases when
+ * the database is dropped and there were rollback requests pushed to the
+ * hash-table.
+ */
+void
+RollbackHTCleanup(Oid dbid)
+{
+ RollbackHashEntry *rh;
+ HASH_SEQ_STATUS status;
+ UndoRecPtr start_urec_ptr;
+
+ /* Fetch the rollback requests */
+ LWLockAcquire(RollbackHTLock, LW_SHARED);
+
+ Assert(hash_get_num_entries(RollbackHT) <= ROLLBACK_HT_SIZE);
+ hash_seq_init(&status, RollbackHT);
+ while (RollbackHT != NULL &&
+ (rh = (RollbackHashEntry *) hash_seq_search(&status)) != NULL)
+ {
+ if (rh->dbid == dbid)
+ {
+ start_urec_ptr = rh->start_urec_ptr;
+ hash_search(RollbackHT, &start_urec_ptr, HASH_REMOVE, NULL);
+ }
+ }
+
+ LWLockRelease(RollbackHTLock);
+
+}
+
+/*
+ * ConditionTransactionUndoActionLock
+ *
+ * Insert a lock showing that the undo action for given transaction is in
+ * progress. This is only done for the main transaction not for the
+ * sub-transaction.
+ */
+bool
+ConditionTransactionUndoActionLock(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ if (LOCKACQUIRE_NOT_AVAIL == LockAcquire(&tag, ExclusiveLock, false, true))
+ return false;
+ else
+ return true;
+}
+
+/*
+ * TransactionUndoActionLockRelease
+ *
+ * Delete the lock showing that the undo action given transaction ID is in
+ * progress.
+ */
+void
+TransactionUndoActionLockRelease(TransactionId xid)
+{
+ LOCKTAG tag;
+
+ SET_LOCKTAG_TRANSACTION_UNDOACTION(tag, xid);
+
+ LockRelease(&tag, ExclusiveLock, false);
+}
diff --git a/src/backend/access/undo/undoactionxlog.c b/src/backend/access/undo/undoactionxlog.c
new file mode 100644
index 0000000..253d8f1
--- /dev/null
+++ b/src/backend/access/undo/undoactionxlog.c
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoactionxlog.c
+ * WAL replay logic for undo actions.
+ *
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/undo/undoactionxlog.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/undoaction_xlog.h"
+#include "access/undoinsert.h"
+#include "access/visibilitymap.h"
+#include "access/xlog.h"
+#include "access/xlogutils.h"
+
+/*
+ * Replay of undo apply progress.
+ */
+static void
+undo_xlog_apply_progress(XLogReaderState *record)
+{
+ xl_undoapply_progress *xlrec = (xl_undoapply_progress *) XLogRecGetData(record);
+
+ /* Update the progress in the transaction header. */
+ PrepareUpdateUndoActionProgress(xlrec->urec_ptr, xlrec->progress);
+ UndoRecordUpdateTransInfo(0);
+}
+
+void
+undoaction_redo(XLogReaderState *record)
+{
+ uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+
+ switch (info)
+ {
+ case XLOG_UNDO_APPLY_PROGRESS:
+ undo_xlog_apply_progress(record);
+ break;
+ default:
+ elog(PANIC, "undoaction_redo: unknown op code %u", info);
+ }
+}
diff --git a/src/backend/access/undo/undodiscard.c b/src/backend/access/undo/undodiscard.c
new file mode 100644
index 0000000..afe5834
--- /dev/null
+++ b/src/backend/access/undo/undodiscard.c
@@ -0,0 +1,447 @@
+/*-------------------------------------------------------------------------
+ *
+ * undodiscard.c
+ * discard undo records
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/undo/undodiscard.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/transam.h"
+#include "access/xlog.h"
+#include "access/xact.h"
+#include "access/undolog.h"
+#include "access/undodiscard.h"
+#include "catalog/pg_tablespace.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/buf.h"
+#include "storage/bufmgr.h"
+#include "storage/shmem.h"
+#include "storage/proc.h"
+#include "utils/resowner.h"
+#include "postmaster/undoloop.h"
+
+static UndoRecPtr FetchLatestUndoPtrForXid(UndoRecPtr urecptr,
+ UnpackedUndoRecord *uur_start,
+ UndoLogControl *log);
+
+/*
+ * Discard the undo for the log
+ *
+ * Search the undo log, get the start record for each transaction until we get
+ * the transaction with xid >= xmin or an invalid xid. Then call undolog
+ * routine to discard upto that point and update the memory structure for the
+ * log slot. We set the hibernate flag if we do not have any undo logs, this
+ * flag is passed to the undo worker wherein it determines if system is idle
+ * and it should sleep for sometime.
+ *
+ * Return the oldest xid remaining in this undo log (which should be >= xmin,
+ * since we'll discard everything older). Return InvalidTransactionId if the
+ * undo log is empty.
+ */
+static TransactionId
+UndoDiscardOneLog(UndoLogControl *log, TransactionId xmin, bool *hibernate)
+{
+ UndoRecPtr undo_recptr, next_insert, from_urecptr;
+ UndoRecPtr next_urecptr = InvalidUndoRecPtr;
+ UnpackedUndoRecord *uur = NULL;
+ bool need_discard = false;
+ bool log_complete = false;
+ TransactionId undoxid = InvalidTransactionId;
+ TransactionId xid = log->oldest_xid;
+ TransactionId latest_discardxid = InvalidTransactionId;
+
+ undo_recptr = log->oldest_data;
+
+ /* There might not be any undo log and hibernation might be needed. */
+ *hibernate = true;
+
+ /* Loop until we run out of discardable transactions. */
+ do
+ {
+ bool pending_abort = false;
+
+ next_insert = UndoLogGetNextInsertPtr(log->logno, xid);
+
+ /*
+ * If the next insert location in the undo log is same as the oldest
+ * data for the log then there is nothing more to discard in this log
+ * so discard upto this point.
+ */
+ if (next_insert == undo_recptr)
+ {
+ /*
+ * If the discard location and the insert location is same then
+ * there is nothing to discard.
+ */
+ if (undo_recptr == log->oldest_data)
+ break;
+ else
+ log_complete = true;
+ }
+ else
+ {
+ /* Fetch the undo record for given undo_recptr. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+
+ Assert(uur != NULL);
+
+ if (!TransactionIdDidCommit(uur->uur_xid) &&
+ TransactionIdPrecedes(uur->uur_xid, xmin) &&
+ uur->uur_progress == 0)
+ {
+ /*
+ * At the time of recovery, we might not have a valid next undo
+ * record pointer and in that case we'll calculate the location
+ * of from pointer using the last record of next insert
+ * location.
+ */
+ if (ConditionTransactionUndoActionLock(uur->uur_xid))
+ {
+ TransactionId xid = uur->uur_xid;
+ UndoLogControl *log = NULL;
+ UndoLogNumber logno;
+
+ logno = UndoRecPtrGetLogNo(undo_recptr);
+ log = UndoLogGet(logno, false);
+
+ /*
+ * If the corresponding log got rewinded to a location
+ * prior to undo_recptr, the undo actions are already
+ * applied.
+ */
+ if (log->meta.insert > undo_recptr)
+ {
+ UndoRecordRelease(uur);
+
+ /* Fetch the undo record under undo action lock. */
+ uur = UndoFetchRecord(undo_recptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the undo actions for the aborted transaction is
+ * already applied then continue discarding the undo log
+ * otherwise discard till current point and stop processing
+ * this undo log.
+ * Also, check this is indeed the transaction id we're
+ * looking for. It is possible that after rewinding
+ * some other transaction has inserted an undo record.
+ */
+ if (uur->uur_xid == xid && uur->uur_progress == 0)
+ {
+ from_urecptr = FetchLatestUndoPtrForXid(undo_recptr, uur, log);
+ (void)PushRollbackReq(from_urecptr, undo_recptr, uur->uur_dbid);
+ pending_abort = true;
+ }
+ }
+
+ TransactionUndoActionLockRelease(xid);
+ }
+ else
+ pending_abort = true;
+ }
+
+ next_urecptr = uur->uur_next;
+ undoxid = uur->uur_xid;
+ xid = undoxid;
+ }
+
+ /* we can discard upto this point. */
+ if (TransactionIdFollowsOrEquals(undoxid, xmin) ||
+ next_urecptr == SpecialUndoRecPtr ||
+ UndoRecPtrGetLogNo(next_urecptr) != log->logno ||
+ log_complete || pending_abort)
+ {
+ /* Hey, I got some undo log to discard, can not hibernate now. */
+ *hibernate = false;
+
+ if (uur != NULL)
+ UndoRecordRelease(uur);
+
+ /*
+ * If Transaction id is smaller than the xmin that means this must
+ * be the last transaction in this undo log, so we need to get the
+ * last insert point in this undo log and discard till that point.
+ * Also, if the transaction has pending abort, we stop discarding
+ * undo from the same location.
+ */
+ if (TransactionIdPrecedes(undoxid, xmin) && !pending_abort)
+ {
+ UndoRecPtr next_insert = InvalidUndoRecPtr;
+
+ /*
+ * Get the last insert location for this transaction Id, if it
+ * returns invalid pointer that means there is new transaction
+ * has started for this undolog. So we need to refetch the undo
+ * and continue the process.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, undoxid);
+ if (!UndoRecPtrIsValid(next_insert))
+ continue;
+
+ undo_recptr = next_insert;
+ need_discard = true;
+ latest_discardxid = undoxid;
+ undoxid = InvalidTransactionId;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_EXCLUSIVE);
+
+ /*
+ * If no more pending undo logs then set the oldest transaction to
+ * InvalidTransactionId.
+ */
+ if (log_complete)
+ log->oldest_xid = InvalidTransactionId;
+ else
+ log->oldest_xid = undoxid;
+
+ log->oldest_data = undo_recptr;
+ LWLockRelease(&log->discard_lock);
+
+ if (need_discard)
+ UndoLogDiscard(undo_recptr, latest_discardxid);
+
+ break;
+ }
+
+ /*
+ * This transaction is smaller than the xmin so lets jump to the next
+ * transaction.
+ */
+ undo_recptr = next_urecptr;
+ latest_discardxid = undoxid;
+
+ if(uur != NULL)
+ {
+ UndoRecordRelease(uur);
+ uur = NULL;
+ }
+
+ need_discard = true;
+ } while (true);
+
+ return undoxid;
+}
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+void
+UndoDiscard(TransactionId oldestXmin, bool *hibernate)
+{
+ TransactionId oldestXidHavingUndo = oldestXmin;
+ UndoLogControl *log = NULL;
+
+ /*
+ * TODO: Ideally we'd arrange undo logs so that we can efficiently find
+ * those with oldest_xid < oldestXmin, but for now we'll just scan all of
+ * them.
+ */
+ while ((log = UndoLogNext(log)))
+ {
+ TransactionId oldest_xid = InvalidTransactionId;
+
+ /* We can't process temporary undo logs. */
+ if (log->meta.persistence == UNDO_TEMP)
+ continue;
+
+ /*
+ * If the first xid of the undo log is smaller than the xmin the try
+ * to discard the undo log.
+ */
+ if (TransactionIdPrecedes(log->oldest_xid, oldestXmin))
+ {
+ /*
+ * If the XID in the discard entry is invalid then start scanning
+ * from the first valid undorecord in the log.
+ */
+ if (!TransactionIdIsValid(log->oldest_xid))
+ {
+ bool full = false;
+ UndoRecPtr urp = UndoLogGetFirstValidRecord(log, &full);
+
+ if (!UndoRecPtrIsValid(urp))
+ {
+ /*
+ * There is nothing to be discarded. If there is also no
+ * more free space, then a call to UndoLogDiscard() will
+ * discard it the undo log completely and free up the
+ * UndoLogControl slot.
+ */
+ if (full)
+ UndoLogDiscard(MakeUndoRecPtr(log->meta.logno,
+ log->meta.discard),
+ InvalidTransactionId);
+ continue;
+ }
+
+ LWLockAcquire(&log->discard_lock, LW_SHARED);
+ log->oldest_data = urp;
+ LWLockRelease(&log->discard_lock);
+ }
+
+ /* Process the undo log. */
+ oldest_xid = UndoDiscardOneLog(log, oldestXmin, hibernate);
+ }
+
+ if (TransactionIdIsValid(oldest_xid) &&
+ TransactionIdPrecedes(oldest_xid, oldestXidHavingUndo))
+ oldestXidHavingUndo = oldest_xid;
+ }
+
+ /*
+ * Update the oldestXidHavingUndo in the shared memory.
+ *
+ * XXX In future if multiple worker can perform discard then we may need
+ * to use compare and swap for updating the shared memory value.
+ */
+ pg_atomic_write_u32(&ProcGlobal->oldestXidHavingUndo, oldestXidHavingUndo);
+}
+
+/*
+ * Fetch the latest urec pointer for the transaction.
+ */
+UndoRecPtr
+FetchLatestUndoPtrForXid(UndoRecPtr urecptr, UnpackedUndoRecord *uur_start,
+ UndoLogControl *log)
+{
+ UndoRecPtr next_urecptr, from_urecptr;
+ uint16 prevlen;
+ UndoLogOffset next_insert;
+ UnpackedUndoRecord *uur;
+ bool refetch = false;
+
+ uur = uur_start;
+
+ while (true)
+ {
+ /* fetch the undo record again if required. */
+ if (refetch)
+ {
+ uur = UndoFetchRecord(urecptr, InvalidBlockNumber,
+ InvalidOffsetNumber, InvalidTransactionId,
+ NULL, NULL);
+ refetch = false;
+ }
+
+ next_urecptr = uur->uur_next;
+ prevlen = UndoLogGetPrevLen(log->logno);
+
+ /*
+ * If this is the last transaction in the log then calculate the latest
+ * urec pointer using next insert location of the undo log. Otherwise,
+ * calculate using next transaction's start pointer.
+ */
+ if (uur->uur_next == SpecialUndoRecPtr)
+ {
+ /*
+ * While fetching the next insert location if the new transaction
+ * has already started in this log then lets re-fetch the undo
+ * record.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+ if (!UndoRecPtrIsValid(next_insert))
+ {
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ refetch = true;
+ continue;
+ }
+
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen,
+ InvalidUndoRecPtr);
+ break;
+ }
+ else if ((UndoRecPtrGetLogNo(next_urecptr) != log->logno) &&
+ UndoLogIsDiscarded(next_urecptr))
+ {
+ /*
+ * If next_urecptr is in different undolog and its already discarded
+ * that means the undo actions for this transaction which are in the
+ * next log has already been executed and we only need to execute
+ * which are remaining in this log.
+ */
+ next_insert = UndoLogGetNextInsertPtr(log->logno, uur->uur_xid);
+
+ Assert(UndoRecPtrIsValid(next_insert));
+ from_urecptr = UndoGetPrevUndoRecptr(next_insert, prevlen,
+ InvalidUndoRecPtr);
+ break;
+ }
+ else
+ {
+ UnpackedUndoRecord *next_uur;
+
+ next_uur = UndoFetchRecord(next_urecptr,
+ InvalidBlockNumber,
+ InvalidOffsetNumber,
+ InvalidTransactionId,
+ NULL, NULL);
+ /*
+ * If the next_urecptr is in the same log then calculate the
+ * from pointer using prevlen.
+ */
+ if (UndoRecPtrGetLogNo(next_urecptr) == log->logno)
+ {
+ from_urecptr =
+ UndoGetPrevUndoRecptr(next_urecptr, next_uur->uur_prevlen,
+ InvalidUndoRecPtr);
+ UndoRecordRelease(next_uur);
+ break;
+ }
+ else
+ {
+ /*
+ * The transaction is overflowed to the next log, so restart
+ * the processing from then next log.
+ */
+ log = UndoLogGet(UndoRecPtrGetLogNo(next_urecptr), false);
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+ uur = next_uur;
+ continue;
+ }
+
+ UndoRecordRelease(next_uur);
+ }
+ }
+
+ if (uur != uur_start)
+ UndoRecordRelease(uur);
+
+ return from_urecptr;
+}
+
+/*
+ * Discard the undo logs for temp tables.
+ */
+void
+TempUndoDiscard(UndoLogNumber logno)
+{
+ UndoLogControl *log = UndoLogGet(logno, false);
+
+ /*
+ * Discard the undo log for temp table only. Ensure that there is
+ * something to be discarded there.
+ */
+ Assert (log->meta.persistence == UNDO_TEMP);
+
+ /* Process the undo log. */
+ UndoLogDiscard(MakeUndoRecPtr(log->logno, log->meta.insert),
+ InvalidTransactionId);
+}
diff --git a/src/backend/access/undo/undoinsert.c b/src/backend/access/undo/undoinsert.c
index 9ffa46c..4e0fe13 100644
--- a/src/backend/access/undo/undoinsert.c
+++ b/src/backend/access/undo/undoinsert.c
@@ -171,7 +171,6 @@ static UnpackedUndoRecord *UndoGetOneRecord(UnpackedUndoRecord *urec,
UndoPersistence persistence);
static void UndoRecordPrepareTransInfo(UndoRecPtr urecptr,
UndoRecPtr xact_urp);
-static void UndoRecordUpdateTransInfo(int idx);
static int UndoGetBufferSlot(RelFileNode rnode, BlockNumber blk,
ReadBufferMode rbm,
UndoPersistence persistence);
@@ -302,12 +301,61 @@ UndoRecordPrepareTransInfo(UndoRecPtr urecptr, UndoRecPtr xact_urp)
}
/*
+ * Update the progress of the undo record in the transaction header.
+ */
+void
+PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress)
+{
+ Buffer buffer = InvalidBuffer;
+ BlockNumber cur_blk;
+ RelFileNode rnode;
+ UndoLogNumber logno = UndoRecPtrGetLogNo(urecptr);
+ UndoLogControl *log;
+ Page page;
+ int already_decoded = 0;
+ int starting_byte;
+ int bufidx;
+ int index = 0;
+
+ log = UndoLogGet(logno, false);
+
+ if (log->meta.persistence == UNDO_TEMP)
+ return;
+
+ UndoRecPtrAssignRelFileNode(rnode, urecptr);
+ cur_blk = UndoRecPtrGetBlockNum(urecptr);
+ starting_byte = UndoRecPtrGetPageOffset(urecptr);
+
+ while (true)
+ {
+ bufidx = UndoGetBufferSlot(rnode, cur_blk,
+ RBM_NORMAL,
+ log->meta.persistence);
+
+ xact_urec_info[xact_urec_info_idx].idx_undo_buffers[index++] = bufidx;
+ buffer = undo_buffer[bufidx].buf;
+ page = BufferGetPage(buffer);
+
+ if (UnpackUndoRecord(&xact_urec_info[xact_urec_info_idx].uur, page, starting_byte,
+ &already_decoded, true))
+ break;
+
+ starting_byte = UndoLogBlockHeaderSize;
+ cur_blk++;
+ }
+
+ xact_urec_info[xact_urec_info_idx].urecptr = urecptr;
+ xact_urec_info[xact_urec_info_idx].uur.uur_progress = progress;
+ xact_urec_info_idx++;
+}
+
+/*
* Overwrite the first undo record of the previous transaction to update its
* next pointer. This will just insert the already prepared record by
* UndoRecordPrepareTransInfo. This must be called under the critical section.
* This will just overwrite the undo header not the data.
*/
-static void
+void
UndoRecordUpdateTransInfo(int idx)
{
UndoLogNumber logno = UndoRecPtrGetLogNo(xact_urec_info[idx].urecptr);
diff --git a/src/backend/access/undo/undorecord.c b/src/backend/access/undo/undorecord.c
index 23ed374..9af8e45 100644
--- a/src/backend/access/undo/undorecord.c
+++ b/src/backend/access/undo/undorecord.c
@@ -98,6 +98,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
*/
if (*already_written == 0)
{
+ work_hdr.urec_rmid = uur->uur_rmid;
work_hdr.urec_type = uur->uur_type;
work_hdr.urec_info = uur->uur_info;
work_hdr.urec_prevlen = uur->uur_prevlen;
@@ -122,6 +123,7 @@ InsertUndoRecord(UnpackedUndoRecord *uur, Page page,
* We should have been passed the same record descriptor as before, or
* caller has messed up.
*/
+ Assert(work_hdr.urec_rmid == uur->uur_rmid);
Assert(work_hdr.urec_type == uur->uur_type);
Assert(work_hdr.urec_info == uur->uur_info);
Assert(work_hdr.urec_prevlen == uur->uur_prevlen);
@@ -285,6 +287,7 @@ UnpackUndoRecord(UnpackedUndoRecord *uur, Page page, int starting_byte,
&my_bytes_decoded, already_decoded, false))
return false;
+ uur->uur_rmid = work_hdr.urec_rmid;
uur->uur_type = work_hdr.urec_type;
uur->uur_info = work_hdr.urec_info;
uur->uur_prevlen = work_hdr.urec_prevlen;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index ff1e178..e7a0bbe 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1381,6 +1381,7 @@ vac_truncate_clog(TransactionId frozenXID,
MultiXactId lastSaneMinMulti)
{
TransactionId nextXID = ReadNewTransactionId();
+ TransactionId oldestXidHavingUndo;
Relation relation;
HeapScanDesc scan;
HeapTuple tuple;
@@ -1475,6 +1476,15 @@ vac_truncate_clog(TransactionId frozenXID,
return;
/*
+ * We can't truncate the clog for transactions that still have undo. The
+ * oldestXidHavingUndo will be only valid for zheap storage engine, so it
+ * won't impact any other storage engine.
+ */
+ oldestXidHavingUndo = pg_atomic_read_u32(&ProcGlobal->oldestXidHavingUndo);
+ if (TransactionIdIsValid(oldestXidHavingUndo))
+ frozenXID = Min(frozenXID, oldestXidHavingUndo);
+
+ /*
* Advance the oldest value for commit timestamps before truncating, so
* that if a user requests a timestamp for a transaction we're truncating
* away right after this point, they get NULL instead of an ugly "file not
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 71c2321..9ce6ff0 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -12,7 +12,7 @@ subdir = src/backend/postmaster
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
- pgarch.o pgstat.o postmaster.o startup.o syslogger.o walwriter.o
+OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o discardworker.o fork_process.o \
+ pgarch.o pgstat.o postmaster.o startup.o syslogger.o undoworker.o walwriter.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index f5db5a8..772b865 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -20,7 +20,9 @@
#include "pgstat.h"
#include "port/atomics.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "storage/dsm.h"
@@ -129,6 +131,15 @@ static const struct
},
{
"ApplyWorkerMain", ApplyWorkerMain
+ },
+ {
+ "UndoLauncherMain", UndoLauncherMain
+ },
+ {
+ "UndoWorkerMain", UndoWorkerMain
+ },
+ {
+ "DiscardWorkerMain", DiscardWorkerMain
}
};
diff --git a/src/backend/postmaster/discardworker.c b/src/backend/postmaster/discardworker.c
new file mode 100644
index 0000000..7cbe6c4
--- /dev/null
+++ b/src/backend/postmaster/discardworker.c
@@ -0,0 +1,170 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.c
+ * The undo discard worker for asynchronous undo management.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/discardworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include <unistd.h>
+
+/* These are always necessary for a bgworker. */
+#include "access/transam.h"
+#include "miscadmin.h"
+#include "postmaster/bgworker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/lwlock.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+#include "access/undodiscard.h"
+#include "pgstat.h"
+#include "postmaster/discardworker.h"
+#include "storage/procarray.h"
+#include "tcop/tcopprot.h"
+#include "utils/guc.h"
+#include "utils/resowner.h"
+
+static void undoworker_sigterm_handler(SIGNAL_ARGS);
+
+/* max sleep time between cycles (100 milliseconds) */
+#define MIN_NAPTIME_PER_CYCLE 100L
+#define DELAYED_NAPTIME 10 * MIN_NAPTIME_PER_CYCLE
+#define MAX_NAPTIME_PER_CYCLE 100 * MIN_NAPTIME_PER_CYCLE
+
+static bool got_SIGTERM = false;
+static bool hibernate = false;
+static long wait_time = MIN_NAPTIME_PER_CYCLE;
+
+/* SIGTERM: set flag to exit at next convenient time */
+static void
+undoworker_sigterm_handler(SIGNAL_ARGS)
+{
+ got_SIGTERM = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+}
+
+/*
+ * DiscardWorkerRegister -- Register a undo discard worker.
+ */
+void
+DiscardWorkerRegister(void)
+{
+ BackgroundWorker bgw;
+
+ /* TODO: This should be configurable. */
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "discard worker");
+ sprintf(bgw.bgw_library_name, "postgres");
+ sprintf(bgw.bgw_function_name, "DiscardWorkerMain");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * DiscardWorkerMain -- Main loop for the undo discard worker.
+ */
+void
+DiscardWorkerMain(Datum main_arg)
+{
+ ereport(LOG,
+ (errmsg("discard worker started")));
+
+ /* Establish signal handlers. */
+ pqsignal(SIGTERM, undoworker_sigterm_handler);
+ BackgroundWorkerUnblockSignals();
+
+ /* Make it easy to identify our processes. */
+ SetConfigOption("application_name", MyBgworkerEntry->bgw_name,
+ PGC_USERSET, PGC_S_SESSION);
+
+ /*
+ * Create resource owner for discard worker as it need to read the undo
+ * records outside the transaction blocks which intern access buffer read
+ * routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ /* Enter main loop */
+ while (!got_SIGTERM)
+ {
+ int rc;
+ TransactionId OldestXmin, oldestXidHavingUndo;
+
+ OldestXmin = GetOldestXmin(NULL, PROCARRAY_FLAGS_DEFAULT);
+
+ oldestXidHavingUndo =
+ pg_atomic_read_u32(&ProcGlobal->oldestXidHavingUndo);
+
+ /*
+ * Call the discard routine if there oldestXidHavingUndo is lagging
+ * behind OldestXmin.
+ */
+ if (OldestXmin != InvalidTransactionId &&
+ TransactionIdPrecedes(oldestXidHavingUndo, OldestXmin))
+ {
+ UndoDiscard(OldestXmin, &hibernate);
+
+ /*
+ * If we got some undo logs to discard or discarded something,
+ * then reset the wait_time as we have got work to do.
+ * Note that if there are some undologs that cannot be discarded,
+ * then above condition will remain unsatisified till oldestXmin
+ * remains unchanged and the wait_time will not reset in that case.
+ */
+ if (!hibernate)
+ wait_time = MIN_NAPTIME_PER_CYCLE;
+ }
+
+ /* Wait for more work. */
+ rc = WaitLatch(&MyProc->procLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wait_time,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN);
+
+ ResetLatch(&MyProc->procLatch);
+
+ /*
+ * Increase the wait_time based on the length of inactivity. If wait_time
+ * is within one second, then increment it by 100 ms at a time. Henceforth,
+ * increment it one second at a time, till it reaches ten seconds. Never
+ * increase the wait_time more than ten seconds, it will be too much of
+ * waiting otherwise.
+ */
+ if (rc & WL_TIMEOUT && hibernate)
+ {
+ wait_time += (wait_time < DELAYED_NAPTIME ?
+ MIN_NAPTIME_PER_CYCLE : DELAYED_NAPTIME);
+ if (wait_time > MAX_NAPTIME_PER_CYCLE)
+ wait_time = MAX_NAPTIME_PER_CYCLE;
+ }
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+
+ ReleaseAuxProcessResources(true);
+
+ /* we're done */
+ ereport(LOG,
+ (errmsg("discard worker shutting down")));
+
+ proc_exit(0);
+}
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 03591bf..6cae438 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -3506,6 +3506,12 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_SYSLOGGER_MAIN:
event_name = "SysLoggerMain";
break;
+ case WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN:
+ event_name = "UndoDiscardWorkerMain";
+ break;
+ case WAIT_EVENT_UNDO_LAUNCHER_MAIN:
+ event_name = "UndoLauncherMain";
+ break;
case WAIT_EVENT_WAL_RECEIVER_MAIN:
event_name = "WalReceiverMain";
break;
@@ -3918,7 +3924,6 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_UNDO_FILE_SYNC:
event_name = "UndoFileSync";
break;
-
case WAIT_EVENT_WALSENDER_TIMELINE_HISTORY_READ:
event_name = "WALSenderTimelineHistoryRead";
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a707d4d..c4acd72 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -111,10 +111,12 @@
#include "port/pg_bswap.h"
#include "postmaster/autovacuum.h"
#include "postmaster/bgworker_internals.h"
+#include "postmaster/discardworker.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/walsender.h"
#include "storage/fd.h"
@@ -981,6 +983,11 @@ PostmasterMain(int argc, char *argv[])
*/
ApplyLauncherRegister();
+ UndoLauncherRegister();
+
+ /* Register the Undo Discard worker. */
+ DiscardWorkerRegister();
+
/*
* process any libraries that should be preloaded at postmaster start
*/
diff --git a/src/backend/postmaster/undoworker.c b/src/backend/postmaster/undoworker.c
new file mode 100644
index 0000000..23d051a
--- /dev/null
+++ b/src/backend/postmaster/undoworker.c
@@ -0,0 +1,644 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.c
+ * undo launcher and undo worker process.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/undoworker.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "funcapi.h"
+#include "miscadmin.h"
+#include "pgstat.h"
+
+#include "access/heapam.h"
+#include "access/htup.h"
+#include "access/htup_details.h"
+#include "access/sysattr.h"
+#include "access/xact.h"
+
+#include "catalog/indexing.h"
+#include "catalog/pg_database.h"
+
+#include "libpq/pqsignal.h"
+
+#include "postmaster/bgworker.h"
+#include "postmaster/fork_process.h"
+#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
+
+#include "replication/slot.h"
+#include "replication/worker_internal.h"
+
+#include "storage/ipc.h"
+#include "storage/lmgr.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+
+#include "tcop/tcopprot.h"
+
+#include "utils/fmgroids.h"
+#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/resowner.h"
+
+/* max sleep time between cycles (100 milliseconds) */
+#define DEFAULT_NAPTIME_PER_CYCLE 100L
+#define DEFAULT_RETRY_NAPTIME 50L
+
+int max_undo_workers = 5;
+
+typedef struct UndoApplyWorker
+{
+ /* Indicates if this slot is used or free. */
+ bool in_use;
+
+ /* Increased everytime the slot is taken by new worker. */
+ uint16 generation;
+
+ /* Pointer to proc array. NULL if not running. */
+ PGPROC *proc;
+
+ /* Database id to connect to. */
+ Oid dbid;
+} UndoApplyWorker;
+
+UndoApplyWorker *MyUndoWorker = NULL;
+
+typedef struct UndoApplyCtxStruct
+{
+ /* Supervisor process. */
+ pid_t launcher_pid;
+
+ /* Background workers. */
+ UndoApplyWorker workers[FLEXIBLE_ARRAY_MEMBER];
+} UndoApplyCtxStruct;
+
+UndoApplyCtxStruct *UndoApplyCtx;
+
+static void undo_worker_onexit(int code, Datum arg);
+static void undo_worker_cleanup(UndoApplyWorker *worker);
+
+static volatile sig_atomic_t got_SIGHUP = false;
+
+/*
+ * Wait for a background worker to start up and attach to the shmem context.
+ *
+ * This is only needed for cleaning up the shared memory in case the worker
+ * fails to attach.
+ */
+static void
+WaitForUndoWorkerAttach(UndoApplyWorker *worker,
+ uint16 generation,
+ BackgroundWorkerHandle *handle)
+{
+ BgwHandleStatus status;
+ int rc;
+
+ for (;;)
+ {
+ pid_t pid;
+
+ CHECK_FOR_INTERRUPTS();
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+
+ /* Worker either died or has started; no need to do anything. */
+ if (!worker->in_use || worker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ LWLockRelease(UndoWorkerLock);
+
+ /* Check if worker has died before attaching, and clean up after it. */
+ status = GetBackgroundWorkerPid(handle, &pid);
+
+ if (status == BGWH_STOPPED)
+ {
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ /* Ensure that this was indeed the worker we waited for. */
+ if (generation == worker->generation)
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+ return;
+ }
+
+ /*
+ * We need timeout because we generally don't get notified via latch
+ * about the worker attach. But we don't expect to have to wait long.
+ */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 10L, WAIT_EVENT_BGWORKER_STARTUP);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+ }
+
+ return;
+}
+
+/*
+ * Get dbid from the worker slot.
+ */
+static Oid
+slot_get_dbid(int slot)
+{
+ Oid dbid;
+
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty,",
+ slot)));
+ }
+
+ dbid = MyUndoWorker->dbid;
+
+ LWLockRelease(UndoWorkerLock);
+
+ return dbid;
+}
+
+/*
+ * Attach to a slot.
+ */
+static void
+undo_worker_attach(int slot)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ MyUndoWorker = &UndoApplyCtx->workers[slot];
+
+ if (!MyUndoWorker->in_use)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is empty, cannot attach",
+ slot)));
+ }
+
+ if (MyUndoWorker->proc)
+ {
+ LWLockRelease(UndoWorkerLock);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("undo worker slot %d is already used by "
+ "another worker, cannot attach", slot)));
+ }
+
+ MyUndoWorker->proc = MyProc;
+ before_shmem_exit(undo_worker_onexit, (Datum) 0);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Walks the workers array and searches for one that matches given
+ * dbid.
+ */
+static UndoApplyWorker *
+undo_worker_find(Oid dbid)
+{
+ int i;
+ UndoApplyWorker *res = NULL;
+
+ Assert(LWLockHeldByMe(UndoWorkerLock));
+
+ /* Search for attached worker for a given db id. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (w->in_use && w->dbid == dbid)
+ {
+ res = w;
+ break;
+ }
+ }
+
+ return res;
+}
+
+/*
+ * Check whether the dbid exist or not.
+ *
+ * Refer comments from GetDatabaseTupleByOid.
+ * FIXME: Should we expose GetDatabaseTupleByOid and directly use it.
+ */
+static bool
+dbid_exist(Oid dboid)
+{
+ HeapTuple tuple;
+ Relation relation;
+ SysScanDesc scan;
+ ScanKeyData key[1];
+ bool result = false;
+
+ /*
+ * form a scan key
+ */
+ ScanKeyInit(&key[0],
+ Anum_pg_database_oid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dboid));
+
+ relation = heap_open(DatabaseRelationId, AccessShareLock);
+ scan = systable_beginscan(relation, DatabaseOidIndexId,
+ criticalSharedRelcachesBuilt,
+ NULL,
+ 1, key);
+
+ tuple = systable_getnext(scan);
+
+ if (HeapTupleIsValid(tuple))
+ result = true;
+
+ /* all done */
+ systable_endscan(scan);
+ heap_close(relation, AccessShareLock);
+
+ return result;
+}
+
+/*
+ * Start new undo apply background worker, if possible otherwise return false.
+ */
+static bool
+undo_worker_launch(Oid dbid)
+{
+ BackgroundWorker bgw;
+ BackgroundWorkerHandle *bgw_handle;
+ uint16 generation;
+ int i;
+ int slot = 0;
+ UndoApplyWorker *worker = NULL;
+
+ /*
+ * We need to do the modification of the shared memory under lock so that
+ * we have consistent view.
+ */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ /* Find unused worker slot. */
+ for (i = 0; i < max_undo_workers; i++)
+ {
+ UndoApplyWorker *w = &UndoApplyCtx->workers[i];
+
+ if (!w->in_use)
+ {
+ worker = w;
+ slot = i;
+ break;
+ }
+ }
+
+ /* There are no more free worker slots */
+ if (worker == NULL)
+ return false;
+
+ /* Prepare the worker slot. */
+ worker->in_use = true;
+ worker->proc = NULL;
+ worker->dbid = dbid;
+ worker->generation++;
+
+ generation = worker->generation;
+ LWLockRelease(UndoWorkerLock);
+
+ /* Register the new dynamic worker. */
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoWorkerMain");
+ snprintf(bgw.bgw_type, BGW_MAXLEN, "undo apply worker");
+ snprintf(bgw.bgw_name, BGW_MAXLEN, "undo apply worker");
+
+ bgw.bgw_restart_time = BGW_NEVER_RESTART;
+ bgw.bgw_notify_pid = MyProcPid;
+ bgw.bgw_main_arg = Int32GetDatum(slot);
+
+ /* Check the database exists or not. */
+
+ StartTransactionCommand();
+ if (!dbid_exist(dbid))
+ {
+ CommitTransactionCommand();
+
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ RollbackHTCleanup(dbid);
+ LWLockRelease(UndoWorkerLock);
+ return true;
+ }
+
+ if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle))
+ {
+ /* Failed to start worker, so clean up the worker slot. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+ undo_worker_cleanup(worker);
+ LWLockRelease(UndoWorkerLock);
+
+ CommitTransactionCommand();
+
+ return false;
+ }
+
+ /* Now wait until it attaches. */
+ WaitForUndoWorkerAttach(worker, generation, bgw_handle);
+
+ CommitTransactionCommand();
+
+ return true;
+}
+
+/*
+ * Detach the worker (cleans up the worker info).
+ */
+static void
+undo_worker_detach(void)
+{
+ /* Block concurrent access. */
+ LWLockAcquire(UndoWorkerLock, LW_EXCLUSIVE);
+
+ undo_worker_cleanup(MyUndoWorker);
+
+ LWLockRelease(UndoWorkerLock);
+}
+
+/*
+ * Clean up worker info.
+ */
+static void
+undo_worker_cleanup(UndoApplyWorker *worker)
+{
+ Assert(LWLockHeldByMeInMode(UndoWorkerLock, LW_EXCLUSIVE));
+
+ worker->in_use = false;
+ worker->proc = NULL;
+ worker->dbid = InvalidOid;
+}
+
+/*
+ * Cleanup function for undo worker launcher.
+ *
+ * Called on undo worker launcher exit.
+ */
+static void
+undo_launcher_onexit(int code, Datum arg)
+{
+ UndoApplyCtx->launcher_pid = 0;
+}
+
+/* SIGHUP: set flag to reload configuration at next convenient time */
+static void
+undo_launcher_sighup(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGHUP = true;
+
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+undo_worker_onexit(int code, Datum arg)
+{
+ undo_worker_detach();
+}
+
+/*
+ * UndoLauncherShmemSize
+ * Compute space needed for undo launcher shared memory
+ */
+Size
+UndoLauncherShmemSize(void)
+{
+ Size size;
+
+ /*
+ * Need the fixed struct and the array of LogicalRepWorker.
+ */
+ size = sizeof(UndoApplyCtxStruct);
+ size = MAXALIGN(size);
+ size = add_size(size, mul_size(max_undo_workers,
+ sizeof(UndoApplyWorker)));
+ return size;
+}
+
+/*
+ * UndoLauncherRegister
+ * Register a background worker running the undo worker launcher.
+ */
+void
+UndoLauncherRegister(void)
+{
+ BackgroundWorker bgw;
+
+ if (max_undo_workers == 0)
+ return;
+
+ memset(&bgw, 0, sizeof(bgw));
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "UndoLauncherMain");
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "undo worker launcher");
+ snprintf(bgw.bgw_type, BGW_MAXLEN,
+ "undo worker launcher");
+ bgw.bgw_restart_time = 5;
+ bgw.bgw_notify_pid = 0;
+ bgw.bgw_main_arg = (Datum) 0;
+
+ RegisterBackgroundWorker(&bgw);
+}
+
+/*
+ * UndoLauncherShmemInit
+ * Allocate and initialize undo worker launcher shared memory
+ */
+void
+UndoLauncherShmemInit(void)
+{
+ bool found;
+
+ UndoApplyCtx = (UndoApplyCtxStruct *)
+ ShmemInitStruct("Undo Worker Launcher Data",
+ UndoLauncherShmemSize(),
+ &found);
+
+ if (!found)
+ memset(UndoApplyCtx, 0, UndoLauncherShmemSize());
+}
+
+/*
+ * Main loop for the undo worker launcher process.
+ */
+void
+UndoLauncherMain(Datum main_arg)
+{
+ MemoryContext tmpctx;
+ MemoryContext oldctx;
+
+ ereport(DEBUG1,
+ (errmsg("undo launcher started")));
+
+ before_shmem_exit(undo_launcher_onexit, (Datum) 0);
+
+ Assert(UndoApplyCtx->launcher_pid == 0);
+ UndoApplyCtx->launcher_pid = MyProcPid;
+
+ /* Establish signal handlers. */
+ pqsignal(SIGHUP, undo_launcher_sighup);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * Establish connection to nailed catalogs (we only ever access
+ * pg_subscription).
+ */
+ BackgroundWorkerInitializeConnection(NULL, NULL, 0);
+
+ /* Use temporary context for the database list and worker info. */
+ tmpctx = AllocSetContextCreate(TopMemoryContext,
+ "Undo worker Launcher context",
+ ALLOCSET_DEFAULT_SIZES);
+ /* Enter main loop */
+ for (;;)
+ {
+ int rc;
+ List *dblist;
+ ListCell *l;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* switch to the temp context. */
+ oldctx = MemoryContextSwitchTo(tmpctx);
+ dblist = RollbackHTGetDBList();
+
+ foreach(l, dblist)
+ {
+ UndoApplyWorker *w;
+ Oid dbid = lfirst_oid(l);
+
+ LWLockAcquire(UndoWorkerLock, LW_SHARED);
+ w = undo_worker_find(dbid);
+ LWLockRelease(UndoWorkerLock);
+
+ if (w == NULL)
+ {
+retry:
+ if (!undo_worker_launch(dbid))
+ {
+ /* Could not launch the worker, retry after sometime, */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_RETRY_NAPTIME,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+ goto retry;
+ }
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+
+ /* Clean the temporary memory. */
+ MemoryContextReset(tmpctx);
+
+ /* Wait for more work. */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ DEFAULT_NAPTIME_PER_CYCLE,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ if (got_SIGHUP)
+ {
+ got_SIGHUP = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+}
+
+/*
+ * UndoWorkerMain -- Main loop for the undo apply worker.
+ */
+void
+UndoWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ Oid dbid;
+
+ dbid = slot_get_dbid(worker_slot);
+
+ /* Setup signal handling */
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /* Attach to slot */
+ undo_worker_attach(worker_slot);
+
+ /* Connect to the database. */
+ BackgroundWorkerInitializeConnectionByOid(dbid, 0, 0);
+
+ /*
+ * Create resource owner for undo worker. Undo worker need this as it
+ * need to read the undo records outside the transaction blocks which
+ * intern access buffer read routine.
+ */
+ CreateAuxProcessResourceOwner();
+
+ RollbackFromHT(dbid);
+
+ ReleaseAuxProcessResources(true);
+
+ proc_exit(0);
+}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index a8aa11a..52e094c 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -159,6 +159,10 @@ LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *recor
ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(record),
buf.origptr);
break;
+ case RM_UNDOACTION_ID:
+ /* Logical decoding is not yet implemented for undoactions. */
+ Assert(0);
+ break;
case RM_NEXT_ID:
elog(ERROR, "unexpected RM_NEXT_ID rmgr_id: %u", (RmgrIds) XLogRecGetRmid(buf.record));
}
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index faedafb..ab043c1 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -29,6 +29,8 @@
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
+#include "postmaster/undoloop.h"
+#include "postmaster/undoworker.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
#include "replication/walreceiver.h"
@@ -150,6 +152,8 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, RollbackHTSize());
+ size = add_size(size, UndoLauncherShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -224,6 +228,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
SUBTRANSShmemInit();
MultiXactShmemInit();
InitBufferPool();
+ InitRollbackHashTable();
/*
* Set up lock manager
@@ -262,6 +267,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
WalSndShmemInit();
WalRcvShmemInit();
ApplyLauncherShmemInit();
+ UndoLauncherShmemInit();
/*
* Set up other modules that need some shared memory space
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index bd07ed6..a47ecd4 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -50,3 +50,5 @@ OldSnapshotTimeMapLock 42
LogicalRepWorkerLock 43
CLogTruncationLock 44
UndoLogLock 45
+RollbackHTLock 46
+UndoWorkerLock 47
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 89c80fb..bd2bc98 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -286,6 +286,8 @@ InitProcGlobal(void)
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemAlloc(sizeof(slock_t));
SpinLockInit(ProcStructLock);
+
+ pg_atomic_init_u32(&ProcGlobal->oldestXidHavingUndo, 0);
}
/*
diff --git a/src/backend/utils/adt/lockfuncs.c b/src/backend/utils/adt/lockfuncs.c
index ffd1970..bdb6606 100644
--- a/src/backend/utils/adt/lockfuncs.c
+++ b/src/backend/utils/adt/lockfuncs.c
@@ -32,6 +32,7 @@ const char *const LockTagTypeNames[] = {
"virtualxid",
"speculative token",
"object",
+ "undoaction",
"userlock",
"advisory"
};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index fd51934..08489a1 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -121,6 +121,7 @@ bool allowSystemTableMods = false;
int work_mem = 1024;
int maintenance_work_mem = 16384;
int max_parallel_maintenance_workers = 2;
+int rollback_overflow_size = 64;
/*
* Primary determinants of sizes of shared-memory structures.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index c5698b5..7a8878d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2862,6 +2862,17 @@ static struct config_int ConfigureNamesInt[] =
},
{
+ {"rollback_overflow_size", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Rollbacks greater than this size are done lazily"),
+ NULL,
+ GUC_UNIT_MB
+ },
+ &rollback_overflow_size,
+ 64, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
{"wal_segment_size", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the size of write ahead log segments."),
NULL,
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index e6742dc..96908ea 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -78,8 +78,8 @@ pg_control_system(PG_FUNCTION_ARGS)
Datum
pg_control_checkpoint(PG_FUNCTION_ARGS)
{
- Datum values[19];
- bool nulls[19];
+ Datum values[20];
+ bool nulls[20];
TupleDesc tupdesc;
HeapTuple htup;
ControlFileData *ControlFile;
@@ -128,6 +128,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
XIDOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 18, "checkpoint_time",
TIMESTAMPTZOID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 19, "oldest_xid_with_epoch_having_undo",
+ TIMESTAMPTZOID, -1, 0);
tupdesc = BlessTupleDesc(tupdesc);
/* Read the control file. */
@@ -202,6 +204,9 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
time_t_to_timestamptz(ControlFile->checkPointCopy.time));
nulls[17] = false;
+ values[18] = Int32GetDatum(ControlFile->checkPointCopy.oldestXidHavingUndo);
+ nulls[18] = false;
+
htup = heap_form_tuple(tupdesc, values, nulls);
PG_RETURN_DATUM(HeapTupleGetDatum(htup));
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1fa02d2..82b9cf6 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -737,4 +737,11 @@
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
+# If often there are large transactions requiring rollbacks, then we can push
+# them to undo-workers for better performance. The size specifeid by the
+# parameter below, determines the minimum size of the rollback requests to be
+# sent to the undo-worker.
+#
+#rollback_overflow_size = 64
+
# Add settings for extensions here
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 895a51f..dd4f12a 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -278,6 +278,8 @@ main(int argc, char *argv[])
ControlFile->checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile->checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidHavingUndo:%u\n"),
+ ControlFile->checkPointCopy.oldestXidHavingUndo);
printf(_("Time of latest checkpoint: %s\n"),
ckpttime_str);
printf(_("Fake LSN counter for unlogged rels: %X/%X\n"),
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index e3213d4..e30375a 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -448,6 +448,7 @@ main(int argc, char *argv[])
if (ControlFile.checkPointCopy.oldestXid < FirstNormalTransactionId)
ControlFile.checkPointCopy.oldestXid += FirstNormalTransactionId;
ControlFile.checkPointCopy.oldestXidDB = InvalidOid;
+ ControlFile.checkPointCopy.oldestXidHavingUndo = 0;
}
if (set_oldest_commit_ts_xid != 0)
@@ -716,6 +717,8 @@ GuessControlValues(void)
ControlFile.checkPointCopy.oldestMultiDB = InvalidOid;
ControlFile.checkPointCopy.time = (pg_time_t) time(NULL);
ControlFile.checkPointCopy.oldestActiveXid = InvalidTransactionId;
+ ControlFile.checkPointCopy.nextXid = 0;
+ ControlFile.checkPointCopy.oldestXidHavingUndo = 0;
ControlFile.state = DB_SHUTDOWNED;
ControlFile.time = (pg_time_t) time(NULL);
@@ -808,6 +811,8 @@ PrintControlValues(bool guessed)
ControlFile.checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile.checkPointCopy.newestCommitTsXid);
+ printf(_("Latest checkpoint's oldestXidHavingUndo:%u\n"),
+ ControlFile.checkPointCopy.oldestXidHavingUndo);
printf(_("Maximum data alignment: %u\n"),
ControlFile.maxAlign);
/* we don't print floatFormat since can't say much useful about it */
@@ -884,6 +889,8 @@ PrintNewControlValues(void)
ControlFile.checkPointCopy.oldestXid);
printf(_("OldestXID's DB: %u\n"),
ControlFile.checkPointCopy.oldestXidDB);
+ printf(_("OldestXidHavingUndo: %u\n"),
+ ControlFile.checkPointCopy.oldestXidHavingUndo);
}
if (set_xid_epoch != -1)
diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c
index e19c265..ef0b1a0 100644
--- a/src/bin/pg_rewind/parsexlog.c
+++ b/src/bin/pg_rewind/parsexlog.c
@@ -29,7 +29,7 @@
* RmgrNames is an array of resource manager names, to make error messages
* a bit nicer.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
name,
static const char *RmgrNames[RM_MAX_ID + 1] = {
diff --git a/src/bin/pg_waldump/rmgrdesc.c b/src/bin/pg_waldump/rmgrdesc.c
index 938150d..3ef0a60 100644
--- a/src/bin/pg_waldump/rmgrdesc.c
+++ b/src/bin/pg_waldump/rmgrdesc.c
@@ -20,6 +20,7 @@
#include "access/nbtxlog.h"
#include "access/rmgr.h"
#include "access/spgxlog.h"
+#include "access/undoaction_xlog.h"
#include "access/undolog_xlog.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -33,7 +34,7 @@
#include "storage/standbydefs.h"
#include "utils/relmapper.h"
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undodesc) \
{ name, desc, identify},
const RmgrDescData RmgrDescTable[RM_MAX_ID + 1] = {
diff --git a/src/include/access/rmgr.h b/src/include/access/rmgr.h
index c9b5c56..e1fb42a 100644
--- a/src/include/access/rmgr.h
+++ b/src/include/access/rmgr.h
@@ -19,7 +19,7 @@ typedef uint8 RmgrId;
* Note: RM_MAX_ID must fit in RmgrId; widening that type will affect the XLOG
* file format.
*/
-#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask) \
+#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,undo,undo_desc) \
symname,
typedef enum RmgrIds
diff --git a/src/include/access/rmgrlist.h b/src/include/access/rmgrlist.h
index 6945e3e..ef0f6ac 100644
--- a/src/include/access/rmgrlist.h
+++ b/src/include/access/rmgrlist.h
@@ -25,26 +25,27 @@
*/
/* symbol name, textual name, redo, desc, identify, startup, cleanup */
-PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL)
-PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL)
-PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL)
-PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL)
-PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL)
-PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL)
-PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL)
-PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask)
-PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask)
-PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask)
-PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask)
-PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask)
-PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask)
-PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask)
-PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask)
-PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL)
-PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL)
-PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask)
-PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL)
-PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL)
+PG_RMGR(RM_XLOG_ID, "XLOG", xlog_redo, xlog_desc, xlog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_XACT_ID, "Transaction", xact_redo, xact_desc, xact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_SMGR_ID, "Storage", smgr_redo, smgr_desc, smgr_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_CLOG_ID, "CLOG", clog_redo, clog_desc, clog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_DBASE_ID, "Database", dbase_redo, dbase_desc, dbase_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_TBLSPC_ID, "Tablespace", tblspc_redo, tblspc_desc, tblspc_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_MULTIXACT_ID, "MultiXact", multixact_redo, multixact_desc, multixact_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_RELMAP_ID, "RelMap", relmap_redo, relmap_desc, relmap_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_STANDBY_ID, "Standby", standby_redo, standby_desc, standby_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_HEAP2_ID, "Heap2", heap2_redo, heap2_desc, heap2_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask, NULL, NULL)
+PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, NULL, NULL, btree_mask, NULL, NULL)
+PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask, NULL, NULL)
+PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask, NULL, NULL)
+PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask, NULL, NULL)
+PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask, NULL, NULL)
+PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask, NULL, NULL)
+PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask, NULL, NULL)
+PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask, NULL, NULL)
+PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOLOG_ID, "UndoLog", undolog_redo, undolog_desc, undolog_identify, NULL, NULL, NULL, NULL, NULL)
+PG_RMGR(RM_UNDOACTION_ID, "UndoAction", undoaction_redo, undoaction_desc, undoaction_identify, NULL, NULL, NULL, NULL, NULL)
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 4a7eab0..92d5d51 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -68,6 +68,10 @@
(AssertMacro(TransactionIdIsNormal(id1) && TransactionIdIsNormal(id2)), \
(int32) ((id1) - (id2)) > 0)
+/* Extract xid from a value comprised of epoch and xid */
+#define GetXidFromEpochXid(epochxid) \
+ ((uint32) (epochxid) & 0XFFFFFFFF)
+
/* ----------
* Object ID (OID) zero is InvalidOid.
*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 6228b09..0b1bcf4 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -41,7 +41,7 @@ extern GlobalTransaction MarkAsPreparing(TransactionId xid, const char *gid,
TimestampTz prepared_at,
Oid owner, Oid databaseid);
-extern void StartPrepare(GlobalTransaction gxact);
+extern void StartPrepare(GlobalTransaction gxact, UndoRecPtr *, UndoRecPtr *);
extern void EndPrepare(GlobalTransaction gxact);
extern bool StandbyTransactionIdIsPrepared(TransactionId xid);
diff --git a/src/include/access/undoaction.h b/src/include/access/undoaction.h
new file mode 100644
index 0000000..5455259
--- /dev/null
+++ b/src/include/access/undoaction.h
@@ -0,0 +1,28 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction.h
+ * undo action prototypes
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_H
+#define UNDOACTION_H
+
+#include "postgres.h"
+
+#include "access/undolog.h"
+#include "access/undorecord.h"
+
+/* undo record information */
+typedef struct UndoRecInfo
+{
+ UndoRecPtr urp; /* undo recptr (undo record location). */
+ UnpackedUndoRecord *uur; /* actual undo record. */
+} UndoRecInfo;
+
+#endif
diff --git a/src/include/access/undoaction_xlog.h b/src/include/access/undoaction_xlog.h
new file mode 100644
index 0000000..cd960d8
--- /dev/null
+++ b/src/include/access/undoaction_xlog.h
@@ -0,0 +1,67 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoaction_xlog.h
+ * undo action XLOG definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undoaction_xlog.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDOACTION_XLOG_H
+#define UNDOACTION_XLOG_H
+
+#include "access/undolog.h"
+#include "access/xlogreader.h"
+#include "lib/stringinfo.h"
+#include "storage/off.h"
+
+/*
+ * WAL record definitions for undoactions.c's WAL operations
+ */
+#define XLOG_UNDO_PAGE 0x00
+#define XLOG_UNDO_RESET_SLOT 0x10
+#define XLOG_UNDO_APPLY_PROGRESS 0x20
+
+/*
+ * xl_undoaction_page flag values, 8 bits are available.
+ */
+#define XLU_PAGE_CLEAR_VISIBILITY_MAP (1<<0)
+#define XLU_INIT_PAGE (1<<1)
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_page
+{
+ UndoRecPtr urec_ptr;
+ TransactionId xid;
+ int trans_slot_id; /* transaction slot id */
+} xl_undoaction_page;
+
+#define SizeOfUndoActionPage (offsetof(xl_undoaction_page, trans_slot_id) + sizeof(int))
+
+/* This is what we need to know about undo apply progress */
+typedef struct xl_undoapply_progress
+{
+ UndoRecPtr urec_ptr;
+ uint32 progress;
+} xl_undoapply_progress;
+
+#define SizeOfUndoActionProgress (offsetof(xl_undoapply_progress, progress) + sizeof(uint32))
+
+/* This is what we need to know about delete */
+typedef struct xl_undoaction_reset_slot
+{
+ UndoRecPtr urec_ptr;
+ int trans_slot_id; /* transaction slot id */
+ uint8 flags;
+} xl_undoaction_reset_slot;
+
+#define SizeOfUndoActionResetSlot (offsetof(xl_undoaction_reset_slot, flags) + sizeof(uint8))
+
+extern void undoaction_redo(XLogReaderState *record);
+extern void undoaction_desc(StringInfo buf, XLogReaderState *record);
+extern const char *undoaction_identify(uint8 info);
+
+#endif /* UNDOACTION_XLOG_H */
diff --git a/src/include/access/undodiscard.h b/src/include/access/undodiscard.h
new file mode 100644
index 0000000..70d6408
--- /dev/null
+++ b/src/include/access/undodiscard.h
@@ -0,0 +1,31 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoinsert.h
+ * undo discard definitions
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/undodiscard.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNDODISCARD_H
+#define UNDODISCARD_H
+
+#include "access/undolog.h"
+#include "access/xlogdefs.h"
+#include "catalog/pg_class.h"
+#include "storage/lwlock.h"
+
+/*
+ * Discard the undo for all the transaction whose xid is smaller than xmin
+ *
+ * Check the DiscardInfo memory array for each slot (every undo log) , process
+ * the undo log for all the slot which have xid smaller than xmin or invalid
+ * xid. Fetch the record from the undo log transaction by transaction until we
+ * find the xid which is not smaller than xmin.
+ */
+extern void UndoDiscard(TransactionId xmin, bool *hibernate);
+
+#endif /* UNDODISCARD_H */
diff --git a/src/include/access/undoinsert.h b/src/include/access/undoinsert.h
index c333f00..5e208ad 100644
--- a/src/include/access/undoinsert.h
+++ b/src/include/access/undoinsert.h
@@ -46,5 +46,7 @@ extern void UndoSetPrepareSize(UnpackedUndoRecord *undorecords, int nrecords,
TransactionId xid, UndoPersistence upersistence);
extern UndoRecPtr UndoGetPrevUndoRecptr(UndoRecPtr urp, uint16 prevlen, UndoRecPtr prevurp);
extern void ResetUndoBuffers(void);
+extern void PrepareUpdateUndoActionProgress(UndoRecPtr urecptr, int progress);
+extern void UndoRecordUpdateTransInfo(int idx);
#endif /* UNDOINSERT_H */
diff --git a/src/include/access/undorecord.h b/src/include/access/undorecord.h
index 0dcf1b1..79eb173 100644
--- a/src/include/access/undorecord.h
+++ b/src/include/access/undorecord.h
@@ -20,6 +20,17 @@
#include "storage/buf.h"
#include "storage/off.h"
+typedef enum undorectype
+{
+ UNDO_INSERT,
+ UNDO_MULTI_INSERT,
+ UNDO_DELETE,
+ UNDO_INPLACE_UPDATE,
+ UNDO_UPDATE,
+ UNDO_XID_LOCK_ONLY,
+ UNDO_XID_MULTI_LOCK_ONLY,
+ UNDO_ITEMID_UNUSED
+} undorectype;
/*
* Every undo record begins with an UndoRecordHeader structure, which is
@@ -30,6 +41,7 @@
*/
typedef struct UndoRecordHeader
{
+ RmgrId urec_rmid; /* RMGR [XXX:TODO: this creates an alignment hole?] */
uint8 urec_type; /* record type code */
uint8 urec_info; /* flag bits */
uint16 urec_prevlen; /* length of previous record in bytes */
@@ -37,7 +49,7 @@ typedef struct UndoRecordHeader
/*
* Transaction id that has modified the tuple present in this undo record.
- * If this is older than oldestXidWithEpochHavingUndo, then we can
+ * If this is older than oldestXidHavingUndo, then we can
* consider the tuple in this undo record as visible.
*/
TransactionId urec_prevxid;
@@ -161,6 +173,7 @@ typedef struct UndoRecordPayload
*/
typedef struct UnpackedUndoRecord
{
+ RmgrId uur_rmid; /* rmgr ID */
uint8 uur_type; /* record type code */
uint8 uur_info; /* flag bits */
uint16 uur_prevlen; /* length of previous record */
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index ddaa633..f20a47c 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -419,6 +419,7 @@ extern XLogRecPtr XactLogAbortRecord(TimestampTz abort_time,
int xactflags, TransactionId twophase_xid,
const char *twophase_gid);
extern void xact_redo(XLogReaderState *record);
+extern void XactPerformUndoActionsIfPending(void);
/* xactdesc.c */
extern void xact_desc(StringInfo buf, XLogReaderState *record);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d18a1cd..b630d8c 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -225,6 +225,9 @@ extern bool XLOG_DEBUG;
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+/* Generate a 64-bit xid by using epoch and 32-bit xid. */
+#define MakeEpochXid(epoch, xid) \
+ ((epoch << 32) | (xid))
/* Checkpoint statistics */
typedef struct CheckpointStatsData
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 3c86037..18c64db 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -21,8 +21,10 @@
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
+#include "access/undorecord.h"
#include "datatype/timestamp.h"
#include "lib/stringinfo.h"
+#include "nodes/pg_list.h"
#include "pgtime.h"
#include "storage/block.h"
#include "storage/relfilenode.h"
@@ -294,9 +296,14 @@ typedef struct RmgrData
void (*rm_startup) (void);
void (*rm_cleanup) (void);
void (*rm_mask) (char *pagedata, BlockNumber blkno);
+ bool (*rm_undo) (List *luinfo, UndoRecPtr urec_ptr, Oid reloid,
+ TransactionId xid, BlockNumber blkno,
+ bool blk_chain_complete, bool rellock,
+ int options);
+ void (*rm_undo_desc) (StringInfo buf, UnpackedUndoRecord *record);
} RmgrData;
-extern const RmgrData RmgrTable[];
+extern PGDLLIMPORT const RmgrData RmgrTable[];
/*
* Exported to support xlog switching from checkpointer
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index a4aa83b..8a52124 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -61,6 +61,13 @@ typedef struct CheckPoint
* set to InvalidTransactionId.
*/
TransactionId oldestActiveXid;
+
+ /*
+ * Oldest transaction id which is having undo. Include this value
+ * in the checkpoint record so that whenever server starts we get proper
+ * value.
+ */
+ uint32 oldestXidHavingUndo;
} CheckPoint;
/* XLOG info values for XLOG rmgr */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index c9e3500..8edbbda 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -245,6 +245,7 @@ extern PGDLLIMPORT bool allowSystemTableMods;
extern PGDLLIMPORT int work_mem;
extern PGDLLIMPORT int maintenance_work_mem;
extern PGDLLIMPORT int max_parallel_maintenance_workers;
+extern PGDLLIMPORT int rollback_overflow_size;
extern int VacuumCostPageHit;
extern int VacuumCostPageMiss;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e6b22f4..319b36f 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -749,6 +749,7 @@ typedef enum BackendState
#define PG_WAIT_TIMEOUT 0x09000000U
#define PG_WAIT_IO 0x0A000000U
#define PG_WAIT_PAGE_TRANS_SLOT 0x0B000000U
+#define PG_WAIT_ROLLBACK_HT 0x0C000000U
/* ----------
* Wait Events - Activity
@@ -774,6 +775,8 @@ typedef enum
WAIT_EVENT_WAL_RECEIVER_MAIN,
WAIT_EVENT_WAL_SENDER_MAIN,
WAIT_EVENT_WAL_WRITER_MAIN,
+ WAIT_EVENT_UNDO_DISCARD_WORKER_MAIN,
+ WAIT_EVENT_UNDO_LAUNCHER_MAIN
} WaitEventActivity;
/* ----------
diff --git a/src/include/postmaster/discardworker.h b/src/include/postmaster/discardworker.h
new file mode 100644
index 0000000..f00c6c4
--- /dev/null
+++ b/src/include/postmaster/discardworker.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * discardworker.h
+ * Exports from postmaster/discardworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/discardworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _DISCARDWORKER_H
+#define _DISCARDWORKER_H
+
+/*
+ * This function will perform multiple actions based on need. (a) retrieve
+ * transactions which have become all-visible and truncate the associated undo
+ * logs or will increment the tail pointer. (b) drop the buffers corresponding
+ * to truncated pages.
+ */
+extern void DiscardWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void DiscardWorkerRegister(void);
+
+#endif /* _DISCARDWORKER_H */
diff --git a/src/include/postmaster/undoloop.h b/src/include/postmaster/undoloop.h
new file mode 100644
index 0000000..f979401
--- /dev/null
+++ b/src/include/postmaster/undoloop.h
@@ -0,0 +1,90 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoloop.h
+ * Exports from postmaster/undoloop.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ *
+ * src/include/postmaster/undoloop.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOLOOP_H
+#define _UNDOLOOP_H
+
+#include "access/undoinsert.h"
+#include "utils/hsearch.h"
+#include "utils/relcache.h"
+
+
+/* Various options while executing the undo actions for the page. */
+#define UNDO_ACTION_UPDATE_TPD 0x0001
+
+/* Remembers the last seen RecentGlobalXmin */
+TransactionId latestRecentGlobalXmin;
+
+/*
+ * This function will read the undo records starting from the undo
+ * from_urecptr till to_urecptr and if to_urecptr is invalid then till the
+ * first undo location of transaction. This also discards the buffers by
+ * calling DropUndoBuffers for which undo log is removed. This function
+ * can be used by RollbackToSavePoint, by Rollback, by undoworker to complete
+ * the work of errored out transactions or when there is an error in single
+ * user mode.
+ */
+extern void execute_undo_actions(UndoRecPtr from_urecptr,
+ UndoRecPtr to_urecptr, bool nopartial, bool rewind, bool rellock);
+extern void process_and_execute_undo_actions_page(UndoRecPtr from_urecptr,
+ Relation rel, Buffer buffer, uint32 epoch,
+ TransactionId xid, int slot_no);
+
+/*
+ * This function will be responsible to truncate the undo logs
+ * for transactions that become all-visible after RecentGlobalXmin has
+ * advanced (value is different than latestRecentGlobalXmin). The easiest
+ * way could be to traverse the undo log array that contains least transaction
+ * id for that undo log and see if it precedes RecentGlobalXmin, then start
+ * discarding the undo log for that transaction (moving the tail pointer of
+ * undo log) till it finds the transaction which is not all-visible. This also
+ * discards the buffers by calling ForgetBuffer for which undo log is
+ * removed. This function can be invoked by undoworker or after commit in
+ * single user mode.
+ */
+extern void recover_undo_pages();
+
+/*
+ * To increase the efficiency of the zheap system, we create a hash table for
+ * the rollbacks. All the rollback requests exceeding certain threshold, are
+ * pushed to this table. Undo worker starts reading the entries from this hash
+ * table one at a time, performs undo actions related to the respective xid and
+ * removes them from the hash table. This way backend is free from performing the
+ * undo actions in case of heavy rollbacks. The data structures and the routines
+ * required for this infrastructure are as follows.
+ */
+
+/* This is the data structure for each hash table entry for rollbacks. */
+typedef struct RollbackHashEntry
+{
+ UndoRecPtr start_urec_ptr;
+ UndoRecPtr end_urec_ptr;
+ Oid dbid;
+} RollbackHashEntry;
+
+extern bool RollbackHTIsFull(void);
+
+/* To push the rollback requests from backend to the respective hash table */
+extern bool PushRollbackReq(UndoRecPtr start_urec_ptr, UndoRecPtr end_urec_ptr,
+ Oid dbid);
+
+/* To perform the undo actions reading from the hash table */
+extern void RollbackFromHT(Oid dbid);
+/* To calculate the size of the hash table size for rollabcks. */
+extern int RollbackHTSize(void);
+/* To remove the all the entries of a database from hash-table */
+extern void RollbackHTCleanup(Oid dbid);
+/* To initialize the hash table in shared memory for rollbacks. */
+extern void InitRollbackHashTable(void);
+extern List *RollbackHTGetDBList(void);
+extern bool ConditionTransactionUndoActionLock(TransactionId xid);
+extern void TransactionUndoActionLockRelease(TransactionId xid);
+#endif /* _UNDOLOOP_H */
diff --git a/src/include/postmaster/undoworker.h b/src/include/postmaster/undoworker.h
new file mode 100644
index 0000000..c277e01
--- /dev/null
+++ b/src/include/postmaster/undoworker.h
@@ -0,0 +1,39 @@
+/*-------------------------------------------------------------------------
+ *
+ * undoworker.h
+ * Exports from postmaster/undoworker.c.
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/postmaster/undoworker.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _UNDOWORKER_H
+#define _UNDOWORKER_H
+
+/* GUC options */
+/* undo worker sleep time between rounds */
+extern int UndoWorkerDelay;
+
+/*
+ * This function will perform multiple actions based on need. (a) retreive
+ * transaction and its corresponding undopoiter from shared memory queue and
+ * call undoloop to perform undo actions. After applying all the undo records
+ * for a particular transaction, it will increment the tail pointer in undo log.
+ * (b) it needs to retrieve transactions which have become all-visible and truncate
+ * the associated undo logs or will increment the tail pointer. (c) udjust the
+ * number of undo workers based on the work required to perform undo actions
+ * (it could be size of shared memory queue containing transactions that needs
+ * aborts). (d) drop the buffers corresponding to truncated pages (e) Sleep for
+ * UndoWorkerDelay, if there is no more work.
+ */
+extern void UndoWorkerMain(Datum main_arg) pg_attribute_noreturn();
+extern void UndoLauncherRegister(void);
+extern void UndoLauncherShmemInit(void);
+extern Size UndoLauncherShmemSize(void);
+extern void UndoLauncherMain(Datum main_arg);
+extern void UndoWorkerMain(Datum main_arg);
+
+#endif /* _UNDOWORKER_H */
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 16b927c..1d3ed2e 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -150,6 +150,8 @@ typedef enum LockTagType
LOCKTAG_VIRTUALTRANSACTION, /* virtual transaction (ditto) */
/* ID info for a virtual transaction is its VirtualTransactionId */
LOCKTAG_SPECULATIVE_TOKEN, /* speculative insertion Xid and token */
+ /* ID info for an transaction undoaction is transaction id */
+ LOCKTAG_TRANSACTION_UNDOACTION, /* transaction (waiting for undoaction) */
/* ID info for a transaction is its TransactionId */
LOCKTAG_OBJECT, /* non-relation database object */
/* ID info for an object is DB OID + CLASS OID + OBJECT OID + SUBID */
@@ -246,6 +248,14 @@ typedef struct LOCKTAG
(locktag).locktag_type = LOCKTAG_SPECULATIVE_TOKEN, \
(locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+#define SET_LOCKTAG_TRANSACTION_UNDOACTION(locktag,xid) \
+ ((locktag).locktag_field1 = (xid), \
+ (locktag).locktag_field2 = 0, \
+ (locktag).locktag_field3 = 0, \
+ (locktag).locktag_field4 = 0, \
+ (locktag).locktag_type = LOCKTAG_TRANSACTION_UNDOACTION, \
+ (locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)
+
#define SET_LOCKTAG_OBJECT(locktag,dboid,classoid,objoid,objsubid) \
((locktag).locktag_field1 = (dboid), \
(locktag).locktag_field2 = (classoid), \
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index d203acb..efde86d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -270,6 +270,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Oldest transaction id which is having undo. */
+ pg_atomic_uint32 oldestXidHavingUndo;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
1.8.3.1
^ permalink raw reply [nested|flat] 37+ messages in thread
* current_logfiles not following group access and instead follows log_file_mode permissions
@ 2019-01-15 04:08 Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Haribabu Kommi @ 2019-01-15 04:08 UTC (permalink / raw)
To: [email protected]
current_logfiles is a meta data file, that stores the current log writing
file, and this file
presents in the data directory. This file doesn't follow the group access
mode set at
the initdb time, but it follows the log_file_mode permissions.
without group access permissions, backup with group access can lead to
failure.
Attached patch fix the problem.
comments?
Regards,
Haribabu Kommi
Fujitsu Australia
Attachments:
[application/octet-stream] 0001-current_logfiles-file-following-group-access-mode.patch (1.6K, ../../CAJrrPGcEotF1P7AWoeQyD3Pqr-0xkQg_Herv98DjbaMj+naozw@mail.gmail.com/3-0001-current_logfiles-file-following-group-access-mode.patch)
download | inline diff:
From 13986b6e47df953685032d2305df1aa393e8a3f5 Mon Sep 17 00:00:00 2001
From: Hari Babu <[email protected]>
Date: Tue, 15 Jan 2019 14:58:21 +1100
Subject: [PATCH] current_logfiles file following group access mode
Earlier current_logfiles file used to follow the
log_file_mode permissions, but this file contains
only the meta data of the log file and following
log_file_mode can leads to backup failure.
---
src/backend/postmaster/syslogger.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/src/backend/postmaster/syslogger.c b/src/backend/postmaster/syslogger.c
index d1ea46deb8..d31d8f03fc 100644
--- a/src/backend/postmaster/syslogger.c
+++ b/src/backend/postmaster/syslogger.c
@@ -30,7 +30,7 @@
#include <unistd.h>
#include <sys/stat.h>
#include <sys/time.h>
-
+#include "common/file_perm.h"
#include "lib/stringinfo.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
@@ -1465,7 +1465,7 @@ update_metainfo_datafile(void)
return;
}
- if ((fh = logfile_open(LOG_METAINFO_DATAFILE_TMP, "w", true)) == NULL)
+ if ((fh = fopen(LOG_METAINFO_DATAFILE_TMP, "w")) == NULL)
{
ereport(LOG,
(errcode_for_file_access(),
@@ -1501,6 +1501,12 @@ update_metainfo_datafile(void)
}
fclose(fh);
+ if (chmod(LOG_METAINFO_DATAFILE_TMP, pg_file_create_mode) != 0)
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not set permissions of file \"%s\": %m",
+ LOG_METAINFO_DATAFILE_TMP)));
+
if (rename(LOG_METAINFO_DATAFILE_TMP, LOG_METAINFO_DATAFILE) != 0)
ereport(LOG,
(errcode_for_file_access(),
--
2.18.0.windows.1
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
@ 2019-01-15 05:15 ` Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Michael Paquier @ 2019-01-15 05:15 UTC (permalink / raw)
To: Haribabu Kommi <[email protected]>; +Cc: [email protected]
On Tue, Jan 15, 2019 at 03:08:41PM +1100, Haribabu Kommi wrote:
> current_logfiles is a meta data file, that stores the current log writing
> file, and this file presents in the data directory. This file
> doesn't follow the group access mode set at the initdb time, but it
> follows the log_file_mode permissions.
>
> Without group access permissions, backup with group access can lead to
> failure. Attached patch fix the problem.
initdb enforces log_file_mode to 0640 when using the group mode, still
if one enforces the parameter value then current_logfiles would just
stick with it. This is not really user-friendly. This impacts also
normal log files as these get included in base backups if the log path
is within the data folder (not everybody uses an absolute path out of
the data folder for the logs).
One way to think about this is that we may want to worry also about
normal log files and document that one had better be careful with the
setting of log_file_mode? Still, as we are talking about a file
aiming at storing meta-data for log files, something like what you
suggest can make sense.
When discussing about pg_current_logfile(), I raised the point about
not including as well in base backups which would also address the
problem reported here. However we decided to keep it because it can
be helpful to know what's the last log file associated to a base
backup for debugging purposes:
https://www.postgresql.org/message-id/[email protected]
Instead of what you are proposing, why not revisiting that and just
exclude the file from base backups. I would be in favor of just doing
that instead of switching the file's permission from log_file_mode to
pg_file_create_mode.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
@ 2019-01-15 08:55 ` Haribabu Kommi <[email protected]>
2019-01-15 14:47 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Haribabu Kommi @ 2019-01-15 08:55 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: [email protected]
On Tue, Jan 15, 2019 at 4:15 PM Michael Paquier <[email protected]> wrote:
> On Tue, Jan 15, 2019 at 03:08:41PM +1100, Haribabu Kommi wrote:
> > current_logfiles is a meta data file, that stores the current log writing
> > file, and this file presents in the data directory. This file
> > doesn't follow the group access mode set at the initdb time, but it
> > follows the log_file_mode permissions.
> >
> > Without group access permissions, backup with group access can lead to
> > failure. Attached patch fix the problem.
>
> initdb enforces log_file_mode to 0640 when using the group mode, still
> if one enforces the parameter value then current_logfiles would just
> stick with it. This is not really user-friendly. This impacts also
> normal log files as these get included in base backups if the log path
> is within the data folder (not everybody uses an absolute path out of
> the data folder for the logs).
>
we got this problem when the log_file_mode is set 0600 but the database
file are with group access permissions. In our scenario, the log files are
outside the data folder, so we faced the problem with current_logfiles
file.
> One way to think about this is that we may want to worry also about
> normal log files and document that one had better be careful with the
> setting of log_file_mode? Still, as we are talking about a file
> aiming at storing meta-data for log files, something like what you
> suggest can make sense.
>
Yes, with log_file_mode less than 0640 containing the log files inside
the data directory can leads to backup failure. Yes, providing extra
information about group access when log_file_mode is getting chosen.
Another option is how about not letting user to choose less than 0640
when the group access mode is enabled?
> When discussing about pg_current_logfile(), I raised the point about
> not including as well in base backups which would also address the
> problem reported here. However we decided to keep it because it can
> be helpful to know what's the last log file associated to a base
> backup for debugging purposes:
>
> https://www.postgresql.org/message-id/[email protected]
>
> Instead of what you are proposing, why not revisiting that and just
> exclude the file from base backups. I would be in favor of just doing
> that instead of switching the file's permission from log_file_mode to
> pg_file_create_mode.
>
I am not sure how much useful having the details of the log file in the
backup.
It may be useful when there is any problem with backup.
Excluding the file in the backup can solve the problem of backup by an
unprivileged user. Is there any scenarios it can cause problems if it
doesn't follow the group access mode?
Regards,
Haribabu Kommi
Fujitsu Australia
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
@ 2019-01-15 14:47 ` Tom Lane <[email protected]>
2019-01-15 15:53 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Tom Lane @ 2019-01-15 14:47 UTC (permalink / raw)
To: Haribabu Kommi <[email protected]>; +Cc: Michael Paquier <[email protected]>; [email protected]
Haribabu Kommi <[email protected]> writes:
> Excluding the file in the backup can solve the problem of backup by an
> unprivileged user. Is there any scenarios it can cause problems if it
> doesn't follow the group access mode?
The point of this file, as I understood it, was to allow someone who's
allowed to read the log files to find out which one is the latest. It
makes zero sense for it to have different permissions from the log files,
because doing that would break its only use-case.
I am wondering what is the use-case for a backup arrangement that's so
fragile it can't cope with varying permissions in the data directory.
regards, tom lane
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 14:47 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
@ 2019-01-15 15:53 ` Tom Lane <[email protected]>
2019-01-16 02:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Tom Lane @ 2019-01-15 15:53 UTC (permalink / raw)
To: Haribabu Kommi <[email protected]>; +Cc: Michael Paquier <[email protected]>; [email protected]
I wrote:
> Haribabu Kommi <[email protected]> writes:
>> Excluding the file in the backup can solve the problem of backup by an
>> unprivileged user. Is there any scenarios it can cause problems if it
>> doesn't follow the group access mode?
> The point of this file, as I understood it, was to allow someone who's
> allowed to read the log files to find out which one is the latest. It
> makes zero sense for it to have different permissions from the log files,
> because doing that would break its only use-case.
On reflection, maybe the problem is not that we're giving the file
the wrong permissions, but that we're putting it in the wrong place?
That is, seems like it should be in the logfile directory not the
data directory. That would certainly simplify the intended use-case,
and it would fix this complaint too.
regards, tom lane
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 14:47 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-15 15:53 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
@ 2019-01-16 02:08 ` Michael Paquier <[email protected]>
2019-01-16 18:22 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Michael Paquier @ 2019-01-16 02:08 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Haribabu Kommi <[email protected]>; [email protected]; Gilles Darold <[email protected]>
On Tue, Jan 15, 2019 at 10:53:30AM -0500, Tom Lane wrote:
> On reflection, maybe the problem is not that we're giving the file
> the wrong permissions, but that we're putting it in the wrong place?
> That is, seems like it should be in the logfile directory not the
> data directory. That would certainly simplify the intended use-case,
> and it would fix this complaint too.
Yeah, thinking more on this one using for this file different
permissions than the log files makes little sense, so what you propose
here seems like a sensible thing to do things. Even if we exclude the
file from native BASE_BACKUP this would not solve the case of custom
backup solutions doing their own copy of things, when they rely on
group-read permissions. This would not solve completely the problem
anyway if log files are in the data folder, but it would address the
case where the log files are in an absolute path out of the data
folder.
I am adding in CC Gilles who implemented current_logfiles for his
input.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 14:47 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-15 15:53 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 02:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
@ 2019-01-16 18:22 ` Stephen Frost <[email protected]>
2019-01-16 18:39 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Stephen Frost @ 2019-01-16 18:22 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Tom Lane <[email protected]>; Haribabu Kommi <[email protected]>; [email protected]; Gilles Darold <[email protected]>
Greetings,
* Michael Paquier ([email protected]) wrote:
> On Tue, Jan 15, 2019 at 10:53:30AM -0500, Tom Lane wrote:
> > On reflection, maybe the problem is not that we're giving the file
> > the wrong permissions, but that we're putting it in the wrong place?
> > That is, seems like it should be in the logfile directory not the
> > data directory. That would certainly simplify the intended use-case,
> > and it would fix this complaint too.
>
> Yeah, thinking more on this one using for this file different
> permissions than the log files makes little sense, so what you propose
> here seems like a sensible thing to do things. Even if we exclude the
> file from native BASE_BACKUP this would not solve the case of custom
> backup solutions doing their own copy of things, when they rely on
> group-read permissions. This would not solve completely the problem
> anyway if log files are in the data folder, but it would address the
> case where the log files are in an absolute path out of the data
> folder.
Actually, I agree with the initial patch on the basis that this file
that's being created (which I'm honestly a bit amazed that we're doing
this way; certainly seems rather grotty to me) is surely not an actual
*log* file and therefore using logfile_open() to open it doesn't seem
quite right. I would have hoped for a way to pass this information that
didn't involve a file at all, but I'll assume that was discussed already
and good reasons put forth as to why we can't avoid it.
I'm not really sure putting it into the logfile directory is such a hot
idea as users might have set up external log file rotation of files in
that directory. Of course, in that case they'd probably signal PG right
afterwards and PG would go write out a new file, but it still seems
pretty awkward. I'm not terribly against solving this issue that way
either though, but I tend to think the originally proposed patch is more
sensible.
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 14:47 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-15 15:53 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 02:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-16 18:22 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
@ 2019-01-16 18:39 ` Tom Lane <[email protected]>
2019-01-16 18:49 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Tom Lane @ 2019-01-16 18:39 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Michael Paquier <[email protected]>; Haribabu Kommi <[email protected]>; [email protected]; Gilles Darold <[email protected]>
Stephen Frost <[email protected]> writes:
> * Michael Paquier ([email protected]) wrote:
>> On Tue, Jan 15, 2019 at 10:53:30AM -0500, Tom Lane wrote:
>>> On reflection, maybe the problem is not that we're giving the file
>>> the wrong permissions, but that we're putting it in the wrong place?
> I'm not really sure putting it into the logfile directory is such a hot
> idea as users might have set up external log file rotation of files in
> that directory. Of course, in that case they'd probably signal PG right
> afterwards and PG would go write out a new file, but it still seems
> pretty awkward. I'm not terribly against solving this issue that way
> either though, but I tend to think the originally proposed patch is more
> sensible.
I dunno, I think that the current design was made without any thought
whatsoever about the log-files-outside-the-data-directory case. If
you're trying to set things up that way, it's because you want to give
logfile read access to people who shouldn't be able to look into the
data directory proper. That makes current_logfiles pretty useless
to such people, as it's now designed.
Now, if the expectation is that current_logfiles is just an internal
working file that users shouldn't access directly, then this argument
is wrong --- but then why is it documented in user-facing docs?
If we're going to accept the patch as-is, then it logically follows
that we should de-document current_logfiles, because we're taking the
position that it's an internal temporary file not meant for user access.
I don't really believe your argument about log rotation: a rotator
would presumably be configured either to pay attention to file name
patterns (which current_logfiles wouldn't match) or to file age
(which current_logfiles shouldn't satisfy either, since it's always
rewritten when we switch logfiles).
If we wanted to worry about that case, a possible solution is to make the
current_logfiles pathname user-configurable so it could be put in some
third directory. But I think that adds more complexity than is justified
--- and not just for us, but for programs trying to find and use
current_logfiles.
regards, tom lane
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 14:47 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-15 15:53 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 02:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-16 18:22 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-16 18:39 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
@ 2019-01-16 18:49 ` Stephen Frost <[email protected]>
2019-01-18 04:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Stephen Frost @ 2019-01-16 18:49 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Michael Paquier <[email protected]>; Haribabu Kommi <[email protected]>; [email protected]; Gilles Darold <[email protected]>
Greetings,
* Tom Lane ([email protected]) wrote:
> Stephen Frost <[email protected]> writes:
> > * Michael Paquier ([email protected]) wrote:
> >> On Tue, Jan 15, 2019 at 10:53:30AM -0500, Tom Lane wrote:
> >>> On reflection, maybe the problem is not that we're giving the file
> >>> the wrong permissions, but that we're putting it in the wrong place?
>
> > I'm not really sure putting it into the logfile directory is such a hot
> > idea as users might have set up external log file rotation of files in
> > that directory. Of course, in that case they'd probably signal PG right
> > afterwards and PG would go write out a new file, but it still seems
> > pretty awkward. I'm not terribly against solving this issue that way
> > either though, but I tend to think the originally proposed patch is more
> > sensible.
>
> I dunno, I think that the current design was made without any thought
> whatsoever about the log-files-outside-the-data-directory case. If
> you're trying to set things up that way, it's because you want to give
> logfile read access to people who shouldn't be able to look into the
> data directory proper. That makes current_logfiles pretty useless
> to such people, as it's now designed.
... or you just want to move the log files to a more sensible location
than the data directory. The justification for log_file_mode existing
is because you might want to have log files with different privileges,
but that's quite a different thing.
> Now, if the expectation is that current_logfiles is just an internal
> working file that users shouldn't access directly, then this argument
> is wrong --- but then why is it documented in user-facing docs?
I really couldn't say why it's documented in the user-facing docs, and
for my 2c I don't really think it should be- there's a function to get
that information. Sprinkling the data directory with files for users to
access directly doesn't exactly fit my view of what a good API looks
like.
The fact that there isn't any discussion about where that file actually
lives does make me suspect you're right that log files outside the data
directory wasn't really contemplated.
> If we're going to accept the patch as-is, then it logically follows
> that we should de-document current_logfiles, because we're taking the
> position that it's an internal temporary file not meant for user access.
... and hopefully we'd get rid of it one day entirely.
> I don't really believe your argument about log rotation: a rotator
> would presumably be configured either to pay attention to file name
> patterns (which current_logfiles wouldn't match) or to file age
> (which current_logfiles shouldn't satisfy either, since it's always
> rewritten when we switch logfiles).
Yes, a good pattern would avoid picking up on this file and most are
configured that way (though they are maybe not as specific as you might
think- the default here is just /var/log/postgresql/*.log).
> If we wanted to worry about that case, a possible solution is to make the
> current_logfiles pathname user-configurable so it could be put in some
> third directory. But I think that adds more complexity than is justified
> --- and not just for us, but for programs trying to find and use
> current_logfiles.
I'd much rather move to get rid of that file rather than increase its
visability- programs should be using the provided function.
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 14:47 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-15 15:53 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 02:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-16 18:22 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-16 18:39 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 18:49 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
@ 2019-01-18 04:08 ` Haribabu Kommi <[email protected]>
2019-01-18 14:50 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Haribabu Kommi @ 2019-01-18 04:08 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Tom Lane <[email protected]>; Michael Paquier <[email protected]>; [email protected]; Gilles Darold <[email protected]>
On Thu, Jan 17, 2019 at 5:49 AM Stephen Frost <[email protected]> wrote:
> Greetings,
>
> * Tom Lane ([email protected]) wrote:
> > Stephen Frost <[email protected]> writes:
> > > * Michael Paquier ([email protected]) wrote:
> > >> On Tue, Jan 15, 2019 at 10:53:30AM -0500, Tom Lane wrote:
> > >>> On reflection, maybe the problem is not that we're giving the file
> > >>> the wrong permissions, but that we're putting it in the wrong place?
> >
> > > I'm not really sure putting it into the logfile directory is such a hot
> > > idea as users might have set up external log file rotation of files in
> > > that directory. Of course, in that case they'd probably signal PG
> right
> > > afterwards and PG would go write out a new file, but it still seems
> > > pretty awkward. I'm not terribly against solving this issue that way
> > > either though, but I tend to think the originally proposed patch is
> more
> > > sensible.
> >
> > I dunno, I think that the current design was made without any thought
> > whatsoever about the log-files-outside-the-data-directory case. If
> > you're trying to set things up that way, it's because you want to give
> > logfile read access to people who shouldn't be able to look into the
> > data directory proper. That makes current_logfiles pretty useless
> > to such people, as it's now designed.
>
> ... or you just want to move the log files to a more sensible location
> than the data directory. The justification for log_file_mode existing
> is because you might want to have log files with different privileges,
> but that's quite a different thing.
>
Thanks for sharing your opinions.
The current_logfiles is used to store the meta data information of current
writing log files, that is different to log files, so giving permissions of
the
log file may not be correct,
> Now, if the expectation is that current_logfiles is just an internal
> > working file that users shouldn't access directly, then this argument
> > is wrong --- but then why is it documented in user-facing docs?
>
> I really couldn't say why it's documented in the user-facing docs, and
> for my 2c I don't really think it should be- there's a function to get
> that information. Sprinkling the data directory with files for users to
> access directly doesn't exactly fit my view of what a good API looks
> like.
>
> The fact that there isn't any discussion about where that file actually
> lives does make me suspect you're right that log files outside the data
> directory wasn't really contemplated.
>
I can only think of reading this file by the user directly when the server
is not available, but I don't find any scenario where that is required?
> > If we're going to accept the patch as-is, then it logically follows
> > that we should de-document current_logfiles, because we're taking the
> > position that it's an internal temporary file not meant for user access.
>
> ... and hopefully we'd get rid of it one day entirely.
>
If there is no use of it when server is offline, it will be better to
remove that
file with an alternative to provide the current log file name.
With group access mode, the default value of log_file_mode is changed,
Attached patch reflects the same in docs.
Regards,
Haribabu Kommi
Fujitsu Australia
Attachments:
[application/octet-stream] 0001-log_file_mode-default-value-update.patch (2.0K, ../../CAJrrPGdVJ0qxZv8hvgV56qsXyXdm03=54XHhPCsLESu-PZ49GQ@mail.gmail.com/3-0001-log_file_mode-default-value-update.patch)
download | inline diff:
From b1def35feeb99542ae6ab55005f302b7314131e0 Mon Sep 17 00:00:00 2001
From: Hari Babu <[email protected]>
Date: Fri, 18 Jan 2019 14:21:51 +1100
Subject: [PATCH] log_file_mode default value update
For group read access cluster, the default value of
log_file_mode is 0640 to allow reading of log files by the
members of the same group.
---
doc/src/sgml/config.sgml | 15 +++++++--------
1 file changed, 7 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index b6f5822b84..8aaf5b133e 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5440,14 +5440,13 @@ local0.* /var/log/postgresql
must start with a <literal>0</literal> (zero).)
</para>
<para>
- The default permissions are <literal>0600</literal>, meaning only the
- server owner can read or write the log files. The other commonly
- useful setting is <literal>0640</literal>, allowing members of the owner's
- group to read the files. Note however that to make use of such a
- setting, you'll need to alter <xref linkend="guc-log-directory"/> to
- store the files somewhere outside the cluster data directory. In
- any case, it's unwise to make the log files world-readable, since
- they might contain sensitive data.
+ The default permissions are either <literal>0600</literal>, meaning only the
+ server owner can read or write the log files or <literal>0640</literal>, that
+ allows any user in the same group can read the log files, based on the new
+ cluster created with <option>--allow-group-access</option> option of <command>initdb</command>
+ command. Note however that to make use of any setting other than default,
+ you'll need to alter <xref linkend="guc-log-directory"/> to store the files
+ somewhere outside the cluster data directory.
</para>
<para>
This parameter can only be set in the <filename>postgresql.conf</filename>
--
2.18.0.windows.1
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 14:47 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-15 15:53 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 02:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-16 18:22 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-16 18:39 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 18:49 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-18 04:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
@ 2019-01-18 14:50 ` Stephen Frost <[email protected]>
2019-01-19 01:49 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Stephen Frost @ 2019-01-18 14:50 UTC (permalink / raw)
To: Haribabu Kommi <[email protected]>; +Cc: Tom Lane <[email protected]>; Michael Paquier <[email protected]>; [email protected]; Gilles Darold <[email protected]>
Greetings,
* Haribabu Kommi ([email protected]) wrote:
> On Thu, Jan 17, 2019 at 5:49 AM Stephen Frost <[email protected]> wrote:
> > * Tom Lane ([email protected]) wrote:
> > > Now, if the expectation is that current_logfiles is just an internal
> > > working file that users shouldn't access directly, then this argument
> > > is wrong --- but then why is it documented in user-facing docs?
> >
> > I really couldn't say why it's documented in the user-facing docs, and
> > for my 2c I don't really think it should be- there's a function to get
> > that information. Sprinkling the data directory with files for users to
> > access directly doesn't exactly fit my view of what a good API looks
> > like.
> >
> > The fact that there isn't any discussion about where that file actually
> > lives does make me suspect you're right that log files outside the data
> > directory wasn't really contemplated.
>
> I can only think of reading this file by the user directly when the server
> is not available, but I don't find any scenario where that is required?
Yeah, I agree, and if the server isn't running then there really isn't
a "current" logfile, as defined, since the server isn't writing to any
particular log file.
> > > If we're going to accept the patch as-is, then it logically follows
> > > that we should de-document current_logfiles, because we're taking the
> > > position that it's an internal temporary file not meant for user access.
> >
> > ... and hopefully we'd get rid of it one day entirely.
>
> If there is no use of it when server is offline, it will be better to
> remove that
> file with an alternative to provide the current log file name.
It'd probably be good to give folks an opportunity to voice their
opinion regarding their use-case for the file existing as it does and
being documented as it is. At first blush, to me anyway, it seems like
maybe this was a case of "over-documenting" of the feature by including
in user-facing documentation something that was really there for
internal reasons, but I could certainly be wrong and maybe there's a
reason why it's really necessary to have the file around for users.
> With group access mode, the default value of log_file_mode is changed,
> Attached patch reflects the same in docs.
Yes, we should update the documentation in this regard, though it's
really an independent thing as that documentation should have been
updated in the original group-access patch, so I'll see about fixing
it and back-patching it.
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 14:47 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-15 15:53 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 02:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-16 18:22 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-16 18:39 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 18:49 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-18 04:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-18 14:50 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
@ 2019-01-19 01:49 ` Michael Paquier <[email protected]>
2019-01-19 15:41 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Michael Paquier @ 2019-01-19 01:49 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Haribabu Kommi <[email protected]>; Tom Lane <[email protected]>; [email protected]; Gilles Darold <[email protected]>
On Fri, Jan 18, 2019 at 09:50:40AM -0500, Stephen Frost wrote:
> It'd probably be good to give folks an opportunity to voice their
> opinion regarding their use-case for the file existing as it does and
> being documented as it is. At first blush, to me anyway, it seems like
> maybe this was a case of "over-documenting" of the feature by including
> in user-facing documentation something that was really there for
> internal reasons, but I could certainly be wrong and maybe there's a
> reason why it's really necessary to have the file around for users.
It's not only that. By keeping the file in its current location, we
can prevent base backups to work even if logs files are out of the
data folder, which is rather user-friendly, and I think that advanced
users of Postgres are careful enough to split log files and main data
folders into different partitions, without symlinks from the data
folder to the log location and with log_directory set to an absolute
path, independent of the rest. So moving current_logfiles out of the
data folder to the base location of the log paths makes quite some
sense in my opinion for consistency.
Using a new GUC to specify where current_logfiles should be located
does not really justify the code complications in my opinion, and I'd
think that we should allow users with log file access to still look at
it, even manually and connected from the host as this can be useful
for debugging purposes (sometimes clocks of systems get changed as
they are not all the time going throuhg ntpd).
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: current_logfiles not following group access and instead follows log_file_mode permissions
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 14:47 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-15 15:53 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 02:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-16 18:22 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-16 18:39 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 18:49 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-18 04:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-18 14:50 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-19 01:49 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
@ 2019-01-19 15:41 ` Stephen Frost <[email protected]>
0 siblings, 0 replies; 37+ messages in thread
From: Stephen Frost @ 2019-01-19 15:41 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Haribabu Kommi <[email protected]>; Tom Lane <[email protected]>; [email protected]; Gilles Darold <[email protected]>
Greetings,
* Michael Paquier ([email protected]) wrote:
> On Fri, Jan 18, 2019 at 09:50:40AM -0500, Stephen Frost wrote:
> > It'd probably be good to give folks an opportunity to voice their
> > opinion regarding their use-case for the file existing as it does and
> > being documented as it is. At first blush, to me anyway, it seems like
> > maybe this was a case of "over-documenting" of the feature by including
> > in user-facing documentation something that was really there for
> > internal reasons, but I could certainly be wrong and maybe there's a
> > reason why it's really necessary to have the file around for users.
>
> It's not only that. By keeping the file in its current location, we
> can prevent base backups to work even if logs files are out of the
> data folder, which is rather user-friendly, and I think that advanced
> users of Postgres are careful enough to split log files and main data
> folders into different partitions, without symlinks from the data
> folder to the log location and with log_directory set to an absolute
> path, independent of the rest. So moving current_logfiles out of the
> data folder to the base location of the log paths makes quite some
> sense in my opinion for consistency.
As discussed up-thread, if we change current_logfiles to work the way
the rest of our data files do, then base backups would work fine with
the file in its current location. I don't buy how having that file in
the logfiles directory is more "consistent" with anything either- it's
certainly not a log file itself.
> Using a new GUC to specify where current_logfiles should be located
> does not really justify the code complications in my opinion, and I'd
> think that we should allow users with log file access to still look at
> it, even manually and connected from the host as this can be useful
> for debugging purposes (sometimes clocks of systems get changed as
> they are not all the time going throuhg ntpd).
I agree that we don't need a new GUC for this. I also don't really see
the use-case for this file being directly exposed to users- we have a
function specifically for this information and that's generally how
users should expect to get information like this- or like what the log
directory *is* to begin with, or where other files reside... I sure hope
that we aren't suggesting that asking users to write a parser for
postgresql.conf, with include directories and files, able to also handle
postgresql.auto.conf, is somehow user-friendly.
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 37+ messages in thread
* [PATCH 2/4] Change-tmplinit-argument
@ 2019-01-17 12:05 Arthur Zakirov <[email protected]>
0 siblings, 0 replies; 37+ messages in thread
From: Arthur Zakirov @ 2019-01-17 12:05 UTC (permalink / raw)
Reviewed-by: Tomas Vondra, Ildus Kurbangaliev
---
contrib/dict_int/dict_int.c | 4 +-
contrib/dict_xsyn/dict_xsyn.c | 4 +-
contrib/unaccent/unaccent.c | 4 +-
src/backend/commands/tsearchcmds.c | 10 ++++-
src/backend/snowball/dict_snowball.c | 4 +-
src/backend/tsearch/dict_ispell.c | 4 +-
src/backend/tsearch/dict_simple.c | 4 +-
src/backend/tsearch/dict_synonym.c | 4 +-
src/backend/tsearch/dict_thesaurus.c | 4 +-
src/backend/utils/cache/ts_cache.c | 13 +++++-
src/include/tsearch/ts_cache.h | 4 ++
src/include/tsearch/ts_public.h | 67 ++++++++++++++++++++++++++--
12 files changed, 105 insertions(+), 21 deletions(-)
diff --git a/contrib/dict_int/dict_int.c b/contrib/dict_int/dict_int.c
index 628b9769c3..ddde55eee4 100644
--- a/contrib/dict_int/dict_int.c
+++ b/contrib/dict_int/dict_int.c
@@ -30,7 +30,7 @@ PG_FUNCTION_INFO_V1(dintdict_lexize);
Datum
dintdict_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictInt *d;
ListCell *l;
@@ -38,7 +38,7 @@ dintdict_init(PG_FUNCTION_ARGS)
d->maxlen = 6;
d->rejectlong = false;
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/contrib/dict_xsyn/dict_xsyn.c b/contrib/dict_xsyn/dict_xsyn.c
index 509e14aee0..15b1a0033a 100644
--- a/contrib/dict_xsyn/dict_xsyn.c
+++ b/contrib/dict_xsyn/dict_xsyn.c
@@ -140,7 +140,7 @@ read_dictionary(DictSyn *d, const char *filename)
Datum
dxsyn_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictSyn *d;
ListCell *l;
char *filename = NULL;
@@ -153,7 +153,7 @@ dxsyn_init(PG_FUNCTION_ARGS)
d->matchsynonyms = false;
d->keepsynonyms = true;
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/contrib/unaccent/unaccent.c b/contrib/unaccent/unaccent.c
index fc5176e338..f3663cefd0 100644
--- a/contrib/unaccent/unaccent.c
+++ b/contrib/unaccent/unaccent.c
@@ -270,12 +270,12 @@ PG_FUNCTION_INFO_V1(unaccent_init);
Datum
unaccent_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
TrieChar *rootTrie = NULL;
bool fileloaded = false;
ListCell *l;
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/commands/tsearchcmds.c b/src/backend/commands/tsearchcmds.c
index 8e5eec22b5..30c5eb72a2 100644
--- a/src/backend/commands/tsearchcmds.c
+++ b/src/backend/commands/tsearchcmds.c
@@ -389,17 +389,25 @@ verify_dictoptions(Oid tmplId, List *dictoptions)
}
else
{
+ DictInitData init_data;
+
/*
* Copy the options just in case init method thinks it can scribble on
* them ...
*/
dictoptions = copyObject(dictoptions);
+ init_data.dict_options = dictoptions;
+ init_data.dict.id = InvalidOid;
+ init_data.dict.xmin = InvalidTransactionId;
+ init_data.dict.xmax = InvalidTransactionId;
+ ItemPointerSetInvalid(&init_data.dict.tid);
+
/*
* Call the init method and see if it complains. We don't worry about
* it leaking memory, since our command will soon be over anyway.
*/
- (void) OidFunctionCall1(initmethod, PointerGetDatum(dictoptions));
+ (void) OidFunctionCall1(initmethod, PointerGetDatum(&init_data));
}
ReleaseSysCache(tup);
diff --git a/src/backend/snowball/dict_snowball.c b/src/backend/snowball/dict_snowball.c
index 5166738310..f30f29865c 100644
--- a/src/backend/snowball/dict_snowball.c
+++ b/src/backend/snowball/dict_snowball.c
@@ -201,14 +201,14 @@ locate_stem_module(DictSnowball *d, const char *lang)
Datum
dsnowball_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictSnowball *d;
bool stoploaded = false;
ListCell *l;
d = (DictSnowball *) palloc0(sizeof(DictSnowball));
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/tsearch/dict_ispell.c b/src/backend/tsearch/dict_ispell.c
index 8b05a477f1..fc9a96abca 100644
--- a/src/backend/tsearch/dict_ispell.c
+++ b/src/backend/tsearch/dict_ispell.c
@@ -29,7 +29,7 @@ typedef struct
Datum
dispell_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictISpell *d;
bool affloaded = false,
dictloaded = false,
@@ -40,7 +40,7 @@ dispell_init(PG_FUNCTION_ARGS)
NIStartBuild(&(d->obj));
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/tsearch/dict_simple.c b/src/backend/tsearch/dict_simple.c
index 2f62ef00c8..c92744641b 100644
--- a/src/backend/tsearch/dict_simple.c
+++ b/src/backend/tsearch/dict_simple.c
@@ -29,7 +29,7 @@ typedef struct
Datum
dsimple_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictSimple *d = (DictSimple *) palloc0(sizeof(DictSimple));
bool stoploaded = false,
acceptloaded = false;
@@ -37,7 +37,7 @@ dsimple_init(PG_FUNCTION_ARGS)
d->accept = true; /* default */
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/tsearch/dict_synonym.c b/src/backend/tsearch/dict_synonym.c
index b6226df940..d3f5f0da3f 100644
--- a/src/backend/tsearch/dict_synonym.c
+++ b/src/backend/tsearch/dict_synonym.c
@@ -91,7 +91,7 @@ compareSyn(const void *a, const void *b)
Datum
dsynonym_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictSyn *d;
ListCell *l;
char *filename = NULL;
@@ -104,7 +104,7 @@ dsynonym_init(PG_FUNCTION_ARGS)
char *line = NULL;
uint16 flags = 0;
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/tsearch/dict_thesaurus.c b/src/backend/tsearch/dict_thesaurus.c
index 75f8deef6a..8962e252e0 100644
--- a/src/backend/tsearch/dict_thesaurus.c
+++ b/src/backend/tsearch/dict_thesaurus.c
@@ -604,7 +604,7 @@ compileTheSubstitute(DictThesaurus *d)
Datum
thesaurus_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictThesaurus *d;
char *subdictname = NULL;
bool fileloaded = false;
@@ -612,7 +612,7 @@ thesaurus_init(PG_FUNCTION_ARGS)
d = (DictThesaurus *) palloc0(sizeof(DictThesaurus));
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/utils/cache/ts_cache.c b/src/backend/utils/cache/ts_cache.c
index 0545efc75b..8bc8d82c76 100644
--- a/src/backend/utils/cache/ts_cache.c
+++ b/src/backend/utils/cache/ts_cache.c
@@ -39,6 +39,7 @@
#include "catalog/pg_ts_template.h"
#include "commands/defrem.h"
#include "tsearch/ts_cache.h"
+#include "tsearch/ts_public.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
#include "utils/fmgroids.h"
@@ -311,11 +312,15 @@ lookup_ts_dictionary_cache(Oid dictId)
MemSet(entry, 0, sizeof(TSDictionaryCacheEntry));
entry->dictId = dictId;
entry->dictCtx = saveCtx;
+ entry->dict_xmin = HeapTupleHeaderGetRawXmin(tpdict->t_data);
+ entry->dict_xmax = HeapTupleHeaderGetRawXmax(tpdict->t_data);
+ entry->dict_tid = tpdict->t_self;
entry->lexizeOid = template->tmpllexize;
if (OidIsValid(template->tmplinit))
{
+ DictInitData init_data;
List *dictoptions;
Datum opt;
bool isnull;
@@ -335,9 +340,15 @@ lookup_ts_dictionary_cache(Oid dictId)
else
dictoptions = deserialize_deflist(opt);
+ init_data.dict_options = dictoptions;
+ init_data.dict.id = dictId;
+ init_data.dict.xmin = entry->dict_xmin;
+ init_data.dict.xmax = entry->dict_xmax;
+ init_data.dict.tid = entry->dict_tid;
+
entry->dictData =
DatumGetPointer(OidFunctionCall1(template->tmplinit,
- PointerGetDatum(dictoptions)));
+ PointerGetDatum(&init_data)));
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/include/tsearch/ts_cache.h b/src/include/tsearch/ts_cache.h
index 77e325d101..2298e0a275 100644
--- a/src/include/tsearch/ts_cache.h
+++ b/src/include/tsearch/ts_cache.h
@@ -54,6 +54,10 @@ typedef struct TSDictionaryCacheEntry
Oid dictId;
bool isvalid;
+ TransactionId dict_xmin; /* XMIN of the dictionary's tuple */
+ TransactionId dict_xmax; /* XMAX of the dictionary's tuple */
+ ItemPointerData dict_tid; /* TID of the dictionary's tuple */
+
/* most frequent fmgr call */
Oid lexizeOid;
FmgrInfo lexize;
diff --git a/src/include/tsearch/ts_public.h b/src/include/tsearch/ts_public.h
index b325fa122c..db028ed6ad 100644
--- a/src/include/tsearch/ts_public.h
+++ b/src/include/tsearch/ts_public.h
@@ -13,6 +13,8 @@
#ifndef _PG_TS_PUBLIC_H_
#define _PG_TS_PUBLIC_H_
+#include "nodes/pg_list.h"
+#include "storage/itemptr.h"
#include "tsearch/ts_type.h"
/*
@@ -81,10 +83,69 @@ extern void readstoplist(const char *fname, StopList *s,
extern bool searchstoplist(StopList *s, char *key);
/*
- * Interface with dictionaries
+ * API for text search dictionaries.
+ *
+ * API functions to manage a text search dictionary are defined by a text search
+ * template. Currently an existing template cannot be altered in order to use
+ * different functions. API consists of the following functions:
+ *
+ * init function
+ * -------------
+ * - optional function which initializes internal structures of the dictionary
+ * - accepts DictInitData structure as an argument and must return a custom
+ * palloc'd structure which stores content of the processed dictionary and
+ * is used by lexize function
+ *
+ * lexize function
+ * ---------------
+ * - normalizes a single word (token) using specific dictionary
+ * - returns a palloc'd array of TSLexeme, with a terminating NULL entry
+ * - accepts the following arguments:
+ *
+ * - dictData - pointer to a structure returned by init function or NULL if
+ * init function wasn't defined by the template
+ * - token - string to normalize (not null-terminated)
+ * - length - length of the token
+ * - dictState - pointer to a DictSubState structure storing current
+ * state of a set of tokens processing and allows to normalize phrases
+ */
+
+/*
+ * A preprocessed dictionary can be stored in shared memory using DSM - this is
+ * decided in the init function. A DSM segment is released after altering or
+ * dropping the dictionary. The segment may still leak, when a backend uses the
+ * dictionary right before dropping - in that case the backend will hold the DSM
+ * untill it disconnects or calls lookup_ts_dictionary_cache().
+ *
+ * DictEntryData represents DSM segment with a preprocessed dictionary. We need
+ * to ensure the content of the DSM segment is still valid, which is what xmin,
+ * xmax and tid are for.
+ */
+typedef struct
+{
+ Oid id; /* OID of the dictionary */
+ TransactionId xmin; /* XMIN of the dictionary's tuple */
+ TransactionId xmax; /* XMAX of the dictionary's tuple */
+ ItemPointerData tid; /* TID of the dictionary's tuple */
+} DictEntryData;
+
+/*
+ * API structure for a dictionary initialization. It is passed as an argument
+ * to a template's init function.
*/
+typedef struct
+{
+ /* List of options for a template's init method */
+ List *dict_options;
+
+ /* Data used to allocate, search and release the DSM segment */
+ DictEntryData dict;
+} DictInitData;
-/* return struct for any lexize function */
+/*
+ * Return struct for any lexize function. They are combined into an array, the
+ * last entry is the terminating entry.
+ */
typedef struct
{
/*----------
@@ -108,7 +169,7 @@ typedef struct
uint16 flags; /* See flag bits below */
- char *lexeme; /* C string */
+ char *lexeme; /* C string (NULL for terminating entry) */
} TSLexeme;
/* Flag bits that can appear in TSLexeme.flags */
--
2.21.0
--------------C3497FBBD55C428FF91561C0
Content-Type: text/x-patch;
name="0003-Retrieve-shared-location-for-dict-v19.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0003-Retrieve-shared-location-for-dict-v19.patch"
^ permalink raw reply [nested|flat] 37+ messages in thread
* [PATCH 2/4] Change-tmplinit-argument
@ 2019-01-17 12:05 Arthur Zakirov <[email protected]>
0 siblings, 0 replies; 37+ messages in thread
From: Arthur Zakirov @ 2019-01-17 12:05 UTC (permalink / raw)
---
contrib/dict_int/dict_int.c | 4 +-
contrib/dict_xsyn/dict_xsyn.c | 4 +-
contrib/unaccent/unaccent.c | 4 +-
src/backend/commands/tsearchcmds.c | 10 ++++-
src/backend/snowball/dict_snowball.c | 4 +-
src/backend/tsearch/dict_ispell.c | 4 +-
src/backend/tsearch/dict_simple.c | 4 +-
src/backend/tsearch/dict_synonym.c | 4 +-
src/backend/tsearch/dict_thesaurus.c | 4 +-
src/backend/utils/cache/ts_cache.c | 13 +++++-
src/include/tsearch/ts_cache.h | 4 ++
src/include/tsearch/ts_public.h | 67 ++++++++++++++++++++++++++--
12 files changed, 105 insertions(+), 21 deletions(-)
diff --git a/contrib/dict_int/dict_int.c b/contrib/dict_int/dict_int.c
index 628b9769c3..ddde55eee4 100644
--- a/contrib/dict_int/dict_int.c
+++ b/contrib/dict_int/dict_int.c
@@ -30,7 +30,7 @@ PG_FUNCTION_INFO_V1(dintdict_lexize);
Datum
dintdict_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictInt *d;
ListCell *l;
@@ -38,7 +38,7 @@ dintdict_init(PG_FUNCTION_ARGS)
d->maxlen = 6;
d->rejectlong = false;
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/contrib/dict_xsyn/dict_xsyn.c b/contrib/dict_xsyn/dict_xsyn.c
index 509e14aee0..15b1a0033a 100644
--- a/contrib/dict_xsyn/dict_xsyn.c
+++ b/contrib/dict_xsyn/dict_xsyn.c
@@ -140,7 +140,7 @@ read_dictionary(DictSyn *d, const char *filename)
Datum
dxsyn_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictSyn *d;
ListCell *l;
char *filename = NULL;
@@ -153,7 +153,7 @@ dxsyn_init(PG_FUNCTION_ARGS)
d->matchsynonyms = false;
d->keepsynonyms = true;
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/contrib/unaccent/unaccent.c b/contrib/unaccent/unaccent.c
index fc5176e338..f3663cefd0 100644
--- a/contrib/unaccent/unaccent.c
+++ b/contrib/unaccent/unaccent.c
@@ -270,12 +270,12 @@ PG_FUNCTION_INFO_V1(unaccent_init);
Datum
unaccent_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
TrieChar *rootTrie = NULL;
bool fileloaded = false;
ListCell *l;
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/commands/tsearchcmds.c b/src/backend/commands/tsearchcmds.c
index 8e5eec22b5..30c5eb72a2 100644
--- a/src/backend/commands/tsearchcmds.c
+++ b/src/backend/commands/tsearchcmds.c
@@ -389,17 +389,25 @@ verify_dictoptions(Oid tmplId, List *dictoptions)
}
else
{
+ DictInitData init_data;
+
/*
* Copy the options just in case init method thinks it can scribble on
* them ...
*/
dictoptions = copyObject(dictoptions);
+ init_data.dict_options = dictoptions;
+ init_data.dict.id = InvalidOid;
+ init_data.dict.xmin = InvalidTransactionId;
+ init_data.dict.xmax = InvalidTransactionId;
+ ItemPointerSetInvalid(&init_data.dict.tid);
+
/*
* Call the init method and see if it complains. We don't worry about
* it leaking memory, since our command will soon be over anyway.
*/
- (void) OidFunctionCall1(initmethod, PointerGetDatum(dictoptions));
+ (void) OidFunctionCall1(initmethod, PointerGetDatum(&init_data));
}
ReleaseSysCache(tup);
diff --git a/src/backend/snowball/dict_snowball.c b/src/backend/snowball/dict_snowball.c
index 5166738310..f30f29865c 100644
--- a/src/backend/snowball/dict_snowball.c
+++ b/src/backend/snowball/dict_snowball.c
@@ -201,14 +201,14 @@ locate_stem_module(DictSnowball *d, const char *lang)
Datum
dsnowball_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictSnowball *d;
bool stoploaded = false;
ListCell *l;
d = (DictSnowball *) palloc0(sizeof(DictSnowball));
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/tsearch/dict_ispell.c b/src/backend/tsearch/dict_ispell.c
index 8b05a477f1..fc9a96abca 100644
--- a/src/backend/tsearch/dict_ispell.c
+++ b/src/backend/tsearch/dict_ispell.c
@@ -29,7 +29,7 @@ typedef struct
Datum
dispell_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictISpell *d;
bool affloaded = false,
dictloaded = false,
@@ -40,7 +40,7 @@ dispell_init(PG_FUNCTION_ARGS)
NIStartBuild(&(d->obj));
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/tsearch/dict_simple.c b/src/backend/tsearch/dict_simple.c
index 2f62ef00c8..c92744641b 100644
--- a/src/backend/tsearch/dict_simple.c
+++ b/src/backend/tsearch/dict_simple.c
@@ -29,7 +29,7 @@ typedef struct
Datum
dsimple_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictSimple *d = (DictSimple *) palloc0(sizeof(DictSimple));
bool stoploaded = false,
acceptloaded = false;
@@ -37,7 +37,7 @@ dsimple_init(PG_FUNCTION_ARGS)
d->accept = true; /* default */
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/tsearch/dict_synonym.c b/src/backend/tsearch/dict_synonym.c
index b6226df940..d3f5f0da3f 100644
--- a/src/backend/tsearch/dict_synonym.c
+++ b/src/backend/tsearch/dict_synonym.c
@@ -91,7 +91,7 @@ compareSyn(const void *a, const void *b)
Datum
dsynonym_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictSyn *d;
ListCell *l;
char *filename = NULL;
@@ -104,7 +104,7 @@ dsynonym_init(PG_FUNCTION_ARGS)
char *line = NULL;
uint16 flags = 0;
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/tsearch/dict_thesaurus.c b/src/backend/tsearch/dict_thesaurus.c
index 75f8deef6a..8962e252e0 100644
--- a/src/backend/tsearch/dict_thesaurus.c
+++ b/src/backend/tsearch/dict_thesaurus.c
@@ -604,7 +604,7 @@ compileTheSubstitute(DictThesaurus *d)
Datum
thesaurus_init(PG_FUNCTION_ARGS)
{
- List *dictoptions = (List *) PG_GETARG_POINTER(0);
+ DictInitData *init_data = (DictInitData *) PG_GETARG_POINTER(0);
DictThesaurus *d;
char *subdictname = NULL;
bool fileloaded = false;
@@ -612,7 +612,7 @@ thesaurus_init(PG_FUNCTION_ARGS)
d = (DictThesaurus *) palloc0(sizeof(DictThesaurus));
- foreach(l, dictoptions)
+ foreach(l, init_data->dict_options)
{
DefElem *defel = (DefElem *) lfirst(l);
diff --git a/src/backend/utils/cache/ts_cache.c b/src/backend/utils/cache/ts_cache.c
index 0545efc75b..8bc8d82c76 100644
--- a/src/backend/utils/cache/ts_cache.c
+++ b/src/backend/utils/cache/ts_cache.c
@@ -39,6 +39,7 @@
#include "catalog/pg_ts_template.h"
#include "commands/defrem.h"
#include "tsearch/ts_cache.h"
+#include "tsearch/ts_public.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
#include "utils/fmgroids.h"
@@ -311,11 +312,15 @@ lookup_ts_dictionary_cache(Oid dictId)
MemSet(entry, 0, sizeof(TSDictionaryCacheEntry));
entry->dictId = dictId;
entry->dictCtx = saveCtx;
+ entry->dict_xmin = HeapTupleHeaderGetRawXmin(tpdict->t_data);
+ entry->dict_xmax = HeapTupleHeaderGetRawXmax(tpdict->t_data);
+ entry->dict_tid = tpdict->t_self;
entry->lexizeOid = template->tmpllexize;
if (OidIsValid(template->tmplinit))
{
+ DictInitData init_data;
List *dictoptions;
Datum opt;
bool isnull;
@@ -335,9 +340,15 @@ lookup_ts_dictionary_cache(Oid dictId)
else
dictoptions = deserialize_deflist(opt);
+ init_data.dict_options = dictoptions;
+ init_data.dict.id = dictId;
+ init_data.dict.xmin = entry->dict_xmin;
+ init_data.dict.xmax = entry->dict_xmax;
+ init_data.dict.tid = entry->dict_tid;
+
entry->dictData =
DatumGetPointer(OidFunctionCall1(template->tmplinit,
- PointerGetDatum(dictoptions)));
+ PointerGetDatum(&init_data)));
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/include/tsearch/ts_cache.h b/src/include/tsearch/ts_cache.h
index 77e325d101..2298e0a275 100644
--- a/src/include/tsearch/ts_cache.h
+++ b/src/include/tsearch/ts_cache.h
@@ -54,6 +54,10 @@ typedef struct TSDictionaryCacheEntry
Oid dictId;
bool isvalid;
+ TransactionId dict_xmin; /* XMIN of the dictionary's tuple */
+ TransactionId dict_xmax; /* XMAX of the dictionary's tuple */
+ ItemPointerData dict_tid; /* TID of the dictionary's tuple */
+
/* most frequent fmgr call */
Oid lexizeOid;
FmgrInfo lexize;
diff --git a/src/include/tsearch/ts_public.h b/src/include/tsearch/ts_public.h
index b325fa122c..db028ed6ad 100644
--- a/src/include/tsearch/ts_public.h
+++ b/src/include/tsearch/ts_public.h
@@ -13,6 +13,8 @@
#ifndef _PG_TS_PUBLIC_H_
#define _PG_TS_PUBLIC_H_
+#include "nodes/pg_list.h"
+#include "storage/itemptr.h"
#include "tsearch/ts_type.h"
/*
@@ -81,10 +83,69 @@ extern void readstoplist(const char *fname, StopList *s,
extern bool searchstoplist(StopList *s, char *key);
/*
- * Interface with dictionaries
+ * API for text search dictionaries.
+ *
+ * API functions to manage a text search dictionary are defined by a text search
+ * template. Currently an existing template cannot be altered in order to use
+ * different functions. API consists of the following functions:
+ *
+ * init function
+ * -------------
+ * - optional function which initializes internal structures of the dictionary
+ * - accepts DictInitData structure as an argument and must return a custom
+ * palloc'd structure which stores content of the processed dictionary and
+ * is used by lexize function
+ *
+ * lexize function
+ * ---------------
+ * - normalizes a single word (token) using specific dictionary
+ * - returns a palloc'd array of TSLexeme, with a terminating NULL entry
+ * - accepts the following arguments:
+ *
+ * - dictData - pointer to a structure returned by init function or NULL if
+ * init function wasn't defined by the template
+ * - token - string to normalize (not null-terminated)
+ * - length - length of the token
+ * - dictState - pointer to a DictSubState structure storing current
+ * state of a set of tokens processing and allows to normalize phrases
+ */
+
+/*
+ * A preprocessed dictionary can be stored in shared memory using DSM - this is
+ * decided in the init function. A DSM segment is released after altering or
+ * dropping the dictionary. The segment may still leak, when a backend uses the
+ * dictionary right before dropping - in that case the backend will hold the DSM
+ * untill it disconnects or calls lookup_ts_dictionary_cache().
+ *
+ * DictEntryData represents DSM segment with a preprocessed dictionary. We need
+ * to ensure the content of the DSM segment is still valid, which is what xmin,
+ * xmax and tid are for.
+ */
+typedef struct
+{
+ Oid id; /* OID of the dictionary */
+ TransactionId xmin; /* XMIN of the dictionary's tuple */
+ TransactionId xmax; /* XMAX of the dictionary's tuple */
+ ItemPointerData tid; /* TID of the dictionary's tuple */
+} DictEntryData;
+
+/*
+ * API structure for a dictionary initialization. It is passed as an argument
+ * to a template's init function.
*/
+typedef struct
+{
+ /* List of options for a template's init method */
+ List *dict_options;
+
+ /* Data used to allocate, search and release the DSM segment */
+ DictEntryData dict;
+} DictInitData;
-/* return struct for any lexize function */
+/*
+ * Return struct for any lexize function. They are combined into an array, the
+ * last entry is the terminating entry.
+ */
typedef struct
{
/*----------
@@ -108,7 +169,7 @@ typedef struct
uint16 flags; /* See flag bits below */
- char *lexeme; /* C string */
+ char *lexeme; /* C string (NULL for terminating entry) */
} TSLexeme;
/* Flag bits that can appear in TSLexeme.flags */
--
2.20.1
--------------9E285E8AF9A980CD1515771F
Content-Type: text/x-patch;
name="0003-Retrieve-shared-location-for-dict-v18.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0003-Retrieve-shared-location-for-dict-v18.patch"
^ permalink raw reply [nested|flat] 37+ messages in thread
* remap the .text segment into huge pages at run time
@ 2022-11-02 06:32 John Naylor <[email protected]>
2022-11-03 17:21 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: John Naylor @ 2022-11-02 06:32 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Andres Freund <[email protected]>; Thomas Munro <[email protected]>
It's been known for a while that Postgres spends a lot of time translating
instruction addresses, and using huge pages in the text segment yields a
substantial performance boost in OLTP workloads [1][2]. The difficulty is,
this normally requires a lot of painstaking work (unless your OS does
superpage promotion, like FreeBSD).
I found an MIT-licensed library "iodlr" from Intel [3] that allows one to
remap the .text segment to huge pages at program start. Attached is a
hackish, Meson-only, "works on my machine" patchset to experiment with this
idea.
0001 adapts the library to our error logging and GUC system. The overview:
- read ELF info to get the start/end addresses of the .text segment
- calculate addresses therein aligned at huge page boundaries
- mmap a temporary region and memcpy the aligned portion of the .text
segment
- mmap aligned start address to a second region with huge pages and
MAP_FIXED
- memcpy over from the temp region and revoke the PROT_WRITE bit
The reason this doesn't "saw off the branch you're standing on" is that the
remapping is done in a function that's forced to live in a different
segment, and doesn't call any non-libc functions living elsewhere:
static void
__attribute__((__section__("lpstub")))
__attribute__((__noinline__))
MoveRegionToLargePages(const mem_range * r, int mmap_flags)
Debug messages show
2022-11-02 12:02:31.064 +07 [26955] DEBUG: .text start: 0x487540
2022-11-02 12:02:31.064 +07 [26955] DEBUG: .text end: 0x96cf12
2022-11-02 12:02:31.064 +07 [26955] DEBUG: aligned .text start: 0x600000
2022-11-02 12:02:31.064 +07 [26955] DEBUG: aligned .text end: 0x800000
2022-11-02 12:02:31.066 +07 [26955] DEBUG: binary mapped to huge pages
2022-11-02 12:02:31.066 +07 [26955] DEBUG: un-mmapping temporary code
region
Here, out of 5MB of Postgres text, only 1 huge page can be used, but that
still saves 512 entries in the TLB and might bring a small improvement. The
un-remapped region below 0x600000 contains the ~600kB of "cold" code, since
the linker puts the cold section first, at least recent versions of ld and
lld.
0002 is my attempt to force the linker's hand and get the entire text
segment mapped to huge pages. It's quite a finicky hack, and easily broken
(see below). That said, it still builds easily within our normal build
process, and maybe there is a better way to get the effect.
It does two things:
- Pass the linker -Wl,-zcommon-page-size=2097152
-Wl,-zmax-page-size=2097152 which aligns .init to a 2MB boundary. That's
done for predictability, but that means the next 2MB boundary is very
nearly 2MB away.
- Add a "cold" __asm__ filler function that just takes up space, enough to
push the end of the .text segment over the next aligned boundary, or to
~8MB in size.
In a non-assert build:
0001:
$ bloaty inst-perf/bin/postgres
FILE SIZE VM SIZE
-------------- --------------
53.7% 4.90Mi 58.7% 4.90Mi .text
...
100.0% 9.12Mi 100.0% 8.35Mi TOTAL
$ readelf -S --wide inst-perf/bin/postgres
[Nr] Name Type Address Off Size ES
Flg Lk Inf Al
...
[12] .init PROGBITS 0000000000486000 086000 00001b 00
AX 0 0 4
[13] .plt PROGBITS 0000000000486020 086020 001520 10
AX 0 0 16
[14] .text PROGBITS 0000000000487540 087540 4e59d2 00
AX 0 0 16
...
0002:
$ bloaty inst-perf/bin/postgres
FILE SIZE VM SIZE
-------------- --------------
46.9% 8.00Mi 69.9% 8.00Mi .text
...
100.0% 17.1Mi 100.0% 11.4Mi TOTAL
$ readelf -S --wide inst-perf/bin/postgres
[Nr] Name Type Address Off Size ES
Flg Lk Inf Al
...
[12] .init PROGBITS 0000000000600000 200000 00001b 00
AX 0 0 4
[13] .plt PROGBITS 0000000000600020 200020 001520 10
AX 0 0 16
[14] .text PROGBITS 0000000000601540 201540 7ff512 00
AX 0 0 16
...
Debug messages with 0002 shows 6MB mapped:
2022-11-02 12:35:28.482 +07 [28530] DEBUG: .text start: 0x601540
2022-11-02 12:35:28.482 +07 [28530] DEBUG: .text end: 0xe00a52
2022-11-02 12:35:28.482 +07 [28530] DEBUG: aligned .text start: 0x800000
2022-11-02 12:35:28.482 +07 [28530] DEBUG: aligned .text end: 0xe00000
2022-11-02 12:35:28.486 +07 [28530] DEBUG: binary mapped to huge pages
2022-11-02 12:35:28.486 +07 [28530] DEBUG: un-mmapping temporary code
region
Since the front is all-cold, and there is very little at the end,
practically all hot pages are now remapped. The biggest problem with the
hackish filler function (in addition to maintainability) is, if explicit
huge pages are turned off in the kernel, attempting mmap() with MAP_HUGETLB
causes complete startup failure if the .text segment is larger than 8MB. I
haven't looked into what's happening there yet, but I didn't want to get
too far in the weeds before getting feedback on whether the entire approach
in this thread is sound enough to justify working further on.
[1] https://www.cs.rochester.edu/u/sandhya/papers/ispass19.pdf
(paper: "On the Impact of Instruction Address Translation Overhead")
[2] https://twitter.com/AndresFreundTec/status/1214305610172289024
[3] https://github.com/intel/iodlr
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[application/x-patch] v1-0002-Put-all-non-cold-.text-in-huge-pages.patch (3.1K, ../../CAFBsxsHx9z45MfsAjELFiPv_kcgCcH_P5jNa=WaeGxO7HU3mag@mail.gmail.com/3-v1-0002-Put-all-non-cold-.text-in-huge-pages.patch)
download | inline diff:
From 9cde401f87937c1982f2355c8f81449514166376 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Mon, 31 Oct 2022 13:59:30 +0700
Subject: [PATCH v1 2/2] Put all non-cold .text in huge pages
Tell linker to align addresses on 2MB boundaries. The .init
section will be so aligned, with the .text section soon after that.
Therefore, the start address of .text should always be align up to
nearly 2MB ahead of the actual start. The first nearly 2MB of .text
will not map to huge pages.
We count on cold sections linking to the front of the .text segment:
Since the cold sections total about 600kB in size, we need ~1.4MB of
additional padding to keep non-cold pages mappable to huge pages. Since
PG has about 5.0MB of .text, we also need an additional 1MB to push
the .text end just past an aligned boundary, so when we align the end
down, only a small number of pages will remain un-remapped at their
original 4kB size.
---
meson.build | 3 +++
src/backend/port/filler.c | 29 +++++++++++++++++++++++++++++
src/backend/port/meson.build | 3 +++
3 files changed, 35 insertions(+)
create mode 100644 src/backend/port/filler.c
diff --git a/meson.build b/meson.build
index bfacbdc0af..450946370c 100644
--- a/meson.build
+++ b/meson.build
@@ -239,6 +239,9 @@ elif host_system == 'freebsd'
elif host_system == 'linux'
sema_kind = 'unnamed_posix'
cppflags += '-D_GNU_SOURCE'
+ # WIP: debug builds are huge
+ # TODO: add portability check
+ ldflags += ['-Wl,-zcommon-page-size=2097152', '-Wl,-zmax-page-size=2097152']
elif host_system == 'netbsd'
# We must resolve all dynamic linking in the core server at program start.
diff --git a/src/backend/port/filler.c b/src/backend/port/filler.c
new file mode 100644
index 0000000000..de4e33bb05
--- /dev/null
+++ b/src/backend/port/filler.c
@@ -0,0 +1,29 @@
+/*
+ * Add enough padding to .text segment to bring the end just
+ * past a 2MB alignment boundary. In practice, this means .text needs
+ * to be at least 8MB. It shouldn't be much larger than this,
+ * because then more hot pages will remain in 4kB pages.
+ *
+ * FIXME: With this filler added, if explicit huge pages are turned off
+ * in the kernel, attempting mmap() with MAP_HUGETLB causes a crash
+ * instead of reporting failure if the .text segment is larger than 8MB.
+ *
+ * See MapStaticCodeToLargePages() in large_page.c
+ *
+ * XXX: The exact amount of filler must be determined experimentally
+ * on platforms of interest, in non-assert builds.
+ *
+ */
+static void
+__attribute__((used))
+__attribute__((cold))
+fill_function(int x)
+{
+ /* TODO: More architectures */
+#ifdef __x86_64__
+__asm__(
+ ".fill 3251000"
+);
+#endif
+ (void) x;
+}
\ No newline at end of file
diff --git a/src/backend/port/meson.build b/src/backend/port/meson.build
index 5ab65115e9..d876712e0c 100644
--- a/src/backend/port/meson.build
+++ b/src/backend/port/meson.build
@@ -16,6 +16,9 @@ if cdata.has('USE_WIN32_SEMAPHORES')
endif
if cdata.has('USE_SYSV_SHARED_MEMORY')
+ if host_system == 'linux'
+ backend_sources += files('filler.c')
+ endif
backend_sources += files('large_page.c')
backend_sources += files('sysv_shmem.c')
endif
--
2.37.3
[application/x-patch] v1-0001-Partly-remap-the-.text-segment-into-huge-pages-at.patch (12.7K, ../../CAFBsxsHx9z45MfsAjELFiPv_kcgCcH_P5jNa=WaeGxO7HU3mag@mail.gmail.com/4-v1-0001-Partly-remap-the-.text-segment-into-huge-pages-at.patch)
download | inline diff:
From 0012baab70779f5fc06c8717392dc76e8f156270 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Mon, 31 Oct 2022 15:24:29 +0700
Subject: [PATCH v1 1/2] Partly remap the .text segment into huge pages at
postmaster start
Based on MIT licensed libary at https://github.com/intel/iodlr
The basic steps are:
- read ELF info to get the start/end addresses of the .text segment
- calculate addresses therein aligned at huge page boundaries
- mmap temporary region and memcpy aligned portion of .text segment
- mmap start address to new region with huge pages and MAP_FIXED
- memcpy over and revoke the PROT_WRITE bit
The Postgres .text segment is ~5.0MB in a non-assert build, so this
method can put 2-4MB into huge pages.
---
src/backend/port/large_page.c | 348 ++++++++++++++++++++++++++++
src/backend/port/meson.build | 1 +
src/backend/postmaster/postmaster.c | 7 +
src/include/port/large_page.h | 18 ++
4 files changed, 374 insertions(+)
create mode 100644 src/backend/port/large_page.c
create mode 100644 src/include/port/large_page.h
diff --git a/src/backend/port/large_page.c b/src/backend/port/large_page.c
new file mode 100644
index 0000000000..66a584f785
--- /dev/null
+++ b/src/backend/port/large_page.c
@@ -0,0 +1,348 @@
+/*-------------------------------------------------------------------------
+ *
+ * large_page.c
+ * Map .text segment of binary to huge pages
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/port/large_page.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+/*
+ * Based on Intel ioldr library:
+ * https://github.com/intel/iodlr.git
+ * MIT license and copyright notice follows
+ */
+
+/*
+ * Copyright (C) 2018 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom
+ * the Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included
+ * in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+ * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES
+ * OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
+ * OR OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * SPDX-License-Identifier: MIT
+ */
+
+#include "postgres.h"
+
+#include <link.h>
+#include <sys/mman.h>
+
+#include "port/large_page.h"
+#include "storage/pg_shmem.h"
+
+typedef struct
+{
+ char *from;
+ char *to;
+} mem_range;
+
+typedef struct
+{
+ uintptr_t start;
+ uintptr_t end;
+ bool found;
+} FindParams;
+
+static inline uintptr_t
+largepage_align_down(uintptr_t addr, size_t hugepagesize)
+{
+ return (addr & ~(hugepagesize - 1));
+}
+
+static inline uintptr_t
+largepage_align_up(uintptr_t addr, size_t hugepagesize)
+{
+ return largepage_align_down(addr + hugepagesize - 1, hugepagesize);
+}
+
+static bool
+FindTextSection(const char *fname, ElfW(Shdr) * text_section)
+{
+ ElfW(Ehdr) ehdr;
+ FILE *bin;
+
+ ElfW(Shdr) * shdrs = NULL;
+ ElfW(Shdr) * sh_strab;
+ char *section_names = NULL;
+
+ bin = fopen(fname, "r");
+ if (bin == NULL)
+ return false;
+
+ /* Read the header. */
+ if (fread(&ehdr, sizeof(ehdr), 1, bin) != 1)
+ return false;;
+
+ /* Read the section headers. */
+ shdrs = (ElfW(Shdr) *) palloc(ehdr.e_shnum * sizeof(ElfW(Shdr)));
+ if (fseek(bin, ehdr.e_shoff, SEEK_SET) != 0)
+ return false;;
+ if (fread(shdrs, sizeof(shdrs[0]), ehdr.e_shnum, bin) != ehdr.e_shnum)
+ return false;;
+
+ /* Read the string table. */
+ sh_strab = &shdrs[ehdr.e_shstrndx];
+ section_names = palloc(sh_strab->sh_size * sizeof(char));
+
+ if (fseek(bin, sh_strab->sh_offset, SEEK_SET) != 0)
+ return false;;
+ if (fread(section_names, sh_strab->sh_size, 1, bin) != 1)
+ return false;;
+
+ /* Find the ".text" section. */
+ for (uint32_t idx = 0; idx < ehdr.e_shnum; idx++)
+ {
+ ElfW(Shdr) * sh = &shdrs[idx];
+ if (!memcmp(§ion_names[sh->sh_name], ".text", 5))
+ {
+ *text_section = *sh;
+ fclose(bin);
+ return true;
+ }
+ }
+ return false;
+}
+
+/* Callback for dl_iterate_phdr to set the start and end of the .text segment */
+static int
+FindMapping(struct dl_phdr_info *hdr, size_t size, void *data)
+{
+ ElfW(Shdr) text_section;
+ FindParams *find_params = (FindParams *) data;
+
+ /*
+ * We are only interested in the mapping matching the main executable.
+ * This has the empty string for a name.
+ */
+ if (hdr->dlpi_name[0] != '\0')
+ return 0;
+
+ /*
+ * Open the info structure for the executable on disk to find the location
+ * of its .text section. We use the base address given to calculate the
+ * .text section offset in memory.
+ */
+ text_section.sh_size = 0;
+#ifdef __linux__
+ if (FindTextSection("/proc/self/exe", &text_section))
+ {
+ find_params->start = hdr->dlpi_addr + text_section.sh_addr;
+ find_params->end = find_params->start + text_section.sh_size;
+ find_params->found = true;
+ return 1;
+ }
+#endif
+ return 0;
+}
+
+/*
+ * Identify and return the text segment in the currently mapped memory region.
+ */
+static bool
+FindTextRegion(mem_range * region)
+{
+ FindParams find_params = {0, 0, false};
+
+ /*
+ * Note: the upstream source worked with shared libraries as well, hence
+ * the iteration over all ojects.
+ */
+ dl_iterate_phdr(FindMapping, &find_params);
+ if (find_params.found)
+ {
+ region->from = (char *) find_params.start;
+ region->to = (char *) find_params.end;
+ }
+
+ return find_params.found;
+}
+
+/*
+ * Move specified region to large pages.
+ *
+ * NB: We need to be very careful:
+ * 1. This function itself should not be moved. We use compiler attributes:
+ * WIP: if these aren't available, the function should do nothing
+ * (__section__) to put it outside the ".text" section
+ * (__noline__) to not inline this function
+ *
+ * 2. This function should not call any function(s) that might be moved.
+ */
+static void
+__attribute__((__section__("lpstub")))
+__attribute__((__noinline__))
+MoveRegionToLargePages(const mem_range * r, int mmap_flags)
+{
+ void *nmem = MAP_FAILED;
+ void *tmem = MAP_FAILED;
+ int ret = 0;
+ int mmap_errno = 0;
+ void *start = r->from;
+ size_t size = r->to - r->from;
+ bool success = false;
+
+ /* Allocate temporary region */
+ nmem = mmap(NULL, size,
+ PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (nmem == MAP_FAILED)
+ {
+ elog(DEBUG1, "failed to allocate temporary region");
+ return;
+ }
+
+ /* copy the original code */
+ memcpy(nmem, r->from, size);
+
+ /*
+ * mmap using the start address with MAP_FIXED so we get exactly the same
+ * virtual address. We already know the original page is r-xp (PROT_READ,
+ * PROT_EXEC, MAP_PRIVATE) We want PROT_WRITE because we are writing into
+ * it.
+ */
+ Assert(mmap_flags & MAP_HUGETLB);
+ tmem = mmap(start, size,
+ PROT_READ | PROT_WRITE | PROT_EXEC,
+ MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED | mmap_flags,
+ -1, 0);
+ mmap_errno = errno;
+
+ if (tmem == MAP_FAILED && huge_pages == HUGE_PAGES_ON)
+ {
+ /*
+ * WIP: need a way for the user to determine total huge pages needed,
+ * perhaps with shared_memory_size_in_huge_pages
+ */
+ errno = mmap_errno;
+ ereport(FATAL,
+ errmsg("mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m", size),
+ (mmap_errno == ENOMEM) ?
+ errhint("This usually means not enough explicit huge pages were "
+ "configured in the kernel") : 0);
+ goto cleanup_tmp;
+ }
+ else if (tmem == MAP_FAILED)
+ {
+ Assert(huge_pages == HUGE_PAGES_TRY);
+
+ errno = mmap_errno;
+ elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m", size);
+
+ /*
+ * try remapping again with normal pages
+ *
+ * XXX we cannot just back out now
+ */
+ tmem = mmap(start, size,
+ PROT_READ | PROT_WRITE | PROT_EXEC,
+ MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED,
+ -1, 0);
+ mmap_errno = errno;
+
+ if (tmem == MAP_FAILED)
+ {
+ /*
+ * If we get here we cannot start the server. It's unlikely we
+ * will fail here after the postmaster successfully set up shared
+ * memory, but maybe we should have a GUC to turn off code
+ * remapping, hinted here.
+ */
+ errno = mmap_errno;
+ ereport(FATAL,
+ errmsg("mmap(%zu) failed for fallback code region: %m", size));
+ goto cleanup_tmp;
+ }
+ }
+ else
+ success = true;
+
+ /* copy the code to the newly mapped area and unset the write bit */
+ memcpy(start, nmem, size);
+ ret = mprotect(start, size, PROT_READ | PROT_EXEC);
+ if (ret < 0)
+ {
+ /* WIP: see note above about GUC and hint */
+ ereport(FATAL,
+ errmsg("failed to protect remapped code pages"));
+
+ /* Cannot start but at least try to clean up after ourselves */
+ munmap(tmem, size);
+ goto cleanup_tmp;
+ }
+
+ if (success)
+ elog(DEBUG1, "binary mapped to huge pages");
+
+cleanup_tmp:
+ /* Release the old/temporary mapped region */
+ elog(DEBUG3, "un-mmapping temporary code region");
+ ret = munmap(nmem, size);
+ if (ret < 0)
+ /* WIP: not sure of severity here */
+ ereport(LOG,
+ errmsg("failed to unmap temporary region"));
+
+ return;
+}
+
+/* Align the region to to be mapped to huge page boundaries. */
+static void
+AlignRegionToPageBoundary(mem_range * r, size_t hugepagesize)
+{
+ r->from = (char *) largepage_align_up((uintptr_t) r->from, hugepagesize);
+ r->to = (char *) largepage_align_down((uintptr_t) r->to, hugepagesize);
+}
+
+
+/* Map the postgres .text segment into huge pages. */
+void
+MapStaticCodeToLargePages(void)
+{
+ size_t hugepagesize;
+ int mmap_flags;
+ mem_range r = {0};
+
+ if (huge_pages == HUGE_PAGES_OFF)
+ return;
+
+ GetHugePageSize(&hugepagesize, &mmap_flags);
+ if (hugepagesize == 0)
+ return;
+
+ FindTextRegion(&r);
+ if (r.from == NULL || r.to == NULL)
+ return;
+
+ elog(DEBUG3, ".text start: %p", r.from);
+ elog(DEBUG3, ".text end: %p", r.to);
+
+ AlignRegionToPageBoundary(&r, hugepagesize);
+
+ elog(DEBUG3, "aligned .text start: %p", r.from);
+ elog(DEBUG3, "aligned .text end: %p", r.to);
+
+ /* check if aligned map region is large enough for huge pages */
+ if (r.to - r.from < hugepagesize || r.from > r.to)
+ return;
+
+ MoveRegionToLargePages(&r, mmap_flags);
+}
diff --git a/src/backend/port/meson.build b/src/backend/port/meson.build
index a22c25dd95..5ab65115e9 100644
--- a/src/backend/port/meson.build
+++ b/src/backend/port/meson.build
@@ -16,6 +16,7 @@ if cdata.has('USE_WIN32_SEMAPHORES')
endif
if cdata.has('USE_SYSV_SHARED_MEMORY')
+ backend_sources += files('large_page.c')
backend_sources += files('sysv_shmem.c')
endif
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 30fb576ac3..b30769c2b2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -106,6 +106,7 @@
#include "pg_getopt.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/large_page.h"
#include "postmaster/autovacuum.h"
#include "postmaster/auxprocess.h"
#include "postmaster/bgworker_internals.h"
@@ -1084,6 +1085,12 @@ PostmasterMain(int argc, char *argv[])
*/
CreateSharedMemoryAndSemaphores();
+ /*
+ * If enough huge pages are available after setting up shared memory, try
+ * to map the binary code to huge pages.
+ */
+ MapStaticCodeToLargePages();
+
/*
* Estimate number of openable files. This must happen after setting up
* semaphores, because on some platforms semaphores count as open files.
diff --git a/src/include/port/large_page.h b/src/include/port/large_page.h
new file mode 100644
index 0000000000..171819dd53
--- /dev/null
+++ b/src/include/port/large_page.h
@@ -0,0 +1,18 @@
+/*-------------------------------------------------------------------------
+ *
+ * large_page.h
+ * Map .text segment of binary to huge pages
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/port/large_page.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef LARGE_PAGE_H
+#define LARGE_PAGE_H
+
+extern void MapStaticCodeToLargePages(void);
+
+#endif /* LARGE_PAGE_H */
--
2.37.3
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: remap the .text segment into huge pages at run time
2022-11-02 06:32 remap the .text segment into huge pages at run time John Naylor <[email protected]>
@ 2022-11-03 17:21 ` Andres Freund <[email protected]>
2022-11-04 18:33 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2022-11-04 21:21 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2022-11-05 05:54 ` Re: remap the .text segment into huge pages at run time John Naylor <[email protected]>
0 siblings, 3 replies; 37+ messages in thread
From: Andres Freund @ 2022-11-03 17:21 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Thomas Munro <[email protected]>
Hi,
On 2022-11-02 13:32:37 +0700, John Naylor wrote:
> It's been known for a while that Postgres spends a lot of time translating
> instruction addresses, and using huge pages in the text segment yields a
> substantial performance boost in OLTP workloads [1][2].
Indeed. Some of that we eventually should address by making our code less
"jumpy", but that's a large amount of work and only going to go so far.
> The difficulty is,
> this normally requires a lot of painstaking work (unless your OS does
> superpage promotion, like FreeBSD).
I still am confused by FreeBSD being able to do this without changing the
section alignment to be big enough. Or is the default alignment on FreeBSD
large enough already?
> I found an MIT-licensed library "iodlr" from Intel [3] that allows one to
> remap the .text segment to huge pages at program start. Attached is a
> hackish, Meson-only, "works on my machine" patchset to experiment with this
> idea.
I wonder how far we can get with just using the linker hints to align
sections. I know that the linux folks are working on promoting sufficiently
aligned executable pages to huge pages too, and might have succeeded already.
IOW, adding the linker flags might be a good first step.
> 0001 adapts the library to our error logging and GUC system. The overview:
>
> - read ELF info to get the start/end addresses of the .text segment
> - calculate addresses therein aligned at huge page boundaries
> - mmap a temporary region and memcpy the aligned portion of the .text
> segment
> - mmap aligned start address to a second region with huge pages and
> MAP_FIXED
> - memcpy over from the temp region and revoke the PROT_WRITE bit
Would mremap()'ing the temporary region also work? That might be simpler and
more robust (you'd see the MAP_HUGETLB failure before doing anything
irreversible). And you then might not even need this:
> The reason this doesn't "saw off the branch you're standing on" is that the
> remapping is done in a function that's forced to live in a different
> segment, and doesn't call any non-libc functions living elsewhere:
>
> static void
> __attribute__((__section__("lpstub")))
> __attribute__((__noinline__))
> MoveRegionToLargePages(const mem_range * r, int mmap_flags)
This would likely need a bunch more gating than the patch, understandably,
has. I think it'd faily horribly if there were .text relocations, for example?
I think there are some architectures that do that by default...
> 0002 is my attempt to force the linker's hand and get the entire text
> segment mapped to huge pages. It's quite a finicky hack, and easily broken
> (see below). That said, it still builds easily within our normal build
> process, and maybe there is a better way to get the effect.
>
> It does two things:
>
> - Pass the linker -Wl,-zcommon-page-size=2097152
> -Wl,-zmax-page-size=2097152 which aligns .init to a 2MB boundary. That's
> done for predictability, but that means the next 2MB boundary is very
> nearly 2MB away.
Yep. FWIW, my notes say
# align sections to 2MB boundaries for hugepage support
# bfd and gold linkers:
# -Wl,-zmax-page-size=0x200000 -Wl,-zcommon-page-size=0x200000
# lld:
# -Wl,-zmax-page-size=0x200000 -Wl,-z,separate-loadable-segments
# then copy binary to tmpfs mounted with -o huge=always
I.e. with lld you need slightly different flags -Wl,-z,separate-loadable-segments
The meson bit should probably just use
cc.get_supported_link_arguments([
'-Wl,-zmax-page-size=0x200000',
'-Wl,-zcommon-page-size=0x200000',
'-Wl,-zseparate-loadable-segments'])
Afaict there's really no reason to not do that by default, allowing kernels
that can promote to huge pages to do so.
My approach to forcing huge pages to be used was to then:
# copy binary to tmpfs mounted with -o huge=always
> - Add a "cold" __asm__ filler function that just takes up space, enough to
> push the end of the .text segment over the next aligned boundary, or to
> ~8MB in size.
I don't understand why this is needed - as long as the pages are aligned to
2MB, why do we need to fill things up on disk? The in-memory contents are the
relevant bit, no?
> Since the front is all-cold, and there is very little at the end,
> practically all hot pages are now remapped. The biggest problem with the
> hackish filler function (in addition to maintainability) is, if explicit
> huge pages are turned off in the kernel, attempting mmap() with MAP_HUGETLB
> causes complete startup failure if the .text segment is larger than 8MB.
I would expect MAP_HUGETLB to always fail if not enabled in the kernel,
independent of the .text segment size?
> +/* Callback for dl_iterate_phdr to set the start and end of the .text segment */
> +static int
> +FindMapping(struct dl_phdr_info *hdr, size_t size, void *data)
> +{
> + ElfW(Shdr) text_section;
> + FindParams *find_params = (FindParams *) data;
> +
> + /*
> + * We are only interested in the mapping matching the main executable.
> + * This has the empty string for a name.
> + */
> + if (hdr->dlpi_name[0] != '\0')
> + return 0;
> +
It's not entirely clear we'd only ever want to do this for the main
executable. E.g. plpgsql could also benefit.
> diff --git a/meson.build b/meson.build
> index bfacbdc0af..450946370c 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -239,6 +239,9 @@ elif host_system == 'freebsd'
> elif host_system == 'linux'
> sema_kind = 'unnamed_posix'
> cppflags += '-D_GNU_SOURCE'
> + # WIP: debug builds are huge
> + # TODO: add portability check
> + ldflags += ['-Wl,-zcommon-page-size=2097152', '-Wl,-zmax-page-size=2097152']
What's that WIP about?
> elif host_system == 'netbsd'
> # We must resolve all dynamic linking in the core server at program start.
> diff --git a/src/backend/port/filler.c b/src/backend/port/filler.c
> new file mode 100644
> index 0000000000..de4e33bb05
> --- /dev/null
> +++ b/src/backend/port/filler.c
> @@ -0,0 +1,29 @@
> +/*
> + * Add enough padding to .text segment to bring the end just
> + * past a 2MB alignment boundary. In practice, this means .text needs
> + * to be at least 8MB. It shouldn't be much larger than this,
> + * because then more hot pages will remain in 4kB pages.
> + *
> + * FIXME: With this filler added, if explicit huge pages are turned off
> + * in the kernel, attempting mmap() with MAP_HUGETLB causes a crash
> + * instead of reporting failure if the .text segment is larger than 8MB.
> + *
> + * See MapStaticCodeToLargePages() in large_page.c
> + *
> + * XXX: The exact amount of filler must be determined experimentally
> + * on platforms of interest, in non-assert builds.
> + *
> + */
> +static void
> +__attribute__((used))
> +__attribute__((cold))
> +fill_function(int x)
> +{
> + /* TODO: More architectures */
> +#ifdef __x86_64__
> +__asm__(
> + ".fill 3251000"
> +);
> +#endif
> + (void) x;
> +}
> \ No newline at end of file
> diff --git a/src/backend/port/meson.build b/src/backend/port/meson.build
> index 5ab65115e9..d876712e0c 100644
> --- a/src/backend/port/meson.build
> +++ b/src/backend/port/meson.build
> @@ -16,6 +16,9 @@ if cdata.has('USE_WIN32_SEMAPHORES')
> endif
>
> if cdata.has('USE_SYSV_SHARED_MEMORY')
> + if host_system == 'linux'
> + backend_sources += files('filler.c')
> + endif
> backend_sources += files('large_page.c')
> backend_sources += files('sysv_shmem.c')
> endif
> --
> 2.37.3
>
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: remap the .text segment into huge pages at run time
2022-11-02 06:32 remap the .text segment into huge pages at run time John Naylor <[email protected]>
2022-11-03 17:21 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
@ 2022-11-04 18:33 ` Andres Freund <[email protected]>
2 siblings, 0 replies; 37+ messages in thread
From: Andres Freund @ 2022-11-04 18:33 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Thomas Munro <[email protected]>
Hi,
This nerd-sniped me badly :)
On 2022-11-03 10:21:23 -0700, Andres Freund wrote:
> On 2022-11-02 13:32:37 +0700, John Naylor wrote:
> > I found an MIT-licensed library "iodlr" from Intel [3] that allows one to
> > remap the .text segment to huge pages at program start. Attached is a
> > hackish, Meson-only, "works on my machine" patchset to experiment with this
> > idea.
>
> I wonder how far we can get with just using the linker hints to align
> sections. I know that the linux folks are working on promoting sufficiently
> aligned executable pages to huge pages too, and might have succeeded already.
>
> IOW, adding the linker flags might be a good first step.
Indeed, I did see that that works to some degree on the 5.19 kernel I was
running. However, it never seems to get around to using huge pages
sufficiently to compete with explicit use of huge pages.
More interestingly, a few days ago, a new madvise hint, MADV_COLLAPSE, was
added into linux 6.1. That explicitly remaps a region and uses huge pages for
it. Of course that's going to take a while to be widely available, but it
seems like a safer approach than the remapping approach from this thread.
I hacked in a MADV_COLLAPSE (with setarch -R, so that I could just hardcode
the address / length), and it seems to work nicely.
With the weird caveat that on fs one needs to make sure that the executable
doesn't reflinks to reuse parts of other files, and that the mold linker and
cp do... Not a concern on ext4, but on xfs. I took to copying the postgres
binary with cp --reflink=never
FWIW, you can see the state of the page mapping in more detail with the
kernel's page-types tool
sudo /home/andres/src/kernel/tools/vm/page-types -L -p 12297 -a 0x555555800,0x555556122
sudo /home/andres/src/kernel/tools/vm/page-types -f /srv/dev/build/m-opt/src/backend/postgres2
Perf results:
c=150;psql -f ~/tmp/prewarm.sql;perf stat -a -e cycles,iTLB-loads,iTLB-load-misses,itlb_misses.walk_active,itlb_misses.walk_completed_4k,itlb_misses.walk_completed_2m_4m,itlb_misses.walk_completed_1g pgbench -n -M prepared -S -P1 -c$c -j$c -T10
without MADV_COLLAPSE:
tps = 1038230.070771 (without initial connection time)
Performance counter stats for 'system wide':
1,184,344,476,152 cycles (71.41%)
2,846,146,710 iTLB-loads (71.43%)
2,021,885,782 iTLB-load-misses # 71.04% of all iTLB cache accesses (71.44%)
75,633,850,933 itlb_misses.walk_active (71.44%)
2,020,962,930 itlb_misses.walk_completed_4k (71.44%)
1,213,368 itlb_misses.walk_completed_2m_4m (57.12%)
2,293 itlb_misses.walk_completed_1g (57.11%)
10.064352587 seconds time elapsed
with MADV_COLLAPSE:
tps = 1113717.114278 (without initial connection time)
Performance counter stats for 'system wide':
1,173,049,140,611 cycles (71.42%)
1,059,224,678 iTLB-loads (71.44%)
653,603,712 iTLB-load-misses # 61.71% of all iTLB cache accesses (71.44%)
26,135,902,949 itlb_misses.walk_active (71.44%)
628,314,285 itlb_misses.walk_completed_4k (71.44%)
25,462,916 itlb_misses.walk_completed_2m_4m (57.13%)
2,228 itlb_misses.walk_completed_1g (57.13%)
Note that while the rate of itlb-misses stays roughly the same, the total
number of iTLB loads reduced substantially, and the number of cycles in which
an itlb miss was in progress is 1/3 of what it was before.
A lot of the remaining misses are from the context switches. The iTLB is
flushed on context switches, and of course pgbench -S is extremely context
switch heavy.
Comparing plain -S with 10 pipelined -S transactions (using -t 100000 / -t
10000 to compare the same amount of work) I get:
without MADV_COLLAPSE:
not pipelined:
tps = 1037732.722805 (without initial connection time)
Performance counter stats for 'system wide':
1,691,411,678,007 cycles (62.48%)
8,856,107 itlb.itlb_flush (62.48%)
4,600,041,062 iTLB-loads (62.48%)
2,598,218,236 iTLB-load-misses # 56.48% of all iTLB cache accesses (62.50%)
100,095,862,126 itlb_misses.walk_active (62.53%)
2,595,376,025 itlb_misses.walk_completed_4k (50.02%)
2,558,713 itlb_misses.walk_completed_2m_4m (50.00%)
2,146 itlb_misses.walk_completed_1g (49.98%)
14.582927646 seconds time elapsed
pipelined:
tps = 161947.008995 (without initial connection time)
Performance counter stats for 'system wide':
1,095,948,341,745 cycles (62.46%)
877,556 itlb.itlb_flush (62.46%)
4,576,237,561 iTLB-loads (62.48%)
307,971,166 iTLB-load-misses # 6.73% of all iTLB cache accesses (62.52%)
15,565,279,213 itlb_misses.walk_active (62.55%)
306,240,104 itlb_misses.walk_completed_4k (50.03%)
1,753,560 itlb_misses.walk_completed_2m_4m (50.00%)
2,189 itlb_misses.walk_completed_1g (49.96%)
9.374687885 seconds time elapsed
with MADV_COLLAPSE:
not pipelined:
tps = 1112040.859643 (without initial connection time)
Performance counter stats for 'system wide':
1,569,546,236,696 cycles (62.50%)
7,094,291 itlb.itlb_flush (62.51%)
1,599,845,097 iTLB-loads (62.51%)
692,042,864 iTLB-load-misses # 43.26% of all iTLB cache accesses (62.51%)
31,529,641,124 itlb_misses.walk_active (62.51%)
669,849,177 itlb_misses.walk_completed_4k (49.99%)
22,708,146 itlb_misses.walk_completed_2m_4m (49.99%)
2,752 itlb_misses.walk_completed_1g (49.99%)
13.611206182 seconds time elapsed
pipelined:
tps = 162484.443469 (without initial connection time)
Performance counter stats for 'system wide':
1,092,897,514,658 cycles (62.48%)
942,351 itlb.itlb_flush (62.48%)
233,996,092 iTLB-loads (62.48%)
102,155,575 iTLB-load-misses # 43.66% of all iTLB cache accesses (62.49%)
6,419,597,286 itlb_misses.walk_active (62.52%)
98,758,409 itlb_misses.walk_completed_4k (50.03%)
3,342,332 itlb_misses.walk_completed_2m_4m (50.02%)
2,190 itlb_misses.walk_completed_1g (49.98%)
9.355239897 seconds time elapsed
The difference in itlb.itlb_flush between pipelined / non-pipelined cases
unsurprisingly is stark.
While the pipelined case still sees a good bit reduced itlb traffic, the total
amount of cycles in which a walk is active is just not large enough to matter,
by the looks of it.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: remap the .text segment into huge pages at run time
2022-11-02 06:32 remap the .text segment into huge pages at run time John Naylor <[email protected]>
2022-11-03 17:21 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
@ 2022-11-04 21:21 ` Andres Freund <[email protected]>
2 siblings, 0 replies; 37+ messages in thread
From: Andres Freund @ 2022-11-04 21:21 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Thomas Munro <[email protected]>
Hi,
On 2022-11-03 10:21:23 -0700, Andres Freund wrote:
> > - Add a "cold" __asm__ filler function that just takes up space, enough to
> > push the end of the .text segment over the next aligned boundary, or to
> > ~8MB in size.
>
> I don't understand why this is needed - as long as the pages are aligned to
> 2MB, why do we need to fill things up on disk? The in-memory contents are the
> relevant bit, no?
I now assume it's because you either observed the mappings set up by the
loader to not include the space between the segments?
With sufficient linker flags the segments are sufficiently aligned both on
disk and in memory to just map more:
bfd: -Wl,-zmax-page-size=0x200000,-zcommon-page-size=0x200000
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
...
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000000c7f58 0x00000000000c7f58 R 0x200000
LOAD 0x0000000000200000 0x0000000000200000 0x0000000000200000
0x0000000000921d39 0x0000000000921d39 R E 0x200000
LOAD 0x0000000000c00000 0x0000000000c00000 0x0000000000c00000
0x00000000002626b8 0x00000000002626b8 R 0x200000
LOAD 0x0000000000fdf510 0x00000000011df510 0x00000000011df510
0x0000000000037fd6 0x000000000006a310 RW 0x200000
gold -Wl,-zmax-page-size=0x200000,-zcommon-page-size=0x200000,--rosegment
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
...
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000009230f9 0x00000000009230f9 R E 0x200000
LOAD 0x0000000000a00000 0x0000000000a00000 0x0000000000a00000
0x000000000033a738 0x000000000033a738 R 0x200000
LOAD 0x0000000000ddf4e0 0x0000000000fdf4e0 0x0000000000fdf4e0
0x000000000003800a 0x000000000006a340 RW 0x200000
lld: -Wl,-zmax-page-size=0x200000,-zseparate-loadable-segments
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x000000000033710c 0x000000000033710c R 0x200000
LOAD 0x0000000000400000 0x0000000000400000 0x0000000000400000
0x0000000000921cb0 0x0000000000921cb0 R E 0x200000
LOAD 0x0000000000e00000 0x0000000000e00000 0x0000000000e00000
0x0000000000020ae0 0x0000000000020ae0 RW 0x200000
LOAD 0x0000000001000000 0x0000000001000000 0x0000000001000000
0x00000000000174ea 0x0000000000049820 RW 0x200000
mold -Wl,-zmax-page-size=0x200000,-zcommon-page-size=0x200000,-zseparate-loadable-segments
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
...
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x000000000032dde9 0x000000000032dde9 R 0x200000
LOAD 0x0000000000400000 0x0000000000400000 0x0000000000400000
0x0000000000921cbe 0x0000000000921cbe R E 0x200000
LOAD 0x0000000000e00000 0x0000000000e00000 0x0000000000e00000
0x00000000002174e8 0x0000000000249820 RW 0x200000
With these flags the "R E" segments all start on a 0x200000/2MiB boundary and
are padded to the next 2MiB boundary. However the OS / dynamic loader only
maps the necessary part, not all the zero padding.
This means that if we were to issue a MADV_COLLAPSE, we can before it do an
mremap() to increase the length of the mapping.
MADV_COLLAPSE without mremap:
tps = 1117335.766756 (without initial connection time)
Performance counter stats for 'system wide':
1,169,012,466,070 cycles (55.53%)
729,146,640,019 instructions # 0.62 insn per cycle (66.65%)
7,062,923 itlb.itlb_flush (66.65%)
1,041,825,587 iTLB-loads (66.65%)
634,272,420 iTLB-load-misses # 60.88% of all iTLB cache accesses (66.66%)
27,018,254,873 itlb_misses.walk_active (66.68%)
610,639,252 itlb_misses.walk_completed_4k (44.47%)
24,262,549 itlb_misses.walk_completed_2m_4m (44.46%)
2,948 itlb_misses.walk_completed_1g (44.43%)
10.039217004 seconds time elapsed
MADV_COLLAPSE with mremap:
tps = 1140869.853616 (without initial connection time)
Performance counter stats for 'system wide':
1,173,272,878,934 cycles (55.53%)
746,008,850,147 instructions # 0.64 insn per cycle (66.65%)
7,538,962 itlb.itlb_flush (66.65%)
799,861,088 iTLB-loads (66.65%)
254,347,048 iTLB-load-misses # 31.80% of all iTLB cache accesses (66.66%)
14,427,296,885 itlb_misses.walk_active (66.69%)
221,811,835 itlb_misses.walk_completed_4k (44.47%)
32,881,405 itlb_misses.walk_completed_2m_4m (44.46%)
3,043 itlb_misses.walk_completed_1g (44.43%)
10.038517778 seconds time elapsed
compared to a run without any huge pages (via THP or MADV_COLLAPSE):
tps = 1034960.102843 (without initial connection time)
Performance counter stats for 'system wide':
1,183,743,785,066 cycles (55.54%)
678,525,810,443 instructions # 0.57 insn per cycle (66.65%)
7,163,304 itlb.itlb_flush (66.65%)
2,952,660,798 iTLB-loads (66.65%)
2,105,431,590 iTLB-load-misses # 71.31% of all iTLB cache accesses (66.66%)
80,593,535,910 itlb_misses.walk_active (66.68%)
2,105,377,810 itlb_misses.walk_completed_4k (44.46%)
1,254,156 itlb_misses.walk_completed_2m_4m (44.46%)
3,366 itlb_misses.walk_completed_1g (44.44%)
10.039821650 seconds time elapsed
So a 7.96% win from no-huge-pages to MADV_COLLAPSE and a further 2.11% win
from there to also using mremap(), yielding a total of 10.23%. It's similar
across runs.
On my system the other libraries unfortunately aren't aligned properly. It'd
be nice to also remap at least libc. The majority of the remaining misses are
from the vdso (too small for a huge page), libc (not aligned properly),
returning from system calls (which flush the itlb) and pgbench / libpq (I
didn't add the mremap there, there's not enough code for a huge page without
it).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: remap the .text segment into huge pages at run time
2022-11-02 06:32 remap the .text segment into huge pages at run time John Naylor <[email protected]>
2022-11-03 17:21 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
@ 2022-11-05 05:54 ` John Naylor <[email protected]>
2022-11-05 08:27 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2 siblings, 1 reply; 37+ messages in thread
From: John Naylor @ 2022-11-05 05:54 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Thomas Munro <[email protected]>
On Sat, Nov 5, 2022 at 1:33 AM Andres Freund <[email protected]> wrote:
> > I wonder how far we can get with just using the linker hints to align
> > sections. I know that the linux folks are working on promoting
sufficiently
> > aligned executable pages to huge pages too, and might have succeeded
already.
> >
> > IOW, adding the linker flags might be a good first step.
>
> Indeed, I did see that that works to some degree on the 5.19 kernel I was
> running. However, it never seems to get around to using huge pages
> sufficiently to compete with explicit use of huge pages.
Oh nice, I didn't know that! There might be some threshold of pages mapped
before it does so. At least, that issue is mentioned in that paper linked
upthread for FreeBSD.
> More interestingly, a few days ago, a new madvise hint, MADV_COLLAPSE, was
> added into linux 6.1. That explicitly remaps a region and uses huge pages
for
> it. Of course that's going to take a while to be widely available, but it
> seems like a safer approach than the remapping approach from this thread.
I didn't know that either, funny timing.
> I hacked in a MADV_COLLAPSE (with setarch -R, so that I could just
hardcode
> the address / length), and it seems to work nicely.
>
> With the weird caveat that on fs one needs to make sure that the
executable
> doesn't reflinks to reuse parts of other files, and that the mold linker
and
> cp do... Not a concern on ext4, but on xfs. I took to copying the postgres
> binary with cp --reflink=never
What happens otherwise? That sounds like a difficult thing to guard against.
> The difference in itlb.itlb_flush between pipelined / non-pipelined cases
> unsurprisingly is stark.
>
> While the pipelined case still sees a good bit reduced itlb traffic, the
total
> amount of cycles in which a walk is active is just not large enough to
matter,
> by the looks of it.
Good to know, thanks for testing. Maybe the pipelined case is something
devs should consider when microbenchmarking, to reduce noise from context
switches.
On Sat, Nov 5, 2022 at 4:21 AM Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2022-11-03 10:21:23 -0700, Andres Freund wrote:
> > > - Add a "cold" __asm__ filler function that just takes up space,
enough to
> > > push the end of the .text segment over the next aligned boundary, or
to
> > > ~8MB in size.
> >
> > I don't understand why this is needed - as long as the pages are
aligned to
> > 2MB, why do we need to fill things up on disk? The in-memory contents
are the
> > relevant bit, no?
>
> I now assume it's because you either observed the mappings set up by the
> loader to not include the space between the segments?
My knowledge is not quite that deep. The iodlr repo has an example "hello
world" program, which links with 8 filler objects, each with 32768
__attribute((used)) dummy functions. I just cargo-culted that idea and
simplified it. Interestingly enough, looking through the commit history,
they used to align the segments via linker flags, but took it out here:
https://github.com/intel/iodlr/pull/25#discussion_r397787559
...saying "I'm not sure why we added this". :/
I quickly tried to align the segments with the linker and then in my patch
have the address for mmap() rounded *down* from the .text start to the
beginning of that segment. It refused to start without logging an error.
BTW, that what I meant before, although I wasn't clear:
> > Since the front is all-cold, and there is very little at the end,
> > practically all hot pages are now remapped. The biggest problem with the
> > hackish filler function (in addition to maintainability) is, if explicit
> > huge pages are turned off in the kernel, attempting mmap() with
MAP_HUGETLB
> > causes complete startup failure if the .text segment is larger than 8MB.
>
> I would expect MAP_HUGETLB to always fail if not enabled in the kernel,
> independent of the .text segment size?
With the file-level hack, it would just fail without a trace with .text >
8MB (I have yet to enable core dumps on this new OS I have...), whereas
without it I did see the failures in the log, and successful fallback.
> With these flags the "R E" segments all start on a 0x200000/2MiB boundary
and
> are padded to the next 2MiB boundary. However the OS / dynamic loader only
> maps the necessary part, not all the zero padding.
>
> This means that if we were to issue a MADV_COLLAPSE, we can before it do
an
> mremap() to increase the length of the mapping.
I see, interesting. What location are you passing for madvise() and
mremap()? The beginning of the segment (for me has .init/.plt) or an
aligned boundary within .text?
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: remap the .text segment into huge pages at run time
2022-11-02 06:32 remap the .text segment into huge pages at run time John Naylor <[email protected]>
2022-11-03 17:21 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2022-11-05 05:54 ` Re: remap the .text segment into huge pages at run time John Naylor <[email protected]>
@ 2022-11-05 08:27 ` Andres Freund <[email protected]>
2022-11-06 06:56 ` Re: remap the .text segment into huge pages at run time John Naylor <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Andres Freund @ 2022-11-05 08:27 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Thomas Munro <[email protected]>
Hi,
On 2022-11-05 12:54:18 +0700, John Naylor wrote:
> On Sat, Nov 5, 2022 at 1:33 AM Andres Freund <[email protected]> wrote:
> > I hacked in a MADV_COLLAPSE (with setarch -R, so that I could just
> hardcode
> > the address / length), and it seems to work nicely.
> >
> > With the weird caveat that on fs one needs to make sure that the
> executable
> > doesn't reflinks to reuse parts of other files, and that the mold linker
> and
> > cp do... Not a concern on ext4, but on xfs. I took to copying the postgres
> > binary with cp --reflink=never
>
> What happens otherwise? That sounds like a difficult thing to guard against.
MADV_COLLAPSE fails, but otherwise things continue on. I think it's mostly an
issue on dev systems, not on prod systems, because there the files will be be
unpacked from a package or such.
> > On 2022-11-03 10:21:23 -0700, Andres Freund wrote:
> > > > - Add a "cold" __asm__ filler function that just takes up space,
> enough to
> > > > push the end of the .text segment over the next aligned boundary, or
> to
> > > > ~8MB in size.
> > >
> > > I don't understand why this is needed - as long as the pages are
> aligned to
> > > 2MB, why do we need to fill things up on disk? The in-memory contents
> are the
> > > relevant bit, no?
> >
> > I now assume it's because you either observed the mappings set up by the
> > loader to not include the space between the segments?
>
> My knowledge is not quite that deep. The iodlr repo has an example "hello
> world" program, which links with 8 filler objects, each with 32768
> __attribute((used)) dummy functions. I just cargo-culted that idea and
> simplified it. Interestingly enough, looking through the commit history,
> they used to align the segments via linker flags, but took it out here:
>
> https://github.com/intel/iodlr/pull/25#discussion_r397787559
>
> ...saying "I'm not sure why we added this". :/
That was about using a linker script, not really linker flags though.
I don't think the dummy functions are a good approach, there were plenty
things after it when I played with them.
> I quickly tried to align the segments with the linker and then in my patch
> have the address for mmap() rounded *down* from the .text start to the
> beginning of that segment. It refused to start without logging an error.
Hm, what linker was that? I did note that you need some additional flags for
some of the linkers.
> > With these flags the "R E" segments all start on a 0x200000/2MiB boundary
> and
> > are padded to the next 2MiB boundary. However the OS / dynamic loader only
> > maps the necessary part, not all the zero padding.
> >
> > This means that if we were to issue a MADV_COLLAPSE, we can before it do
> an
> > mremap() to increase the length of the mapping.
>
> I see, interesting. What location are you passing for madvise() and
> mremap()? The beginning of the segment (for me has .init/.plt) or an
> aligned boundary within .text?
I started postgres with setarch -R, looked at /proc/$pid/[s]maps to see the
start/end of the r-xp mapped segment. Here's my hacky code, with a bunch of
comments added.
void *addr = (void*) 0x555555800000;
void *end = (void *) 0x555555e09000;
size_t advlen = (uintptr_t) end - (uintptr_t) addr;
const size_t bound = 1024*1024*2 - 1;
size_t advlen_up = (advlen + bound - 1) & ~(bound - 1);
void *r2;
/*
* Increase size of mapping to cover the tailing padding to the next
* segment. Otherwise all the code in that range can't be put into
* a huge page (access in the non-mapped range needs to cause a fault,
* hence can't be in the huge page).
* XXX: Should proably assert that that space is actually zeroes.
*/
r2 = mremap(addr, advlen, advlen_up, 0);
if (r2 == MAP_FAILED)
fprintf(stderr, "mremap failed: %m\n");
else if (r2 != addr)
fprintf(stderr, "mremap wrong addr: %m\n");
else
advlen = advlen_up;
/*
* The docs for MADV_COLLAPSE say there should be at least one page
* in the mapped space "for every eligible hugepage-aligned/sized
* region to be collapsed". I just forced that. But probably not
* necessary.
*/
r = madvise(addr, advlen, MADV_WILLNEED);
if (r != 0)
fprintf(stderr, "MADV_WILLNEED failed: %m\n");
r = madvise(addr, advlen, MADV_POPULATE_READ);
if (r != 0)
fprintf(stderr, "MADV_POPULATE_READ failed: %m\n");
/*
* Make huge pages out of it. Requires at least linux 6.1. We could
* fall back to MADV_HUGEPAGE if it fails, but it doesn't do all that
* much in older kernels.
*/
#define MADV_COLLAPSE 25
r = madvise(addr, advlen, MADV_COLLAPSE);
if (r != 0)
fprintf(stderr, "MADV_COLLAPSE failed: %m\n");
A real version would have to open /proc/self/maps and do this for at least
postgres' r-xp mapping. We could do it for libraries too, if they're suitably
aligned (both in memory and on-disk).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: remap the .text segment into huge pages at run time
2022-11-02 06:32 remap the .text segment into huge pages at run time John Naylor <[email protected]>
2022-11-03 17:21 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2022-11-05 05:54 ` Re: remap the .text segment into huge pages at run time John Naylor <[email protected]>
2022-11-05 08:27 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
@ 2022-11-06 06:56 ` John Naylor <[email protected]>
2022-11-06 18:16 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: John Naylor @ 2022-11-06 06:56 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Thomas Munro <[email protected]>
On Sat, Nov 5, 2022 at 3:27 PM Andres Freund <[email protected]> wrote:
> > simplified it. Interestingly enough, looking through the commit history,
> > they used to align the segments via linker flags, but took it out here:
> >
> > https://github.com/intel/iodlr/pull/25#discussion_r397787559
> >
> > ...saying "I'm not sure why we added this". :/
>
> That was about using a linker script, not really linker flags though.
Oops, the commit I was referring to pointed to that discussion, but I
should have shown it instead:
--- a/large_page-c/example/Makefile
+++ b/large_page-c/example/Makefile
@@ -28,7 +28,6 @@ OBJFILES= \
filler16.o \
OBJS=$(addprefix $(OBJDIR)/,$(OBJFILES))
-LDFLAGS=-Wl,-z,max-page-size=2097152
But from what you're saying, this flag wouldn't have been enough anyway...
> I don't think the dummy functions are a good approach, there were plenty
> things after it when I played with them.
To be technical, the point wasn't to have no code after it, but to have no
*hot* code *before* it, since with the iodlr approach the first 1.99MB of
.text is below the first aligned boundary within that section. But yeah,
I'm happy to ditch that hack entirely.
> > > With these flags the "R E" segments all start on a 0x200000/2MiB
boundary
> > and
> > > are padded to the next 2MiB boundary. However the OS / dynamic loader
only
> > > maps the necessary part, not all the zero padding.
> > >
> > > This means that if we were to issue a MADV_COLLAPSE, we can before it
do
> > an
> > > mremap() to increase the length of the mapping.
> >
> > I see, interesting. What location are you passing for madvise() and
> > mremap()? The beginning of the segment (for me has .init/.plt) or an
> > aligned boundary within .text?
> /*
> * Make huge pages out of it. Requires at least linux 6.1. We
could
> * fall back to MADV_HUGEPAGE if it fails, but it doesn't do all
that
> * much in older kernels.
> */
About madvise(), I take it MADV_HUGEPAGE and MADV_COLLAPSE only work for
THP? The man page seems to indicate that.
In the support work I've done, the standard recommendation is to turn THP
off, especially if they report sudden performance problems. If explicit
HP's are used for shared mem, maybe THP is less of a risk? I need to look
back at the tests that led to that advice...
> A real version would have to open /proc/self/maps and do this for at least
I can try and generalize your above sketch into a v2 patch.
> postgres' r-xp mapping. We could do it for libraries too, if they're
suitably
> aligned (both in memory and on-disk).
It looks like plpgsql is only 27 standard pages in size...
Regarding glibc, we could try moving a couple of the hotter functions into
PG, using smaller and simpler coding, if that has better frontend cache
behavior. The paper "Understanding and Mitigating Front-End Stalls in
Warehouse-Scale Computers" talks about this, particularly section 4.4
regarding memcmp().
> > I quickly tried to align the segments with the linker and then in my
patch
> > have the address for mmap() rounded *down* from the .text start to the
> > beginning of that segment. It refused to start without logging an error.
>
> Hm, what linker was that? I did note that you need some additional flags
for
> some of the linkers.
BFD, but I wouldn't worry about that failure too much, since the
mremap()/madvise() strategy has a lot fewer moving parts.
On the subject of linkers, though, one thing that tripped me up was trying
to change the linker with Meson. First I tried
-Dc_args='-fuse-ld=lld'
but that led to warnings like this when :
/usr/bin/ld: warning: -z separate-loadable-segments ignored
When using this in the top level meson.build
elif host_system == 'linux'
sema_kind = 'unnamed_posix'
cppflags += '-D_GNU_SOURCE'
# Align the loadable segments to 2MB boundaries to support remapping to
# huge pages.
ldflags += cc.get_supported_link_arguments([
'-Wl,-zmax-page-size=0x200000',
'-Wl,-zcommon-page-size=0x200000',
'-Wl,-zseparate-loadable-segments'
])
According to
https://mesonbuild.com/howtox.html#set-linker
I need to add CC_LD=lld to the env vars before invoking, which got rid of
the warning. Then I wanted to verify that lld was actually used, and in
https://releases.llvm.org/14.0.0/tools/lld/docs/index.html
it says I can run this and it should show “Linker: LLD”, but that doesn't
appear for me:
$ readelf --string-dump .comment inst-perf/bin/postgres
String dump of section '.comment':
[ 0] GCC: (GNU) 12.2.1 20220819 (Red Hat 12.2.1-2)
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: remap the .text segment into huge pages at run time
2022-11-02 06:32 remap the .text segment into huge pages at run time John Naylor <[email protected]>
2022-11-03 17:21 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2022-11-05 05:54 ` Re: remap the .text segment into huge pages at run time John Naylor <[email protected]>
2022-11-05 08:27 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2022-11-06 06:56 ` Re: remap the .text segment into huge pages at run time John Naylor <[email protected]>
@ 2022-11-06 18:16 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 37+ messages in thread
From: Andres Freund @ 2022-11-06 18:16 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Thomas Munro <[email protected]>
Hi,
On 2022-11-06 13:56:10 +0700, John Naylor wrote:
> On Sat, Nov 5, 2022 at 3:27 PM Andres Freund <[email protected]> wrote:
> > I don't think the dummy functions are a good approach, there were plenty
> > things after it when I played with them.
>
> To be technical, the point wasn't to have no code after it, but to have no
> *hot* code *before* it, since with the iodlr approach the first 1.99MB of
> .text is below the first aligned boundary within that section. But yeah,
> I'm happy to ditch that hack entirely.
Just because code is colder than the alternative branch, doesn't necessary
mean it's entirely cold overall. I saw hits to things after the dummy function
to have a perf effect.
> > > > With these flags the "R E" segments all start on a 0x200000/2MiB
> boundary
> > > and
> > > > are padded to the next 2MiB boundary. However the OS / dynamic loader
> only
> > > > maps the necessary part, not all the zero padding.
> > > >
> > > > This means that if we were to issue a MADV_COLLAPSE, we can before it
> do
> > > an
> > > > mremap() to increase the length of the mapping.
> > >
> > > I see, interesting. What location are you passing for madvise() and
> > > mremap()? The beginning of the segment (for me has .init/.plt) or an
> > > aligned boundary within .text?
>
> > /*
> > * Make huge pages out of it. Requires at least linux 6.1. We
> could
> > * fall back to MADV_HUGEPAGE if it fails, but it doesn't do all
> that
> > * much in older kernels.
> > */
>
> About madvise(), I take it MADV_HUGEPAGE and MADV_COLLAPSE only work for
> THP? The man page seems to indicate that.
MADV_HUGEPAGE works as long as /sys/kernel/mm/transparent_hugepage/enabled is
to always or madvise. My understanding is that MADV_COLLAPSE will work even
if /sys/kernel/mm/transparent_hugepage/enabled is set to never.
> In the support work I've done, the standard recommendation is to turn THP
> off, especially if they report sudden performance problems.
I think that's pretty much an outdated suggestion FWIW. Largely caused by Red
Hat extremely aggressively backpatching transparent hugepages into RHEL 6
(IIRC). Lots of improvements have been made to THP since then. I've tried to
see negative effects maybe 2-3 years back, without success.
I really don't see a reason to ever set
/sys/kernel/mm/transparent_hugepage/enabled to 'never', rather than just 'madvise'.
> If explicit HP's are used for shared mem, maybe THP is less of a risk? I
> need to look back at the tests that led to that advice...
I wouldn't give that advice to customers anymore, unless they use extremely
old platforms or unless there's very concrete evidence.
> > A real version would have to open /proc/self/maps and do this for at least
>
> I can try and generalize your above sketch into a v2 patch.
Cool.
> > postgres' r-xp mapping. We could do it for libraries too, if they're
> suitably
> > aligned (both in memory and on-disk).
>
> It looks like plpgsql is only 27 standard pages in size...
>
> Regarding glibc, we could try moving a couple of the hotter functions into
> PG, using smaller and simpler coding, if that has better frontend cache
> behavior. The paper "Understanding and Mitigating Front-End Stalls in
> Warehouse-Scale Computers" talks about this, particularly section 4.4
> regarding memcmp().
I think the amount of work necessary for that is nontrivial and continual. So
I'm loathe to go there.
> > > I quickly tried to align the segments with the linker and then in my
> patch
> > > have the address for mmap() rounded *down* from the .text start to the
> > > beginning of that segment. It refused to start without logging an error.
> >
> > Hm, what linker was that? I did note that you need some additional flags
> for
> > some of the linkers.
>
> BFD, but I wouldn't worry about that failure too much, since the
> mremap()/madvise() strategy has a lot fewer moving parts.
>
> On the subject of linkers, though, one thing that tripped me up was trying
> to change the linker with Meson. First I tried
>
> -Dc_args='-fuse-ld=lld'
It's -Dc_link_args=...
> but that led to warnings like this when :
> /usr/bin/ld: warning: -z separate-loadable-segments ignored
>
> When using this in the top level meson.build
>
> elif host_system == 'linux'
> sema_kind = 'unnamed_posix'
> cppflags += '-D_GNU_SOURCE'
> # Align the loadable segments to 2MB boundaries to support remapping to
> # huge pages.
> ldflags += cc.get_supported_link_arguments([
> '-Wl,-zmax-page-size=0x200000',
> '-Wl,-zcommon-page-size=0x200000',
> '-Wl,-zseparate-loadable-segments'
> ])
>
>
> According to
>
> https://mesonbuild.com/howtox.html#set-linker
>
> I need to add CC_LD=lld to the env vars before invoking, which got rid of
> the warning. Then I wanted to verify that lld was actually used, and in
>
> https://releases.llvm.org/14.0.0/tools/lld/docs/index.html
You can just look at build.ninja, fwiw. Or use ninja -v (in postgres's cases
with -d keeprsp, because the commandline ends up being long enough for a
response file being used).
> it says I can run this and it should show “Linker: LLD”, but that doesn't
> appear for me:
>
> $ readelf --string-dump .comment inst-perf/bin/postgres
>
> String dump of section '.comment':
> [ 0] GCC: (GNU) 12.2.1 20220819 (Red Hat 12.2.1-2)
That's added by the compiler, not the linker. See e.g.:
$ readelf --string-dump .comment src/backend/postgres_lib.a.p/storage_ipc_procarray.c.o
String dump of section '.comment':
[ 1] GCC: (Debian 12.2.0-9) 12.2.0
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 37+ messages in thread
* [PATCH v1 2/7] Row pattern recognition patch (parse/analysis).
@ 2023-06-25 11:48 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 37+ messages in thread
From: Tatsuo Ishii @ 2023-06-25 11:48 UTC (permalink / raw)
---
src/backend/parser/parse_agg.c | 7 ++
src/backend/parser/parse_clause.c | 196 +++++++++++++++++++++++++++++-
src/backend/parser/parse_expr.c | 4 +
src/backend/parser/parse_func.c | 3 +
4 files changed, 209 insertions(+), 1 deletion(-)
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index 85cd47b7ae..aa7a1cee80 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
errkind = true;
break;
+ case EXPR_KIND_RPR_DEFINE:
+ errkind = true;
+ break;
+
/*
* There is intentionally no default: case here, so that the
* compiler will warn if we add a new ParseExprKind without
@@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc,
case EXPR_KIND_CYCLE_MARK:
errkind = true;
break;
+ case EXPR_KIND_RPR_DEFINE:
+ errkind = true;
+ break;
/*
* There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c
index f61f794755..5bb11add3c 100644
--- a/src/backend/parser/parse_clause.c
+++ b/src/backend/parser/parse_clause.c
@@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name);
static Node *transformFrameOffset(ParseState *pstate, int frameOptions,
Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc,
Node *clause);
-
+static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef);
+static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef);
+static List *transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef);
+static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef);
/*
* transformFromClause -
@@ -2949,6 +2952,10 @@ transformWindowDefinitions(ParseState *pstate,
rangeopfamily, rangeopcintype,
&wc->endInRangeFunc,
windef->endOffset);
+
+ /* Process Row Pattern Recognition related clauses */
+ transformRPR(pstate, wc, windef);
+
wc->runCondition = NIL;
wc->winref = winref;
@@ -3814,3 +3821,190 @@ transformFrameOffset(ParseState *pstate, int frameOptions,
return node;
}
+
+/*
+ * transformRPR
+ * Process Row Pattern Recognition related clauses
+ */
+static void
+transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef)
+{
+ /* Check Frame option. Frame must start at current row */
+
+ /*
+ * Window definition exists?
+ */
+ if (windef == NULL)
+ return;
+
+ /*
+ * Row Pattern Common Syntax clause exists?
+ */
+ if (windef->rpCommonSyntax == NULL)
+ return;
+
+ if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("FRAME must start at current row when row patttern recognition is used")));
+
+ /* Transform AFTER MACH SKIP TO clause */
+ wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo;
+ if (wc->rpSkipTo != ST_NEXT_ROW)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("only AFTER MATCH SKIP TO NEXT_ROW is supported")));
+
+ /* Transform SEEK or INITIAL clause */
+ wc->initial = windef->rpCommonSyntax->initial;
+
+ /* Transform DEFINE clause into list of TargetEntry's */
+ wc->defineClause = transformDefineClause(pstate, wc, windef);
+
+ /* Check PATTERN clause and copy to patternClause */
+ transformPatternClause(pstate, wc, windef);
+
+ /* Transform MEASURE clause */
+ transformMeasureClause(pstate, wc, windef);
+}
+
+/*
+ * transformDefineClause Process DEFINE clause and transform ResTarget into
+ * list of TargetEntry.
+ *
+ * XXX we only support column reference in row pattern definition search
+ * condition, e.g. "price". <row pattern definition variable name>.<column
+ * reference> is not supported, e.g. "A.price".
+ */
+static List *
+transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef)
+{
+ ListCell *lc;
+ ResTarget *restarget, *r;
+ List *restargets;
+
+
+ /*
+ * If Row Definition Common Syntax exists, DEFINE clause must exist.
+ * (the raw parser should have already checked it.)
+ */
+ Assert(windef->rpCommonSyntax->rpDefs != NULL);
+
+ /*
+ * Check for duplicate row pattern definition variables. The standard
+ * requires that no two row pattern definition variable names shall be
+ * equivalent.
+ */
+ restargets = NIL;
+ foreach(lc, windef->rpCommonSyntax->rpDefs)
+ {
+ char *name;
+ ListCell *l;
+
+ restarget = (ResTarget *)lfirst(lc);
+ name = restarget->name;
+
+ /*
+ * Make sure that row pattern definition search condition is a boolean
+ * expression.
+ */
+ transformWhereClause(pstate, restarget->val,
+ EXPR_KIND_RPR_DEFINE, "DEFINE");
+
+ foreach(l, restargets)
+ {
+ char *n;
+
+ r = (ResTarget *) lfirst(l);
+ n = r->name;
+
+ if (!strcmp(n, name))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause",
+ name),
+ parser_errposition(pstate, exprLocation((Node *)r))));
+ }
+ restargets = lappend(restargets, restarget);
+ }
+ list_free(restargets);
+
+ return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs,
+ EXPR_KIND_RPR_DEFINE);
+}
+
+/*
+ * transformPatternClause
+ * Process PATTERN clause and return PATTERN clause in the raw parse tree
+ */
+static List *
+transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef)
+{
+ List *patterns;
+ ListCell *lc, *l;
+
+ /*
+ * Row Pattern Common Syntax clause exists?
+ */
+ if (windef->rpCommonSyntax == NULL)
+ return NULL;
+
+ /*
+ * Primary row pattern variable names in PATTERN clause must appear in
+ * DEFINE clause as row pattern definition variable names.
+ */
+ wc->patternVariable = NIL;
+ wc->patternRegexp = NIL;
+ foreach(lc, windef->rpCommonSyntax->rpPatterns)
+ {
+ A_Expr *a;
+ char *name;
+ char *regexp;
+ bool found = false;
+
+ if (!IsA(lfirst(lc), A_Expr))
+ ereport(ERROR,
+ errmsg("node type is not A_Expr"));
+
+ a = (A_Expr *)lfirst(lc);
+ name = strVal(a->lexpr);
+
+ foreach(l, windef->rpCommonSyntax->rpDefs)
+ {
+ ResTarget *restarget = (ResTarget *)lfirst(l);
+
+ if (!strcmp(restarget->name, name))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause",
+ name),
+ parser_errposition(pstate, exprLocation((Node *)a))));
+ wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name)));
+ regexp = strVal(lfirst(list_head(a->name)));
+ wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp)));
+ }
+ return patterns;
+}
+
+/*
+ * transformMeasureClause
+ * Process MEASURE clause
+ */
+static List *
+transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef)
+{
+ if (windef->rowPatternMeasures == NIL)
+ return NIL;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s","MEASURE clause is not supported yet"),
+ parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures))));
+}
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index 346fd272b6..20231d9ec0 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -541,6 +541,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref)
case EXPR_KIND_COPY_WHERE:
case EXPR_KIND_GENERATED_COLUMN:
case EXPR_KIND_CYCLE_MARK:
+ case EXPR_KIND_RPR_DEFINE:
/* okay */
break;
@@ -1754,6 +1755,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink)
case EXPR_KIND_VALUES:
case EXPR_KIND_VALUES_SINGLE:
case EXPR_KIND_CYCLE_MARK:
+ case EXPR_KIND_RPR_DEFINE:
/* okay */
break;
case EXPR_KIND_CHECK_CONSTRAINT:
@@ -3133,6 +3135,8 @@ ParseExprKindName(ParseExprKind exprKind)
return "GENERATED AS";
case EXPR_KIND_CYCLE_MARK:
return "CYCLE";
+ case EXPR_KIND_RPR_DEFINE:
+ return "DEFINE";
/*
* There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index b3f0b6a137..2ff3699538 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location)
case EXPR_KIND_CYCLE_MARK:
errkind = true;
break;
+ case EXPR_KIND_RPR_DEFINE:
+ errkind = true;
+ break;
/*
* There is intentionally no default: case here, so that the
--
2.25.1
----Next_Part(Sun_Jun_25_21_05_09_2023_126)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v1-0003-Row-pattern-recognition-patch-planner.patch"
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: [18] Policy on IMMUTABLE functions and Unicode updates
@ 2024-07-24 19:12 Jeff Davis <[email protected]>
2024-07-24 19:19 ` Re: [18] Policy on IMMUTABLE functions and Unicode updates Robert Haas <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Jeff Davis @ 2024-07-24 19:12 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Laurenz Albe <[email protected]>; pgsql-hackers; Daniel Verite <[email protected]>; Noah Misch <[email protected]>; Tom Lane <[email protected]>; Jeremy Schneider <[email protected]>
On Wed, 2024-07-24 at 14:47 -0400, Robert Haas wrote:
> On Wed, Jul 24, 2024 at 1:45 PM Jeff Davis <[email protected]> wrote:
> > There's a qualitative difference between a collation update which
> > can
> > break your PKs and FKs, and a ctype update which definitely will
> > not.
>
> I don't think that's true. All you need is a unique index on
> UPPER(somecol).
Primary keys are on plain column references, not expressions; and don't
support WHERE clauses, so I don't see how a ctype update would affect a
PK.
In any case, you are correct that Unicode updates could put some
constraints at risk, including unique indexes, CHECK, and partition
constraints. But someone has to actually use one of the affected
functions somewhere, and that's the main distinction that I'm trying to
draw.
The reason why collation is qualitatively a much bigger problem is
because there's no obvious indication that you are doing anything
related to collation at all. A very plain "CREATE TABLE x(t text
PRIMARY KEY)" is at risk.
Regards,
Jeff Davis
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: [18] Policy on IMMUTABLE functions and Unicode updates
2024-07-24 19:12 Re: [18] Policy on IMMUTABLE functions and Unicode updates Jeff Davis <[email protected]>
@ 2024-07-24 19:19 ` Robert Haas <[email protected]>
2024-07-24 19:43 ` Re: [18] Policy on IMMUTABLE functions and Unicode updates Jeremy Schneider <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Robert Haas @ 2024-07-24 19:19 UTC (permalink / raw)
To: Jeff Davis <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Laurenz Albe <[email protected]>; pgsql-hackers; Daniel Verite <[email protected]>; Noah Misch <[email protected]>; Tom Lane <[email protected]>; Jeremy Schneider <[email protected]>
On Wed, Jul 24, 2024 at 3:12 PM Jeff Davis <[email protected]> wrote:
> In any case, you are correct that Unicode updates could put some
> constraints at risk, including unique indexes, CHECK, and partition
> constraints. But someone has to actually use one of the affected
> functions somewhere, and that's the main distinction that I'm trying to
> draw.
>
> The reason why collation is qualitatively a much bigger problem is
> because there's no obvious indication that you are doing anything
> related to collation at all. A very plain "CREATE TABLE x(t text
> PRIMARY KEY)" is at risk.
Well, I don't know. I agree that collation is a much bigger problem,
but not for that reason. I think a user who is familiar with the
problems in this area will see the danger either way, and one who
isn't, won't. For me, the only real difference is that a unique index
on a text column is a lot more common than one that involves UPPER.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: [18] Policy on IMMUTABLE functions and Unicode updates
2024-07-24 19:12 Re: [18] Policy on IMMUTABLE functions and Unicode updates Jeff Davis <[email protected]>
2024-07-24 19:19 ` Re: [18] Policy on IMMUTABLE functions and Unicode updates Robert Haas <[email protected]>
@ 2024-07-24 19:43 ` Jeremy Schneider <[email protected]>
2024-07-24 19:47 ` Re: [18] Policy on IMMUTABLE functions and Unicode updates Robert Haas <[email protected]>
0 siblings, 1 reply; 37+ messages in thread
From: Jeremy Schneider @ 2024-07-24 19:43 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Daniel Verite <[email protected]>; Jeff Davis <[email protected]>; Laurenz Albe <[email protected]>; Noah Misch <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers
On Wed, Jul 24, 2024 at 12:47 PM Robert Haas <[email protected]> wrote:
On Wed, Jul 24, 2024 at 1:45 PM Jeff Davis <[email protected]> wrote:
> There's a qualitative difference between a collation update which can
> break your PKs and FKs, and a ctype update which definitely will not.
I don't think that's true. All you need is a unique index on UPPER(somecol).
I doubt it’s common to have unique on upper()
But non-unique indexes for case insensitive searches will be more common.
Historically this is the most common way people did case insensitive on
oracle.
Changing ctype would mean these queries can return wrong results
The impact would be similar to the critical problem TripAdvisor hit in 2014
with their read replicas, in the Postgres email thread I linked above
-Jeremy
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: [18] Policy on IMMUTABLE functions and Unicode updates
2024-07-24 19:12 Re: [18] Policy on IMMUTABLE functions and Unicode updates Jeff Davis <[email protected]>
2024-07-24 19:19 ` Re: [18] Policy on IMMUTABLE functions and Unicode updates Robert Haas <[email protected]>
2024-07-24 19:43 ` Re: [18] Policy on IMMUTABLE functions and Unicode updates Jeremy Schneider <[email protected]>
@ 2024-07-24 19:47 ` Robert Haas <[email protected]>
0 siblings, 0 replies; 37+ messages in thread
From: Robert Haas @ 2024-07-24 19:47 UTC (permalink / raw)
To: Jeremy Schneider <[email protected]>; +Cc: Daniel Verite <[email protected]>; Jeff Davis <[email protected]>; Laurenz Albe <[email protected]>; Noah Misch <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers
On Wed, Jul 24, 2024 at 3:43 PM Jeremy Schneider
<[email protected]> wrote:
> But non-unique indexes for case insensitive searches will be more common. Historically this is the most common way people did case insensitive on oracle.
>
> Changing ctype would mean these queries can return wrong results
Yeah. I mentioned earlier that I very recently saw a customer query
with UPPER() in the join condition. If someone is doing foo JOIN bar
ON upper(foo.x) = upper(bar.x), it is not unlikely that one or both of
those expressions are indexed. Not guaranteed, of course, but very
plausible.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 37+ messages in thread
end of thread, other threads:[~2024-07-24 19:47 UTC | newest]
Thread overview: 37+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-11-05 11:40 Re: Undo worker and transaction rollback Dilip Kumar <[email protected]>
2018-11-05 12:32 ` Dilip Kumar <[email protected]>
2018-11-30 15:13 ` Dmitry Dolgov <[email protected]>
2018-12-03 12:44 ` Dilip Kumar <[email protected]>
2018-12-04 09:43 ` Dilip Kumar <[email protected]>
2018-12-31 05:34 ` Dilip Kumar <[email protected]>
2019-01-11 04:09 ` Dilip Kumar <[email protected]>
2019-01-23 04:09 ` Amit Kapila <[email protected]>
2019-01-24 09:01 ` Dilip Kumar <[email protected]>
2019-01-15 04:08 current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 05:15 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-15 08:55 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-15 14:47 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-15 15:53 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 02:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-16 18:22 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-16 18:39 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Tom Lane <[email protected]>
2019-01-16 18:49 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-18 04:08 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Haribabu Kommi <[email protected]>
2019-01-18 14:50 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-19 01:49 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Michael Paquier <[email protected]>
2019-01-19 15:41 ` Re: current_logfiles not following group access and instead follows log_file_mode permissions Stephen Frost <[email protected]>
2019-01-17 12:05 [PATCH 2/4] Change-tmplinit-argument Arthur Zakirov <[email protected]>
2019-01-17 12:05 [PATCH 2/4] Change-tmplinit-argument Arthur Zakirov <[email protected]>
2022-11-02 06:32 remap the .text segment into huge pages at run time John Naylor <[email protected]>
2022-11-03 17:21 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2022-11-04 18:33 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2022-11-04 21:21 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2022-11-05 05:54 ` Re: remap the .text segment into huge pages at run time John Naylor <[email protected]>
2022-11-05 08:27 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2022-11-06 06:56 ` Re: remap the .text segment into huge pages at run time John Naylor <[email protected]>
2022-11-06 18:16 ` Re: remap the .text segment into huge pages at run time Andres Freund <[email protected]>
2023-06-25 11:48 [PATCH v1 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]>
2024-07-24 19:12 Re: [18] Policy on IMMUTABLE functions and Unicode updates Jeff Davis <[email protected]>
2024-07-24 19:19 ` Re: [18] Policy on IMMUTABLE functions and Unicode updates Robert Haas <[email protected]>
2024-07-24 19:43 ` Re: [18] Policy on IMMUTABLE functions and Unicode updates Jeremy Schneider <[email protected]>
2024-07-24 19:47 ` Re: [18] Policy on IMMUTABLE functions and Unicode updates Robert Haas <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox