public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v2 6/7] TupleHashTable.entrysize was unused except for instrumentation..
23+ messages / 8 participants
[nested] [flat]
* [PATCH v2 6/7] TupleHashTable.entrysize was unused except for instrumentation..
@ 2020-02-15 23:19 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 23+ messages in thread
From: Justin Pryzby @ 2020-02-15 23:19 UTC (permalink / raw)
---
src/backend/executor/execGrouping.c | 1 -
src/include/nodes/execnodes.h | 1 -
2 files changed, 2 deletions(-)
diff --git a/src/backend/executor/execGrouping.c b/src/backend/executor/execGrouping.c
index de0205f..d76a630 100644
--- a/src/backend/executor/execGrouping.c
+++ b/src/backend/executor/execGrouping.c
@@ -186,7 +186,6 @@ BuildTupleHashTableExt(PlanState *parent,
hashtable->tab_collations = collations;
hashtable->tablecxt = tablecxt;
hashtable->tempcxt = tempcxt;
- hashtable->entrysize = entrysize;
hashtable->tableslot = NULL; /* will be made on first lookup */
hashtable->inputslot = NULL;
hashtable->in_hash_funcs = NULL;
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index f8c93dd..2bcd140 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -724,7 +724,6 @@ typedef struct TupleHashTableData
Oid *tab_collations; /* collations for hash and comparison */
MemoryContext tablecxt; /* memory context containing table */
MemoryContext tempcxt; /* context for function evaluations */
- Size entrysize; /* actual size to make each hash entry */
TupleTableSlot *tableslot; /* slot for referencing table entries */
/* The following fields are set transiently for each table search: */
TupleTableSlot *inputslot; /* current input tuple's slot */
--
2.7.4
--DiL7RhKs8rK9YGuF
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0007-Update-comment-obsolete-since-69c3936a.patch"
^ permalink raw reply [nested|flat] 23+ messages in thread
* [PATCH v7 6/9] TupleHashTable.entrysize was unused except for instrumentation..
@ 2020-02-15 23:19 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 23+ messages in thread
From: Justin Pryzby @ 2020-02-15 23:19 UTC (permalink / raw)
---
src/backend/executor/execGrouping.c | 1 -
src/include/nodes/execnodes.h | 1 -
2 files changed, 2 deletions(-)
diff --git a/src/backend/executor/execGrouping.c b/src/backend/executor/execGrouping.c
index 009d27b9a8..98fd4bf8bd 100644
--- a/src/backend/executor/execGrouping.c
+++ b/src/backend/executor/execGrouping.c
@@ -183,7 +183,6 @@ BuildTupleHashTableExt(PlanState *parent,
hashtable->tab_collations = collations;
hashtable->tablecxt = tablecxt;
hashtable->tempcxt = tempcxt;
- hashtable->entrysize = entrysize;
hashtable->tableslot = NULL; /* will be made on first lookup */
hashtable->inputslot = NULL;
hashtable->in_hash_funcs = NULL;
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 11fe866c1c..3a335d87a0 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -726,7 +726,6 @@ typedef struct TupleHashTableData
Oid *tab_collations; /* collations for hash and comparison */
MemoryContext tablecxt; /* memory context containing table */
MemoryContext tempcxt; /* context for function evaluations */
- Size entrysize; /* actual size to make each hash entry */
TupleTableSlot *tableslot; /* slot for referencing table entries */
/* The following fields are set transiently for each table search: */
TupleTableSlot *inputslot; /* current input tuple's slot */
--
2.17.0
--UfEAyuTBtIjiZzX6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0007-Update-comment-obsolete-since-69c3936a.patch"
^ permalink raw reply [nested|flat] 23+ messages in thread
* [PATCH v9 7/8] TupleHashTable.entrysize was unused except for instrumentation..
@ 2020-02-15 23:19 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 23+ messages in thread
From: Justin Pryzby @ 2020-02-15 23:19 UTC (permalink / raw)
---
src/backend/executor/execGrouping.c | 1 -
src/include/nodes/execnodes.h | 1 -
2 files changed, 2 deletions(-)
diff --git a/src/backend/executor/execGrouping.c b/src/backend/executor/execGrouping.c
index 009d27b9a8..98fd4bf8bd 100644
--- a/src/backend/executor/execGrouping.c
+++ b/src/backend/executor/execGrouping.c
@@ -183,7 +183,6 @@ BuildTupleHashTableExt(PlanState *parent,
hashtable->tab_collations = collations;
hashtable->tablecxt = tablecxt;
hashtable->tempcxt = tempcxt;
- hashtable->entrysize = entrysize;
hashtable->tableslot = NULL; /* will be made on first lookup */
hashtable->inputslot = NULL;
hashtable->in_hash_funcs = NULL;
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 10239aea4f..bca77d52b6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -730,7 +730,6 @@ typedef struct TupleHashTableData
Oid *tab_collations; /* collations for hash and comparison */
MemoryContext tablecxt; /* memory context containing table */
MemoryContext tempcxt; /* context for function evaluations */
- Size entrysize; /* actual size to make each hash entry */
TupleTableSlot *tableslot; /* slot for referencing table entries */
/* The following fields are set transiently for each table search: */
TupleTableSlot *inputslot; /* current input tuple's slot */
--
2.17.0
--wwtQuX191/I956S7
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0008-Update-comment-obsolete-since-69c3936a.patch"
^ permalink raw reply [nested|flat] 23+ messages in thread
* [PATCH v5 6/7] TupleHashTable.entrysize was unused except for instrumentation..
@ 2020-02-15 23:19 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 23+ messages in thread
From: Justin Pryzby @ 2020-02-15 23:19 UTC (permalink / raw)
---
src/backend/executor/execGrouping.c | 1 -
src/include/nodes/execnodes.h | 1 -
2 files changed, 2 deletions(-)
diff --git a/src/backend/executor/execGrouping.c b/src/backend/executor/execGrouping.c
index de0205f..d76a630 100644
--- a/src/backend/executor/execGrouping.c
+++ b/src/backend/executor/execGrouping.c
@@ -186,7 +186,6 @@ BuildTupleHashTableExt(PlanState *parent,
hashtable->tab_collations = collations;
hashtable->tablecxt = tablecxt;
hashtable->tempcxt = tempcxt;
- hashtable->entrysize = entrysize;
hashtable->tableslot = NULL; /* will be made on first lookup */
hashtable->inputslot = NULL;
hashtable->in_hash_funcs = NULL;
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 11fe866..3a335d8 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -726,7 +726,6 @@ typedef struct TupleHashTableData
Oid *tab_collations; /* collations for hash and comparison */
MemoryContext tablecxt; /* memory context containing table */
MemoryContext tempcxt; /* context for function evaluations */
- Size entrysize; /* actual size to make each hash entry */
TupleTableSlot *tableslot; /* slot for referencing table entries */
/* The following fields are set transiently for each table search: */
TupleTableSlot *inputslot; /* current input tuple's slot */
--
2.7.4
--M0YLxmUXciMpOLPE
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0007-Update-comment-obsolete-since-69c3936a.patch"
^ permalink raw reply [nested|flat] 23+ messages in thread
* [PATCH v4 6/7] TupleHashTable.entrysize was unused except for instrumentation..
@ 2020-02-15 23:19 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 23+ messages in thread
From: Justin Pryzby @ 2020-02-15 23:19 UTC (permalink / raw)
---
src/backend/executor/execGrouping.c | 1 -
src/include/nodes/execnodes.h | 1 -
2 files changed, 2 deletions(-)
diff --git a/src/backend/executor/execGrouping.c b/src/backend/executor/execGrouping.c
index de0205f..d76a630 100644
--- a/src/backend/executor/execGrouping.c
+++ b/src/backend/executor/execGrouping.c
@@ -186,7 +186,6 @@ BuildTupleHashTableExt(PlanState *parent,
hashtable->tab_collations = collations;
hashtable->tablecxt = tablecxt;
hashtable->tempcxt = tempcxt;
- hashtable->entrysize = entrysize;
hashtable->tableslot = NULL; /* will be made on first lookup */
hashtable->inputslot = NULL;
hashtable->in_hash_funcs = NULL;
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 87b1127..3f9ac5e 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -727,7 +727,6 @@ typedef struct TupleHashTableData
Oid *tab_collations; /* collations for hash and comparison */
MemoryContext tablecxt; /* memory context containing table */
MemoryContext tempcxt; /* context for function evaluations */
- Size entrysize; /* actual size to make each hash entry */
TupleTableSlot *tableslot; /* slot for referencing table entries */
/* The following fields are set transiently for each table search: */
TupleTableSlot *inputslot; /* current input tuple's slot */
--
2.7.4
--jI8keyz6grp/JLjh
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0007-Update-comment-obsolete-since-69c3936a.patch"
^ permalink raw reply [nested|flat] 23+ messages in thread
* [PATCH v3 6/7] TupleHashTable.entrysize was unused except for instrumentation..
@ 2020-02-15 23:19 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 23+ messages in thread
From: Justin Pryzby @ 2020-02-15 23:19 UTC (permalink / raw)
---
src/backend/executor/execGrouping.c | 1 -
src/include/nodes/execnodes.h | 1 -
2 files changed, 2 deletions(-)
diff --git a/src/backend/executor/execGrouping.c b/src/backend/executor/execGrouping.c
index de0205f..d76a630 100644
--- a/src/backend/executor/execGrouping.c
+++ b/src/backend/executor/execGrouping.c
@@ -186,7 +186,6 @@ BuildTupleHashTableExt(PlanState *parent,
hashtable->tab_collations = collations;
hashtable->tablecxt = tablecxt;
hashtable->tempcxt = tempcxt;
- hashtable->entrysize = entrysize;
hashtable->tableslot = NULL; /* will be made on first lookup */
hashtable->inputslot = NULL;
hashtable->in_hash_funcs = NULL;
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index df56330..6eac7de 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -724,7 +724,6 @@ typedef struct TupleHashTableData
Oid *tab_collations; /* collations for hash and comparison */
MemoryContext tablecxt; /* memory context containing table */
MemoryContext tempcxt; /* context for function evaluations */
- Size entrysize; /* actual size to make each hash entry */
TupleTableSlot *tableslot; /* slot for referencing table entries */
/* The following fields are set transiently for each table search: */
TupleTableSlot *inputslot; /* current input tuple's slot */
--
2.7.4
--SNIs70sCzqvszXB4
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v3-0007-Update-comment-obsolete-since-69c3936a.patch"
^ permalink raw reply [nested|flat] 23+ messages in thread
* [PATCH v6 6/7] TupleHashTable.entrysize was unused except for instrumentation..
@ 2020-02-15 23:19 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 23+ messages in thread
From: Justin Pryzby @ 2020-02-15 23:19 UTC (permalink / raw)
---
src/backend/executor/execGrouping.c | 1 -
src/include/nodes/execnodes.h | 1 -
2 files changed, 2 deletions(-)
diff --git a/src/backend/executor/execGrouping.c b/src/backend/executor/execGrouping.c
index 009d27b9a8..98fd4bf8bd 100644
--- a/src/backend/executor/execGrouping.c
+++ b/src/backend/executor/execGrouping.c
@@ -183,7 +183,6 @@ BuildTupleHashTableExt(PlanState *parent,
hashtable->tab_collations = collations;
hashtable->tablecxt = tablecxt;
hashtable->tempcxt = tempcxt;
- hashtable->entrysize = entrysize;
hashtable->tableslot = NULL; /* will be made on first lookup */
hashtable->inputslot = NULL;
hashtable->in_hash_funcs = NULL;
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 11fe866c1c..3a335d87a0 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -726,7 +726,6 @@ typedef struct TupleHashTableData
Oid *tab_collations; /* collations for hash and comparison */
MemoryContext tablecxt; /* memory context containing table */
MemoryContext tempcxt; /* context for function evaluations */
- Size entrysize; /* actual size to make each hash entry */
TupleTableSlot *tableslot; /* slot for referencing table entries */
/* The following fields are set transiently for each table search: */
TupleTableSlot *inputslot; /* current input tuple's slot */
--
2.17.0
--xHbokkKX1kTiQeDC
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0007-Update-comment-obsolete-since-69c3936a.patch"
^ permalink raw reply [nested|flat] 23+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 23+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2024-08-12 08:34 =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
0 siblings, 1 reply; 23+ messages in thread
From: =?ISO-8859-1?B?Y2NhNTUwNw==?= @ 2024-08-12 08:34 UTC (permalink / raw)
To: =?ISO-8859-1?B?QmVydHJhbmQgRHJvdXZvdA==?= <[email protected]>; +Cc: =?ISO-8859-1?B?TWljaGFlbCBQYXF1aWVy?= <[email protected]>; =?ISO-8859-1?B?cGdzcWwtaGFja2Vycw==?= <[email protected]>
Hi,
4, 5 ===
> if (SnapBuildCurrentState(builder) < SNAPBUILD_BUILDING_SNAPSHOT ||
> (SnapBuildCurrentState(builder) == SNAPBUILD_BUILDING_SNAPSHOT && info != XLOG_HEAP_INPLACE) ||
> ctx->fast_forward)
> return;
I think during fast forward, we also need handle the xlog that marks a transaction
as catalog modifying, or the snapshot might lose some transactions?
> That way we'd still rely on what's being done in the XLOG_HEAP_INPLACE case
+ if (SnapBuildCurrentState(builder) >= SNAPBUILD_BUILDING_SNAPSHOT)
+ {
+ /* Currently only XLOG_HEAP_INPLACE means a catalog modifying */
+ if (info == XLOG_HEAP_INPLACE && TransactionIdIsValid(xid))
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
We only call ReorderBufferXidSetCatalogChanges() for the xlog that marks a transaction as catalog
modifying, and we don't care about the other steps being done in the xlog, so I think the current
approach is ok.
--
Regards,
ChangAo Chen
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2024-08-12 10:35 Bertrand Drouvot <[email protected]>
parent: =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
0 siblings, 2 replies; 23+ messages in thread
From: Bertrand Drouvot @ 2024-08-12 10:35 UTC (permalink / raw)
To: cca5507 <[email protected]>; +Cc: Michael Paquier <[email protected]>; pgsql-hackers <[email protected]>
Hi,
On Mon, Aug 12, 2024 at 04:34:25PM +0800, cca5507 wrote:
> Hi,
>
>
> 4, 5 ===
>
>
> > if (SnapBuildCurrentState(builder) < SNAPBUILD_BUILDING_SNAPSHOT ||
> > (SnapBuildCurrentState(builder) == SNAPBUILD_BUILDING_SNAPSHOT && info != XLOG_HEAP_INPLACE) ||
> > ctx->fast_forward)
> > return;
>
>
>
> I think during fast forward, we also need handle the xlog that marks a transaction
> as catalog modifying, or the snapshot might lose some transactions?
I think it's fine to skip during fast forward as we are not generating logical
changes. It's done that way in master, in your proposal and in my "if" proposals.
Note that my proposals related to the if conditions are for heap2_decode and
heap_decode (not xact_decode).
> > That way we'd still rely on what's being done in the XLOG_HEAP_INPLACE case
>
>
> + if (SnapBuildCurrentState(builder) >= SNAPBUILD_BUILDING_SNAPSHOT)
> + {
> + /* Currently only XLOG_HEAP_INPLACE means a catalog modifying */
> + if (info == XLOG_HEAP_INPLACE && TransactionIdIsValid(xid))
> + ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
> + }
>
>
>
> We only call ReorderBufferXidSetCatalogChanges() for the xlog that marks a transaction as catalog
> modifying, and we don't care about the other steps being done in the xlog, so I think the current
> approach is ok.
Yeah, I think your proposal does not do anything wrong. I just prefer to put
everything in a single if condition (as per my proposal) so that we can jump
directly in the appropriate case. I think that way the code is easier to maintain
instead of having to deal with the same info values in distinct places.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2024-08-12 11:38 =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
parent: Bertrand Drouvot <[email protected]>
1 sibling, 0 replies; 23+ messages in thread
From: =?ISO-8859-1?B?Y2NhNTUwNw==?= @ 2024-08-12 11:38 UTC (permalink / raw)
To: =?ISO-8859-1?B?QmVydHJhbmQgRHJvdXZvdA==?= <[email protected]>; +Cc: =?ISO-8859-1?B?TWljaGFlbCBQYXF1aWVy?= <[email protected]>; =?ISO-8859-1?B?cGdzcWwtaGFja2Vycw==?= <[email protected]>
Hi,
Thanks for the comments!
- I think it's fine to skip during fast forward as we are not generating logical
- changes. It's done that way in master, in your proposal and in my "if" proposals.
- Note that my proposals related to the if conditions are for heap2_decode and
- heap_decode (not xact_decode).
But note that in xact_decode(), case XLOG_XACT_INVALIDATIONS, we call
ReorderBufferXidSetCatalogChanges() even if we are fast-forwarding, it might
be better to be consistent.
In addition, we don't decode anything during fast forward, but the snapshot might
serialize to disk. If we skip calling ReorderBufferXidSetCatalogChanges(), the snapshot
may be wrong on disk.
--
Regards,
ChangAo Chen
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2024-08-13 04:23 =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
parent: Bertrand Drouvot <[email protected]>
1 sibling, 2 replies; 23+ messages in thread
From: =?ISO-8859-1?B?Y2NhNTUwNw==?= @ 2024-08-13 04:23 UTC (permalink / raw)
To: =?ISO-8859-1?B?QmVydHJhbmQgRHJvdXZvdA==?= <[email protected]>; +Cc: =?ISO-8859-1?B?TWljaGFlbCBQYXF1aWVy?= <[email protected]>; =?ISO-8859-1?B?cGdzcWwtaGFja2Vycw==?= <[email protected]>
Hi,
I refactor the code and fix the git apply warning according to [1].
Here are the new version patches.
--
Regards,
ChangAo Chen
[1] https://www.postgresql.org/message-id/Zrmh7X8jYCbFYXjH%40ip-10-97-1-34.eu-west-3.compute.internal
Attachments:
[application/octet-stream] v3-0002-Add-test-case-snapshot_build-for-test_decoding.patch (5.0K, ../../[email protected]/3-v3-0002-Add-test-case-snapshot_build-for-test_decoding.patch)
download | inline diff:
From 51838f4cba785231ad6bd6d0f1bc105b9778392f Mon Sep 17 00:00:00 2001
From: ChangAo Chen <[email protected]>
Date: Sat, 10 Aug 2024 17:34:26 +0800
Subject: [PATCH v3 2/2] Add test case snapshot_build for test_decoding.
---
contrib/test_decoding/Makefile | 2 +-
.../test_decoding/expected/snapshot_build.out | 33 +++++++++++++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/specs/snapshot_build.spec | 46 +++++++++++++++++++
4 files changed, 81 insertions(+), 1 deletion(-)
create mode 100644 contrib/test_decoding/expected/snapshot_build.out
create mode 100644 contrib/test_decoding/specs/snapshot_build.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..8113e2d99c 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -9,7 +9,7 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
- skip_snapshot_restore
+ skip_snapshot_restore snapshot_build
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/snapshot_build.out b/contrib/test_decoding/expected/snapshot_build.out
new file mode 100644
index 0000000000..0fcf20cce8
--- /dev/null
+++ b/contrib/test_decoding/expected/snapshot_build.out
@@ -0,0 +1,33 @@
+Parsed test spec with 4 sessions
+
+starting permutation: s1_begin s1_insert s2_init s3_begin s3_insert s4_create s1_commit s4_begin s4_insert s3_commit s4_commit s2_get_changes
+step s1_begin: BEGIN;
+step s1_insert: INSERT INTO tbl1 VALUES (1);
+step s2_init: SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'test_decoding'); <waiting ...>
+step s3_begin: BEGIN;
+step s3_insert: INSERT INTO tbl1 VALUES (1);
+step s4_create: CREATE TABLE tbl2 (val1 integer);
+step s1_commit: COMMIT;
+step s4_begin: BEGIN;
+step s4_insert: INSERT INTO tbl2 VALUES (1);
+step s3_commit: COMMIT;
+step s2_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s4_commit: COMMIT;
+step s2_get_changes: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+------------------------------------------
+BEGIN
+table public.tbl2: INSERT: val1[integer]:1
+COMMIT
+(3 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc81a2..2b7e80ba71 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -63,6 +63,7 @@ tests += {
'twophase_snapshot',
'slot_creation_error',
'skip_snapshot_restore',
+ 'snapshot_build',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/snapshot_build.spec b/contrib/test_decoding/specs/snapshot_build.spec
new file mode 100644
index 0000000000..334531dd21
--- /dev/null
+++ b/contrib/test_decoding/specs/snapshot_build.spec
@@ -0,0 +1,46 @@
+# Test snapshot build correctly, it must track committed transactions during BUILDING_SNAPSHOT
+
+setup
+{
+ DROP TABLE IF EXISTS tbl1;
+ DROP TABLE IF EXISTS tbl2;
+ CREATE TABLE tbl1 (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl1;
+ DROP TABLE tbl2;
+ SELECT 'stop' FROM pg_drop_replication_slot('isolation_slot');
+}
+
+session "s1"
+setup { SET synchronous_commit=on; }
+step "s1_begin" { BEGIN; }
+step "s1_insert" { INSERT INTO tbl1 VALUES (1); }
+step "s1_commit" { COMMIT; }
+
+session "s2"
+setup { SET synchronous_commit=on; }
+step "s2_init" { SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'test_decoding'); }
+step "s2_get_changes" { SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert" { INSERT INTO tbl1 VALUES (1); }
+step "s3_commit" { COMMIT; }
+
+session "s4"
+setup { SET synchronous_commit=on; }
+step "s4_create" { CREATE TABLE tbl2 (val1 integer); }
+step "s4_begin" { BEGIN; }
+step "s4_insert" { INSERT INTO tbl2 VALUES (1); }
+step "s4_commit" { COMMIT; }
+
+# T1: s1_begin -> s1_insert -> BUILDING_SNAPSHOT -> s1_commit -> FULL_SNAPSHOT
+# T2: BUILDING_SNAPSHOT -> s3_begin -> s3_insert -> FULL_SNAPSHOT -> s3_commit -> CONSISTENT
+# T3: BUILDING_SNAPSHOT -> s4_create -> FULL_SNAPSHOT
+# T4: FULL_SNAPSHOT -> s4_begin -> s4_insert -> CONSISTENT -> s4_commit
+# The snapshot must track T3 or the replay of T4 will fail because its snapshot cannot see tbl2
+permutation "s1_begin" "s1_insert" "s2_init" "s3_begin" "s3_insert" "s4_create" "s1_commit" "s4_begin" "s4_insert" "s3_commit" "s4_commit" "s2_get_changes"
--
2.34.1
[application/octet-stream] v4-0001-Track-transactions-committed-in-BUILDING_SNAPSHOT.patch (5.4K, ../../[email protected]/4-v4-0001-Track-transactions-committed-in-BUILDING_SNAPSHOT.patch)
download | inline diff:
From b35711f5ef0941f79127acbec55a8ede5e27a5eb Mon Sep 17 00:00:00 2001
From: ChangAo Chen <[email protected]>
Date: Tue, 13 Aug 2024 11:45:07 +0800
Subject: [PATCH v4] Track transactions committed in BUILDING_SNAPSHOT.
The historic snapshot previously didn't track transactions committed
in BUILDING_SNAPSHOT, and this might result in a transaction taking
an incorrect snapshot and logical decoding being interrupted. So we
need track these transactions.
We also need handle the xlog that marks a transaction as containing
catalog changes in BUILDING_SNAPSHOT because the historic snapshot
only tracks catalog modifying transactions.
---
src/backend/replication/logical/decode.c | 45 +++++++++++++++++++++---
1 file changed, 40 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index d687ceee33..6df558ad18 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -206,12 +206,16 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
uint8 info = XLogRecGetInfo(r) & XLOG_XACT_OPMASK;
/*
- * If the snapshot isn't yet fully built, we cannot decode anything, so
- * bail out.
+ * If the snapshot hasn't started building yet, the transaction won't be
+ * decoded or tracked by the snapshot, so bail out.
*/
- if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT)
+ if (SnapBuildCurrentState(builder) < SNAPBUILD_BUILDING_SNAPSHOT)
return;
+ /*
+ * Note that if the snapshot isn't yet fully built, the xlog is only used
+ * to build the snapshot and won't be decoded.
+ */
switch (info)
{
case XLOG_XACT_COMMIT:
@@ -282,9 +286,11 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
TransactionId xid;
xl_xact_invals *invals;
+ bool has_snapshot;
xid = XLogRecGetXid(r);
invals = (xl_xact_invals *) XLogRecGetData(r);
+ has_snapshot = SnapBuildCurrentState(builder) >= SNAPBUILD_FULL_SNAPSHOT;
/*
* Execute the invalidations for xid-less transactions,
@@ -293,7 +299,7 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
*/
if (TransactionIdIsValid(xid))
{
- if (!ctx->fast_forward)
+ if (!ctx->fast_forward && has_snapshot)
ReorderBufferAddInvalidations(reorder, xid,
buf->origptr,
invals->nmsgs,
@@ -301,7 +307,7 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
ReorderBufferXidSetCatalogChanges(ctx->reorder, xid,
buf->origptr);
}
- else if (!ctx->fast_forward)
+ else if (!ctx->fast_forward && has_snapshot)
ReorderBufferImmediateInvalidation(ctx->reorder,
invals->nmsgs,
invals->msgs);
@@ -407,6 +413,8 @@ heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
uint8 info = XLogRecGetInfo(buf->record) & XLOG_HEAP_OPMASK;
TransactionId xid = XLogRecGetXid(buf->record);
SnapBuild *builder = ctx->snapshot_builder;
+ /* True if the xlog marks the transaction as containing catalog changes */
+ bool set_catalog_changes = (info == XLOG_HEAP2_NEW_CID);
ReorderBufferProcessXid(ctx->reorder, xid, buf->origptr);
@@ -416,7 +424,19 @@ heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT ||
ctx->fast_forward)
+ {
+ /*
+ * If the transaction contains catalog changes, we need mark it in
+ * reorder buffer before return as the snapshot only tracks catalog
+ * modifying transactions. The transaction before BUILDING_SNAPSHOT
+ * won't be tracked anyway(see SnapBuildCommitTxn), so skip it.
+ */
+ if (set_catalog_changes && TransactionIdIsValid(xid) &&
+ SnapBuildCurrentState(builder) >= SNAPBUILD_BUILDING_SNAPSHOT)
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+
return;
+ }
switch (info)
{
@@ -466,6 +486,8 @@ heap_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
uint8 info = XLogRecGetInfo(buf->record) & XLOG_HEAP_OPMASK;
TransactionId xid = XLogRecGetXid(buf->record);
SnapBuild *builder = ctx->snapshot_builder;
+ /* True if the xlog marks the transaction as containing catalog changes */
+ bool set_catalog_changes = (info == XLOG_HEAP_INPLACE);
ReorderBufferProcessXid(ctx->reorder, xid, buf->origptr);
@@ -475,7 +497,19 @@ heap_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT ||
ctx->fast_forward)
+ {
+ /*
+ * If the transaction contains catalog changes, we need mark it in
+ * reorder buffer before return as the snapshot only tracks catalog
+ * modifying transactions. The transaction before BUILDING_SNAPSHOT
+ * won't be tracked anyway(see SnapBuildCommitTxn), so skip it.
+ */
+ if (set_catalog_changes && TransactionIdIsValid(xid) &&
+ SnapBuildCurrentState(builder) >= SNAPBUILD_BUILDING_SNAPSHOT)
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+
return;
+ }
switch (info)
{
@@ -1301,6 +1335,7 @@ DecodeTXNNeedSkip(LogicalDecodingContext *ctx, XLogRecordBuffer *buf,
Oid txn_dbid, RepOriginId origin_id)
{
if (SnapBuildXactNeedsSkip(ctx->snapshot_builder, buf->origptr) ||
+ SnapBuildCurrentState(ctx->snapshot_builder) < SNAPBUILD_CONSISTENT ||
(txn_dbid != InvalidOid && txn_dbid != ctx->slot->data.database) ||
FilterByOrigin(ctx, origin_id))
return true;
--
2.34.1
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2024-08-13 06:19 Bertrand Drouvot <[email protected]>
parent: =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
1 sibling, 0 replies; 23+ messages in thread
From: Bertrand Drouvot @ 2024-08-13 06:19 UTC (permalink / raw)
To: cca5507 <[email protected]>; +Cc: Michael Paquier <[email protected]>; pgsql-hackers <[email protected]>
Hi,
On Tue, Aug 13, 2024 at 12:23:04PM +0800, cca5507 wrote:
> Hi,
>
> I refactor the code and fix the git apply warning according to [1].
>
>
> Here are the new version patches.
Thanks!
1 ===
+ /* True if the xlog marks the transaction as containing catalog changes */
+ bool set_catalog_changes = (info == XLOG_HEAP2_NEW_CID);
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT ||
ctx->fast_forward)
+ {
+ /*
+ * If the transaction contains catalog changes, we need mark it in
+ * reorder buffer before return as the snapshot only tracks catalog
+ * modifying transactions. The transaction before BUILDING_SNAPSHOT
+ * won't be tracked anyway(see SnapBuildCommitTxn), so skip it.
+ */
+ if (set_catalog_changes && TransactionIdIsValid(xid) &&
+ SnapBuildCurrentState(builder) >= SNAPBUILD_BUILDING_SNAPSHOT)
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+
return;
+ }
I still prefer to replace the above with:
if (SnapBuildCurrentState(builder) < SNAPBUILD_BUILDING_SNAPSHOT ||
(SnapBuildCurrentState(builder) == SNAPBUILD_BUILDING_SNAPSHOT && info != XLOG_HEAP2_NEW_CID) ||
ctx->fast_forward)
return;
Let's see what others think.
2 ===
+ /* True if the xlog marks the transaction as containing catalog changes */
+ bool set_catalog_changes = (info == XLOG_HEAP_INPLACE);
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT ||
ctx->fast_forward)
+ {
+ /*
+ * If the transaction contains catalog changes, we need mark it in
+ * reorder buffer before return as the snapshot only tracks catalog
+ * modifying transactions. The transaction before BUILDING_SNAPSHOT
+ * won't be tracked anyway(see SnapBuildCommitTxn), so skip it.
+ */
+ if (set_catalog_changes && TransactionIdIsValid(xid) &&
+ SnapBuildCurrentState(builder) >= SNAPBUILD_BUILDING_SNAPSHOT)
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+
return;
+ }
I still prefer to replace the above with:
if (SnapBuildCurrentState(builder) < SNAPBUILD_BUILDING_SNAPSHOT ||
(SnapBuildCurrentState(builder) == SNAPBUILD_BUILDING_SNAPSHOT && info != XLOG_HEAP_INPLACE) ||
ctx->fast_forward)
return;
Let's see what others think.
3 ===
I re-read your comments in [0] and it looks like you've concern about
the 2 "if" I'm proposing above and the fast forward handling. Is that the case
or is your fast forward concern unrelated to my proposals?
Not sure what happened but it looks like your reply in [0] is not part of the
initial thread [1], but created a new thread instead, making the whole
conversation difficult to follow.
[0]: https://www.postgresql.org/message-id/tencent_8DEC9842690A9B6AFD52D4659EF0700E9409%40qq.com
[1]: https://www.postgresql.org/message-id/flat/tencent_6AAF072A7623A11A85C0B5FD290232467808%40qq.com
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2025-03-26 10:15 Ajin Cherian <[email protected]>
0 siblings, 1 reply; 23+ messages in thread
From: Ajin Cherian @ 2025-03-26 10:15 UTC (permalink / raw)
To: cca5507 <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Michael Paquier <[email protected]>; pgsql-hackers <[email protected]>
On Wed, Mar 26, 2025 at 9:11 PM cca5507 <[email protected]> wrote:
>
> Hi,
>
> Thanks for the comments!
>
> Here are the new version patches, I think it will be more clear.
>
> --
+ {
+ /*
+ * Note that during or after BUILDING_SNAPSHOT, we need handle the xlog
+ * that might mark a transaction as catalog modifying because
the snapshot
+ * only tracks catalog modifying transactions. The transaction before
+ * BUILDING_SNAPSHOT will not be tracked anyway(see
SnapBuildCommitTxn()
+ * for details), so just return.
+ */
+ if (SnapBuildCurrentState(builder) >= SNAPBUILD_BUILDING_SNAPSHOT)
+ {
+ /* Currently only XLOG_HEAP2_NEW_CID means a catalog modifying */
+ if (info == XLOG_HEAP2_NEW_CID && TransactionIdIsValid(xid))
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid,
buf->origptr);
+ }
Any reason why you avoided calling SnapBuildProcessNewCid here and
only called ReorderBufferXidSetCatalogChanges? If any, please mention
in the comments the reason.
regards,
Ajin Cherian
Fujitsu Australia
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2025-03-26 12:51 =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
parent: Ajin Cherian <[email protected]>
0 siblings, 0 replies; 23+ messages in thread
From: =?ISO-8859-1?B?Y2NhNTUwNw==?= @ 2025-03-26 12:51 UTC (permalink / raw)
To: =?ISO-8859-1?B?QWppbiBDaGVyaWFu?= <[email protected]>; +Cc: =?ISO-8859-1?B?QmVydHJhbmQgRHJvdXZvdA==?= <[email protected]>; =?ISO-8859-1?B?TWljaGFlbCBQYXF1aWVy?= <[email protected]>; =?ISO-8859-1?B?cGdzcWwtaGFja2Vycw==?= <[email protected]>
Hi,
There are 3 threads currently.
The latest patches are v4-0001 and v3-0002 in [2].
[1]https://www.postgresql.org/message-id/flat/[email protected]
[2]https://www.postgresql.org/message-id/flat/[email protected]
[3]https://www.postgresql.org/message-id/flat/[email protected]
--
Regards,
ChangAo Chen
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2025-03-27 02:30 Ajin Cherian <[email protected]>
0 siblings, 1 reply; 23+ messages in thread
From: Ajin Cherian @ 2025-03-27 02:30 UTC (permalink / raw)
To: cca5507 <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Michael Paquier <[email protected]>; pgsql-hackers <[email protected]>
On Thu, Mar 27, 2025 at 1:13 PM cca5507 <[email protected]> wrote:
>
> Hi,
>
> - IIUC your "fast forward" concern is not related to this particular thread but you
> - think it's already an issue on the master branch (outside of the BUILDING_SNAPSHOT
> - handling we are discussing here), is that correct? (that's also what your coding
> - changes makes me think of). If so, I'd suggest to open a dedicated thread for that
> - particular "fast forward" point and do the coding in the current thread as if the
> - fast forward is not an issue.
>
> - Does that make sense?
>
> Yes.
>
> But I think the v4-0001 in [1] is fine.
>
@@ -1301,6 +1335,7 @@ DecodeTXNNeedSkip(LogicalDecodingContext *ctx,
XLogRecordBuffer *buf,
Oid txn_dbid, RepOriginId origin_id)
{
if (SnapBuildXactNeedsSkip(ctx->snapshot_builder, buf->origptr) ||
+ SnapBuildCurrentState(ctx->snapshot_builder) < SNAPBUILD_CONSISTENT ||
(txn_dbid != InvalidOid && txn_dbid != ctx->slot->data.database) ||
FilterByOrigin(ctx, origin_id))
return true;
I don't see an explanation as to why this change was added? Maybe I
missed it in the multiple threads. With this, we are no longer
decoding changes which were in the state SNAPBUILD_FULL_SNAPSHOT. Why
did that change?
regards,
Ajin Cherian
Fujitsu Australia
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2025-03-27 03:47 =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
parent: Ajin Cherian <[email protected]>
0 siblings, 1 reply; 23+ messages in thread
From: =?ISO-8859-1?B?Y2NhNTUwNw==?= @ 2025-03-27 03:47 UTC (permalink / raw)
To: =?ISO-8859-1?B?QWppbiBDaGVyaWFu?= <[email protected]>; +Cc: =?ISO-8859-1?B?QmVydHJhbmQgRHJvdXZvdA==?= <[email protected]>; =?ISO-8859-1?B?TWljaGFlbCBQYXF1aWVy?= <[email protected]>; =?ISO-8859-1?B?cGdzcWwtaGFja2Vycw==?= <[email protected]>
Hi,
According to the comment above DecodeTXNNeedSkip(), transactions committed before SNAPBUILD_CONSISTENT state won't be decoded. It's important for initial table data synchronization because the table sync workers use the consistent snapshot to copy data so transactions before SNAPBUILD_CONSISTENT are already included in the initial data.
This change may be redundant with SnapBuildXactNeedsSkip(), I add just for safe.
--
Regards,
ChangAo Chen
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2025-03-27 04:57 Ajin Cherian <[email protected]>
parent: =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
0 siblings, 1 reply; 23+ messages in thread
From: Ajin Cherian @ 2025-03-27 04:57 UTC (permalink / raw)
To: cca5507 <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Michael Paquier <[email protected]>; pgsql-hackers <[email protected]>
On Thu, Mar 27, 2025 at 2:47 PM cca5507 <[email protected]> wrote:
>
> Hi,
>
> According to the comment above DecodeTXNNeedSkip(), transactions committed before SNAPBUILD_CONSISTENT state won't be decoded. It's important for initial table data synchronization because the table sync workers use the consistent snapshot to copy data so transactions before SNAPBUILD_CONSISTENT are already included in the initial data.
>
> This change may be redundant with SnapBuildXactNeedsSkip(), I add just for safe.
>
> --
I understand your explanation but was there a bug on HEAD that this is
solving? If so, can you write a test case to show this? This will make
it more convincing that this change as well is required. Your other
change has a test case to confirm, but this doesn't.
regards,
Ajin Cherian
Fujitsu Australia
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2025-03-27 07:26 =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
parent: Ajin Cherian <[email protected]>
0 siblings, 0 replies; 23+ messages in thread
From: =?ISO-8859-1?B?Y2NhNTUwNw==?= @ 2025-03-27 07:26 UTC (permalink / raw)
To: =?ISO-8859-1?B?QWppbiBDaGVyaWFu?= <[email protected]>; +Cc: =?ISO-8859-1?B?QmVydHJhbmQgRHJvdXZvdA==?= <[email protected]>; =?ISO-8859-1?B?TWljaGFlbCBQYXF1aWVy?= <[email protected]>; =?ISO-8859-1?B?cGdzcWwtaGFja2Vycw==?= <[email protected]>
Hi,
This change doesn't solve a bug. But I think it makes the code and comments more consistent. I currently have no idea about how to test it.
--
Regards,
ChangAo Chen
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2025-06-27 10:29 Ajin Cherian <[email protected]>
parent: =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
1 sibling, 1 reply; 23+ messages in thread
From: Ajin Cherian @ 2025-06-27 10:29 UTC (permalink / raw)
To: cca5507 <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Michael Paquier <[email protected]>; pgsql-hackers <[email protected]>
On Fri, Jun 27, 2025 at 3:48 PM cca5507 <[email protected]> wrote:
>
> Hi,
>
> I refactor the code and fix the git apply warning according to [1].
>
> Here are the new version patches.
>
> --
> Regards,
> ChangAo Chen
>
> [1] https://www.postgresql.org/message-id/Zrmh7X8jYCbFYXjH%40ip-10-97-1-34.eu-west-3.compute.internal
I see this problem is similar to the bug reported in [1], and your fix
also addresses the issue reported there. Although I like your approach
of tracking changes starting from the BUILDING_SNAPSHOT state, I’d
like to suggest an alternative.
While debugging that issue, my plan was not to track catalog changes
prior to SNAPBUILD_CONSISTENT, but instead to ensure we don’t use
snapshots built before SNAPBUILD_CONSISTENT, since we don’t track
catalog changes in those states. We should discard previously built
snapshots and rebuild them once we reach the SNAPBUILD_CONSISTENT
state. At that point, all necessary transactions would have been
committed, and builder->xmin would have advanced enough to decode all
transactions from then on.
The problem is that previously built snapshots hang around without the
latest xmin and xmax, and we tend to reuse them. We should ensure that
all txn->base_snapshot and builder->snapshot snapshots built in the
SNAPBUILD_FULL_SNAPSHOT state are rebuilt once we reach
SNAPBUILD_CONSISTENT. For this, we need to track when the snapshot was
built. There is already a field in ReorderBufferTXN -
'base_snapshot_lsn' which we can use. If base_snapshot_lsn <
builder->start_decoding_at, then we should rebuild the snapshot. Just
a thought.
regards,
Ajin Cherian
Fujitsu Australia
[1] - https://www.postgresql.org/message-id/18509-983f064d174ea880%40postgresql.org
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2025-06-28 03:44 =?gb18030?B?Y2NhNTUwNw==?= <[email protected]>
parent: Ajin Cherian <[email protected]>
0 siblings, 1 reply; 23+ messages in thread
From: =?gb18030?B?Y2NhNTUwNw==?= @ 2025-06-28 03:44 UTC (permalink / raw)
To: =?gb18030?B?QWppbiBDaGVyaWFu?= <[email protected]>; +Cc: =?gb18030?B?QmVydHJhbmQgRHJvdXZvdA==?= <[email protected]>; =?gb18030?B?TWljaGFlbCBQYXF1aWVy?= <[email protected]>; =?gb18030?B?cGdzcWwtaGFja2Vycw==?= <[email protected]>
Hi,
If I understand correctly, this may break the basic principle:
The aim is to build a snapshot that behaves the same as a freshly taken MVCC snapshot would have at the time the XLogRecord was generated.
--
Regards,
ChangAo Chen
------------------ Original ------------------
From: "Ajin Cherian" <[email protected]>;
Date: Fri, Jun 27, 2025 06:29 PM
To: "cca5507"<[email protected]>;
Cc: "Bertrand Drouvot"<[email protected]>;"Michael Paquier"<[email protected]>;"pgsql-hackers"<[email protected]>;
Subject: Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
On Fri, Jun 27, 2025 at 3:48 PM cca5507 <[email protected]> wrote:
>
> Hi,
>
> I refactor the code and fix the git apply warning according to [1].
>
> Here are the new version patches.
>
> --
> Regards,
> ChangAo Chen
>
> [1] https://www.postgresql.org/message-id/Zrmh7X8jYCbFYXjH%40ip-10-97-1-34.eu-west-3.compute.internal
I see this problem is similar to the bug reported in [1], and your fix
also addresses the issue reported there. Although I like your approach
of tracking changes starting from the BUILDING_SNAPSHOT state, I’d
like to suggest an alternative.
While debugging that issue, my plan was not to track catalog changes
prior to SNAPBUILD_CONSISTENT, but instead to ensure we don’t use
snapshots built before SNAPBUILD_CONSISTENT, since we don’t track
catalog changes in those states. We should discard previously built
snapshots and rebuild them once we reach the SNAPBUILD_CONSISTENT
state. At that point, all necessary transactions would have been
committed, and builder->xmin would have advanced enough to decode all
transactions from then on.
The problem is that previously built snapshots hang around without the
latest xmin and xmax, and we tend to reuse them. We should ensure that
all txn->base_snapshot and builder->snapshot snapshots built in the
SNAPBUILD_FULL_SNAPSHOT state are rebuilt once we reach
SNAPBUILD_CONSISTENT. For this, we need to track when the snapshot was
built. There is already a field in ReorderBufferTXN -
'base_snapshot_lsn' which we can use. If base_snapshot_lsn <
builder->start_decoding_at, then we should rebuild the snapshot. Just
a thought.
regards,
Ajin Cherian
Fujitsu Australia
[1] - https://www.postgresql.org/message-id/18509-983f064d174ea880%40postgresql.org
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state
@ 2025-07-01 23:47 Michael Paquier <[email protected]>
parent: =?gb18030?B?Y2NhNTUwNw==?= <[email protected]>
0 siblings, 1 reply; 23+ messages in thread
From: Michael Paquier @ 2025-07-01 23:47 UTC (permalink / raw)
To: cca5507 <[email protected]>; +Cc: Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; pgsql-hackers <[email protected]>
On Sat, Jun 28, 2025 at 11:44:52AM +0800, cca5507 wrote:
> If I understand correctly, this may break the basic principle:
>
> The aim is to build a snapshot that behaves the same as a
> freshly taken MVCC snapshot would have at the time the
> XLogRecord was generated.
Please note that we prefer bottom-posting when sending messages on
pgsql-hackers and the community mailing lists, as defined in this
link:
https://en.wikipedia.org/wiki/Posting_style#Bottom-posting
Top-posting, as you did in your last email, is breaking the logic and
flow of the discussion.
Some more useful documentation from the PostgreSQL wiki:
https://wiki.postgresql.org/wiki/Mailing_Lists
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 23+ messages in thread
* Re: Historic snapshot doesn't track txns committed inBUILDING_SNAPSHOT state
@ 2025-11-21 08:16 =?utf-8?B?Y2NhNTUwNw==?= <[email protected]>
parent: Michael Paquier <[email protected]>
0 siblings, 0 replies; 23+ messages in thread
From: =?utf-8?B?Y2NhNTUwNw==?= @ 2025-11-21 08:16 UTC (permalink / raw)
To: =?utf-8?B?TWljaGFlbCBQYXF1aWVy?= <[email protected]>; +Cc: =?utf-8?B?QWppbiBDaGVyaWFu?= <[email protected]>; =?utf-8?B?QmVydHJhbmQgRHJvdXZvdA==?= <[email protected]>; =?utf-8?B?cGdzcWwtaGFja2Vycw==?= <[email protected]>
Hi all,
Attach the rebased patches.
Some useful link:
Discussion:
https://www.postgresql.org/message-id/flat/[email protected]
https://www.postgresql.org/message-id/flat/[email protected]
https://www.postgresql.org/message-id/flat/[email protected]
Bug Report:
https://www.postgresql.org/message-id/flat/18509-983f064d174ea880%40postgresql.org
https://www.postgresql.org/message-id/flat/19109-4a9ce36715d69632%40postgresql.org
CF Entry:
https://commitfest.postgresql.org/patch/5029/
--
Regards,
ChangAo Chen
Attachments:
[application/octet-stream] v6-0001-Track-transactions-committed-in-BUILDING_SNAPSHOT.patch (3.9K, ../../[email protected]/3-v6-0001-Track-transactions-committed-in-BUILDING_SNAPSHOT.patch)
download | inline diff:
From 12dd3434ef13609b324bbbbe68a3f0e2a48934a2 Mon Sep 17 00:00:00 2001
From: ChangAo Chen <[email protected]>
Date: Fri, 21 Nov 2025 15:19:22 +0800
Subject: [PATCH v6 1/2] Track transactions committed in BUILDING_SNAPSHOT.
The historic snapshot previously didn't track transactions committed
in BUILDING_SNAPSHOT, this might result in a transaction taking an
incorrect snapshot and logical decoding being interrupted. So we need
to track these transactions.
We also need to handle the xlog which means a catalog change in BUILDING_SNAPSHOT
because the historic snapshot only tracks catalog modifying transactions.
---
src/backend/replication/logical/decode.c | 33 ++++++++++++++++++++----
1 file changed, 28 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index cc03f0706e9..de1bed30781 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -206,12 +206,16 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
uint8 info = XLogRecGetInfo(r) & XLOG_XACT_OPMASK;
/*
- * If the snapshot isn't yet fully built, we cannot decode anything, so
- * bail out.
+ * If the snapshot hasn't started building yet, the transaction won't be
+ * decoded or tracked by the snapshot, so bail out.
*/
- if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT)
+ if (SnapBuildCurrentState(builder) < SNAPBUILD_BUILDING_SNAPSHOT)
return;
+ /*
+ * Note that if the snapshot isn't yet fully built, the xlog is only used
+ * to build the snapshot and won't be decoded.
+ */
switch (info)
{
case XLOG_XACT_COMMIT:
@@ -282,18 +286,24 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
TransactionId xid;
xl_xact_invals *invals;
+ bool has_snapshot;
xid = XLogRecGetXid(r);
invals = (xl_xact_invals *) XLogRecGetData(r);
+ has_snapshot =
+ SnapBuildCurrentState(builder) >= SNAPBUILD_FULL_SNAPSHOT;
/*
* Execute the invalidations for xid-less transactions,
* otherwise, accumulate them so that they can be processed at
* the commit time.
+ *
+ * Note that we only need to do this when we are not fast-forwarding
+ * and there is a snapshot.
*/
if (TransactionIdIsValid(xid))
{
- if (!ctx->fast_forward)
+ if (!ctx->fast_forward && has_snapshot)
ReorderBufferAddInvalidations(reorder, xid,
buf->origptr,
invals->nmsgs,
@@ -301,7 +311,7 @@ xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
ReorderBufferXidSetCatalogChanges(ctx->reorder, xid,
buf->origptr);
}
- else if (!ctx->fast_forward)
+ else if (!ctx->fast_forward && has_snapshot)
ReorderBufferImmediateInvalidation(ctx->reorder,
invals->nmsgs,
invals->msgs);
@@ -419,7 +429,19 @@ heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* SnapBuildProcessRunningXacts().
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT)
+ {
+ /*
+ * If we are building snapshot and the xlog means a catalog
+ * change, we need to mark it in the reorder buffer.
+ *
+ * Now only XLOG_HEAP2_NEW_CID means a catalog change.
+ */
+ if (SnapBuildCurrentState(builder) >= SNAPBUILD_BUILDING_SNAPSHOT &&
+ TransactionIdIsValid(xid) && info == XLOG_HEAP2_NEW_CID)
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+
return;
+ }
switch (info)
{
@@ -1306,6 +1328,7 @@ DecodeTXNNeedSkip(LogicalDecodingContext *ctx, XLogRecordBuffer *buf,
Oid txn_dbid, RepOriginId origin_id)
{
if (SnapBuildXactNeedsSkip(ctx->snapshot_builder, buf->origptr) ||
+ SnapBuildCurrentState(ctx->snapshot_builder) < SNAPBUILD_CONSISTENT ||
(txn_dbid != InvalidOid && txn_dbid != ctx->slot->data.database) ||
FilterByOrigin(ctx, origin_id))
return true;
--
2.34.1
[application/octet-stream] v6-0002-Add-test-case-snapshot_build-for-test_decoding.patch (5.2K, ../../[email protected]/4-v6-0002-Add-test-case-snapshot_build-for-test_decoding.patch)
download | inline diff:
From d4747efa7c8103ec259a725051fff3bc4849dc17 Mon Sep 17 00:00:00 2001
From: ChangAo Chen <[email protected]>
Date: Fri, 21 Nov 2025 15:48:27 +0800
Subject: [PATCH v6 2/2] Add test case snapshot_build for test_decoding.
---
contrib/test_decoding/Makefile | 3 +-
.../test_decoding/expected/snapshot_build.out | 33 +++++++++++++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/specs/snapshot_build.spec | 46 +++++++++++++++++++
4 files changed, 82 insertions(+), 1 deletion(-)
create mode 100644 contrib/test_decoding/expected/snapshot_build.out
create mode 100644 contrib/test_decoding/specs/snapshot_build.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index acbcaed2feb..60210726566 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -9,7 +9,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
- skip_snapshot_restore invalidation_distribution parallel_session_origin
+ skip_snapshot_restore invalidation_distribution parallel_session_origin \
+ snapshot_build
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/snapshot_build.out b/contrib/test_decoding/expected/snapshot_build.out
new file mode 100644
index 00000000000..0fcf20cce86
--- /dev/null
+++ b/contrib/test_decoding/expected/snapshot_build.out
@@ -0,0 +1,33 @@
+Parsed test spec with 4 sessions
+
+starting permutation: s1_begin s1_insert s2_init s3_begin s3_insert s4_create s1_commit s4_begin s4_insert s3_commit s4_commit s2_get_changes
+step s1_begin: BEGIN;
+step s1_insert: INSERT INTO tbl1 VALUES (1);
+step s2_init: SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'test_decoding'); <waiting ...>
+step s3_begin: BEGIN;
+step s3_insert: INSERT INTO tbl1 VALUES (1);
+step s4_create: CREATE TABLE tbl2 (val1 integer);
+step s1_commit: COMMIT;
+step s4_begin: BEGIN;
+step s4_insert: INSERT INTO tbl2 VALUES (1);
+step s3_commit: COMMIT;
+step s2_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s4_commit: COMMIT;
+step s2_get_changes: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+------------------------------------------
+BEGIN
+table public.tbl2: INSERT: val1[integer]:1
+COMMIT
+(3 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index 99310555e6c..252a39b7727 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -65,6 +65,7 @@ tests += {
'skip_snapshot_restore',
'invalidation_distribution',
'parallel_session_origin',
+ 'snapshot_build',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/snapshot_build.spec b/contrib/test_decoding/specs/snapshot_build.spec
new file mode 100644
index 00000000000..334531dd219
--- /dev/null
+++ b/contrib/test_decoding/specs/snapshot_build.spec
@@ -0,0 +1,46 @@
+# Test snapshot build correctly, it must track committed transactions during BUILDING_SNAPSHOT
+
+setup
+{
+ DROP TABLE IF EXISTS tbl1;
+ DROP TABLE IF EXISTS tbl2;
+ CREATE TABLE tbl1 (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl1;
+ DROP TABLE tbl2;
+ SELECT 'stop' FROM pg_drop_replication_slot('isolation_slot');
+}
+
+session "s1"
+setup { SET synchronous_commit=on; }
+step "s1_begin" { BEGIN; }
+step "s1_insert" { INSERT INTO tbl1 VALUES (1); }
+step "s1_commit" { COMMIT; }
+
+session "s2"
+setup { SET synchronous_commit=on; }
+step "s2_init" { SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'test_decoding'); }
+step "s2_get_changes" { SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert" { INSERT INTO tbl1 VALUES (1); }
+step "s3_commit" { COMMIT; }
+
+session "s4"
+setup { SET synchronous_commit=on; }
+step "s4_create" { CREATE TABLE tbl2 (val1 integer); }
+step "s4_begin" { BEGIN; }
+step "s4_insert" { INSERT INTO tbl2 VALUES (1); }
+step "s4_commit" { COMMIT; }
+
+# T1: s1_begin -> s1_insert -> BUILDING_SNAPSHOT -> s1_commit -> FULL_SNAPSHOT
+# T2: BUILDING_SNAPSHOT -> s3_begin -> s3_insert -> FULL_SNAPSHOT -> s3_commit -> CONSISTENT
+# T3: BUILDING_SNAPSHOT -> s4_create -> FULL_SNAPSHOT
+# T4: FULL_SNAPSHOT -> s4_begin -> s4_insert -> CONSISTENT -> s4_commit
+# The snapshot must track T3 or the replay of T4 will fail because its snapshot cannot see tbl2
+permutation "s1_begin" "s1_insert" "s2_init" "s3_begin" "s3_insert" "s4_create" "s1_commit" "s4_begin" "s4_insert" "s3_commit" "s4_commit" "s2_get_changes"
--
2.34.1
^ permalink raw reply [nested|flat] 23+ messages in thread
end of thread, other threads:[~2025-11-21 08:16 UTC | newest]
Thread overview: 23+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-02-15 23:19 [PATCH v7 6/9] TupleHashTable.entrysize was unused except for instrumentation.. Justin Pryzby <[email protected]>
2020-02-15 23:19 [PATCH v4 6/7] TupleHashTable.entrysize was unused except for instrumentation.. Justin Pryzby <[email protected]>
2020-02-15 23:19 [PATCH v6 6/7] TupleHashTable.entrysize was unused except for instrumentation.. Justin Pryzby <[email protected]>
2020-02-15 23:19 [PATCH v3 6/7] TupleHashTable.entrysize was unused except for instrumentation.. Justin Pryzby <[email protected]>
2020-02-15 23:19 [PATCH v2 6/7] TupleHashTable.entrysize was unused except for instrumentation.. Justin Pryzby <[email protected]>
2020-02-15 23:19 [PATCH v9 7/8] TupleHashTable.entrysize was unused except for instrumentation.. Justin Pryzby <[email protected]>
2020-02-15 23:19 [PATCH v5 6/7] TupleHashTable.entrysize was unused except for instrumentation.. Justin Pryzby <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2024-08-12 08:34 Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
2024-08-12 10:35 ` Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state Bertrand Drouvot <[email protected]>
2024-08-12 11:38 ` Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
2024-08-13 04:23 ` Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
2024-08-13 06:19 ` Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state Bertrand Drouvot <[email protected]>
2025-06-27 10:29 ` Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state Ajin Cherian <[email protected]>
2025-06-28 03:44 ` Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state =?gb18030?B?Y2NhNTUwNw==?= <[email protected]>
2025-07-01 23:47 ` Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state Michael Paquier <[email protected]>
2025-11-21 08:16 ` Re: Historic snapshot doesn't track txns committed inBUILDING_SNAPSHOT state =?utf-8?B?Y2NhNTUwNw==?= <[email protected]>
2025-03-26 10:15 Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state Ajin Cherian <[email protected]>
2025-03-26 12:51 ` Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
2025-03-27 02:30 Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state Ajin Cherian <[email protected]>
2025-03-27 03:47 ` Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
2025-03-27 04:57 ` Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state Ajin Cherian <[email protected]>
2025-03-27 07:26 ` Re: Historic snapshot doesn't track txns committed in BUILDING_SNAPSHOT state =?ISO-8859-1?B?Y2NhNTUwNw==?= <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox